Multi-UAVs Strategies for Ad Hoc Network with
Multi-Agent Reinforcement Learning
George Karimata∗ , Jin Nakazato∗ , Gia Khanh Tran† , Katsuya Suto‡ , Manabu Tsukada∗ and Hiroshi Esaki∗∗ The University of Tokyo, Japan† Tokyo Institute of Technology, Japan,‡ The University of Electro-Communications, Japan

Email: {karimata-george030, jin-nakazato, mtsukada}@g.ecc.u-tokyo.ac.jp
multiple UAVs for data collection from IoT sensors, navigatingaround no-fly zones and within limited flight distances [6].

Abstract—In recent years, extensive research has focused onleveraging advanced technologies beyond 5G and for Industry 5.0
to promote sustainability and prosperity in society. Our study
advances this effort by seeking to create an aerial perspective
using Unmanned Aerial Vehicles (UAVs). This paper introduces
a method for optimizing UAV deployment strategies using multiagent reinforcement learning, facilitating the formation of a flying
ad hoc network. The results demonstrate practical cooperation
among UAVs in flight.
Index Terms—UAV, flying ad hoc network, multi-agent reinforcement learning, multi-agent transformer

However, many studies, including those mentioned above,
focus on the optimal deployment of UAVs for static users andflight routing without considering communication range constraints. This paper presents a novel approach by addressing
the deployment and routing issues for moving UAVs within an
ad hoc network subject to communication distance limitations.
The goal of this paper is to derive an optimal strategy for
determining the flight paths of a UAV smarm using MultiAgent Reinforcement Learning (MARL), a proven solution
for cooperative tasks. Given the dynamic nature of the scenario—where multiple UAVs must navigate toward destinations within an ad hoc network—collaboration is imperative.
MARL has been extensively studied; for instance, MultiAgent Proximal Policy Optimization (MAPPO) [7] extends
the single-agent Proximal Policy Optimization (PPO) [8] algorithm by incorporating other agents’ data into value calculations. In contrast, Heterogeneous-Agent Proximal Policy Optimization (HAPPO) [9] addresses the limitations of MAPPO
by accommodating agents with varying action spaces. Wen etal. conceptualized MARL through a sequential model. They
introduced a Multi-Agent Transformer (MAT) [10], as depictedin Fig. 1, combined with the Transformer architecture [11],known for its efficacy with sequential data, such as naturallanguage. In this paper, simulations were performed using
MAT. The primary contributions of this paper are as follows.

I. I NTRODUCTION
Recently, Industry 5.0 has been envisioned by the European
Commission as a roadmap for the future, aiming to cultivatea prosperous society through the strategic use of advanced
technologies such as Artificial Intelligence (AI), Robotics, Big
Data, and the Internet of Things (IoT) [1]. These technologies find practical applications in Digital Twins (DT), which
serve as the conduit between the real and virtual worlds [2].
Unmanned Aerial Vehicles (UAVs), capable of freely moving
in three dimensions, are instrumental in realizing Industry
5.0. These UAVs are slated for deployment across a rangeof sectors. Furthermore, recent research has shed light on
the capabilities of Non-Terrestrial Networks (NTNs) to equip
UAVs and Low Earth Orbit satellites with robust communication systems. These networks are precious when terrestrial
infrastructure is compromised by natural disasters such asearthquakes and typhoons [3]. Among the promising solutionsin this arena are UAV base stations (UAV-BS), which offer
the flexibility to adjust their positions rapidly. These stations
are increasingly seen as vital for enhancing communication
performance in areas experiencing sudden spikes in data trafficor regions affected by disasters [4].
There are many studies on UAV deployment and flightrouting challenges. Park et al. proposed an algorithm to move
UAVs according to the distance between UAVs and the numberof mobile devices connected to them [5]. It has been successful in properly positioning the UAV swarm for stationary
users and increasing communication capacity. Bayerlein et al.
used multi-agent reinforcement learning to efficiently navigate

979-8-3503-3094-6/24/$31.00 ©2024 IEEE

Identify flight routes to destinations that maintain a functional ad hoc network, considering communication range
constraints.
• Coordinate the movement of UAVs to preserve the integrity of the ad hoc network through strategic positioning.
•The remainder of this paper is organized as follows. Section
II introduces our proposed methodology, Section III discusses
the simulation setup and results. Finally, Section IV concludes
this paper.

729

ICOIN 2024

Fig. 3. Positional relationship of four cases(1) The reward Rl involved in Landmarks in Condition 1 isformulated as follows.
rli
Rl =Fig. 1. Conventional multi-agent learning paradigm (left) and the multi-agent
sequential decision paradigm (right) [10]li ∈Lrl i =0 (if any UAV has already visited li )− min (dist(li , uj )) (else)uj ∈U

If any UAV has visited a Landmark, no penalty is applied tothe reward for that Landmark. However, if no UAV has reacheda particular Landmark, the distance from that Landmark tothe nearest UAV is deducted from the reward. This systemenhances the reward when a UAV approaches an unreachedLandmark, thus preventing the scenario where all UAVs converge on the same Landmark.
(2) The reward Ru associated with the UAV in Condition 2is determined by itself and the UAV uneighber closest to itself.
0 (if itself is at BS)Ru =ru (else)min (0, Lwave − dist(itself, uneighber ))(if uneighber is at BS)ru =
− |Lwave − dist(itself, uneighber )|
(else)UAV Control

Base Station

Data ProcessFig. 2. Proposed system model

II. S YSTEM D ESIGN
A. Design of Reward Function
The following description summarizes the settings assumed
in the proposed system model, as illustrated in Figure 2. Within
this model, a group of UAVs forms an ad hoc network. TheUAVs are initially located at the BS and move around thetarget area to collect data from specific destinations, referred
to as Landmarks. The UAV can obtain information such as theposition and velocity of all UAVs via an ad hoc network.

When the UAV is positioned at BS, it should not accrue
any rewards for maintaining its position there. However, whena UAV is located outside BS, its reward system will be
influenced by the proximity of other UAVs. If uneighber is at
BS and the distance is within radio range Lwave , no penalty
is given. The reason is that if a penalty is given, once theUAV leaves BS, it will leave BS by Lwave and will havedifficulty approaching the Landmarks in the vicinity of BS. On
the other hand, when uneighber is at BS, and the distance isout of radio range, or when uneighber is outside BS, penalize
the difference between Lwave and the distance to uneighber
to keep the distance at Lwave . These settings are expected toseparate the UAVs by Lwave , especially between UAVs outside
BS, and to distribute the UAVs.
(3) The connectivity reward Rc in Condition 3 is determinedusing an adjacency matrix, which is set to 1 when the distance
between BS and UAV is less than Lwave and 0 when thedistance is greater than Lwave . When the adjacency matrix
is multiplied by the number of UAVs n, it can be determinedthat the connection is possible when all components are greaterthan or equal to 1. For example, when BS and three UAVs arein the positional relationship of four cases, as shown in Fig. 3,

There is a limit to the reach of radio waves, and let Lwave
be the distance.
• Initial positions of UAVs are the same as BS.
• Set of UAVs is U and U = {u1 , . . . , un }.
• Set of Landmarks is L and L = {l1 , . . . , lm }.
•The conditions to be met are as follows.

1) At least one UAV should reach each Landmark.
2) UAVs should be reasonably far from each other tominimize radio interference.
3) Ensure the connectivity of ad hoc network.
4) The number of flying UAVs should be minimized tosatisfy the above conditions.
The rewards should be set such that these four conditions
are met. We set the reward that a single UAV will receivefor each condition. Hereinafter, dist(Entity1 , Entity2 ) shallrepresent the distance between Entity1 and Entity2 .

730

the adjacency matrix is A1 , A2 , A3 , A4 from left to right.
The powers of each are as follows.
1
1A1 = 00
1
1A2 = 00
1
1
A3 = 00
1
0A4 = 
00

1
11
101
10

1
1
10011
11
1
100
1
100
1
0000
10

01
0
1
00
1
1
00
0
1
00
0
1

22= 11
222(A2 ) = 10
222(A3 ) = 10
102(A4 ) = 
00(A1 )

224221221232112322320122001000010

1212012200
0
1
00
0
1

46= 
33
453(A2 ) = 31
453
(A3 ) = 301
03(A4 ) = 
00(A1 )3

61066364357633675575035400
1000010TABLE IH ARDWARE S PECIFICATIONSOSCPUGPUMEMSSD
36

34
135
4

00
01

00

0
1TABLE IIS IMULATION PARAMETERSParameter
Total Steps
Total Steps for Episode
Working Threads
Episodes
UAVs nLandmarks mRadio Wave Distance Lwave
Acceleration Vector Magnitude aArea Range for Landmarks limit

We know that communication can hop from BS to all UAVs
if all components of the adjacency matrix multiplied by nare greater than or equal to 1. Using this, the reward Rc isformulated as follows. The number of 0 elements and allelements of the adjacency squared to the power of n are
defined as Nzero and Nall , respectively.

The objective of this study is to employ MARL to devise
a methodology for determining the flight paths of a group ofUAVs that collectively form an ad hoc network. We conducted
MARL with UAVs as agents to acquire strategies. We utilize
the MAT within our framework. The training protocol proceeds
as follows: At the start of each episode, the environment is initialized. Subsequently, the policy dictates the agents’ actions,which are informed by the collective observed data and thepreceding actions of other agents. Adhering to the principles
of a Markov process, the forthcoming state is contingent on
the current actions and state, prompting an update in thesystem. Upon updating, the reward for each agent is computed.
This sequence of steps is iteratively conducted throughout theepisode. Upon its conclusion, the policy is revised to reflect
the cumulative observations, actions, and rewards acquired forthe episode.

Since the diagonal component is always greater than orequal to 1, the reward is subtracted according to the amountof zeros in the other components.
(4) Finally, set the reward Rb related to BS in Condition 4.
Value
128,000,000
100
256
5,000
625.00.50
13.0

B. MARL Training−Nzero−NzeroRc ==Nall − n − 1n(n + 1)

Rb =Windows 11
12th Gen Intel(R) Core(TM) i9-12900HX 2.30GHzNVIDIA GeForce RTX 4080 Laptop GPU
64GB
1TB0
(if the UAV is at BS)−1 (else)

If not at BS, the penalty will be given. The reward
earned by a single UAV is determined by adding up thesefour rewards (Rl , Ru , Rc , Rb ) with their respective weights(wl , wu , wc , wb ).

III. N UMERICAL R ESULTS
A. Setup Conditions

R = wl Rl + wu Ru + wc Rc + wb Rb (wl = wu = 1)

The hardware specifications used in this study are summarized in Table I. The main software used was Python 3.11.6
and PyTorch 2.0.1+cu118. The simulations were performed
using the publicly available training model MAT [12] and theMulti-Agent Particle Environment (MPE) [13] environment,
modified to fit this study. The five available actions of agentsare to accelerate with a magnitude a in either the x-y positiveor negative direction or not to accelerate. The following
equations update the agent’s velocity v and position p. The
decay rate γ = 0.75 and the microtome dt = 0.1 are set.

In this paper, since we consider that the UAV swarm isgoing to Landmarks while maintaining the ad hoc network,
we set the weight wc of the reward Rc to ensure connectivity is sufficiently large. Maximal at Rc = 0 when all
UAVs are connectable. The next largest value that can be−2−2n= n+1
when only one UAV istaken is Rc = n(n+1)
3isolated, as in (A3 ) . Rl accounts for the largest portion ofR, and its lowest value is suppressed from below by roughly− maxli ∈L (dist(BS, li )) ∗ m (= −limit). Therefore, we set
wc = limit ∗ n+1
2 to increase the penalty for connectivity overother rewards if even one UAV is isolated and fails to ensure
connectivity. Also, by having one UAV relay away from BS,
the UAV group can be expected to be approximately Lwave
closer to the Landmark and Rl larger. The weight of Rb iswb = Lwave .

vcurrent = vprevious ∗ γ + a ∗ dt
pcurrent = pprevious + vcurrent ∗ dt
When initializing the environment for each episode, Landmarks are placed uniformly at random within a 2∗limit square731

$YHUDJHHSLVRGHUHZDUGV

(SLVRGH 6WHS 

%6
8$98$9

%68$98$98$9

8$98$98$98$98$9
/DQGPDUN/DQGPDUN

8$9

8$9

H

6WHS

Fig. 4. Average episode rewards
,QGLYLGXDOUHZDUGV

(SLVRGH 6WHS 

/DQGPDUN/DQGPDUN(SLVRGH 6WHS (SLVRGH 6WHS 

8$98$9

%6

8$98$9

%6

8$98$98$98$98$98$98$98$98$9



6WHS


H

8$9

/DQGPDUN/DQGPDUN

8$98$9

Fig. 5. Individual rewards

8$9centered at BS. The simulation parameters are summarized inTable II.

/DQGPDUN

8$9

/DQGPDUN

Fig. 6. Snapshots of each situation in episode 5,000B. Simulation Results
The average episode rewards, as depicted in Fig. 4, alongwith the individual rewards of agents, illustrated in Fig. 5,both exhibit an upward trend with the progression of learning.
Sequential snapshots taken at steps 25, 50, 75, and 100
within episode 5,000, presented in Fig. 6, demonstrate that theUAVs can move while preserving network connectivity. Theseimages further reveal the strategic movements of the UAVs.
Some reach or are near Landmarks, while others strategicallymaintain a certain distance from the BS. These movements
collectively suggest coordinated behavior among UAVs.

R EFERENCES[1] S. K. Jagatheesaperumal and M. Rahouti. Building digital twins ofcyber physical systems with metaverse for industry 5.0 and beyond. IT
Professional, 24(6):34–40, 2022.
[2] K. Inokuchi et al. Semantic digital twin for interoperability andcomprehensive management of data assets. In 2023 IEEE International
Conference on Metaverse Computing, Networking and Applications(MetaCom), pages 217–225, 2023.
[3] M. M. Azari and thers. Evolution of non-terrestrial networks from 5g to6g: A survey. IEEE Communications Surveys & Tutorials, 24(4):2633–2672, 2022.
[4] G. K. Tran et al. NFV/SDN as an Enabler for Dynamic Placement
Method of mmwave Embedded UAV Access Base Stations. MDPINetwork, 2022.
[5] S. Park et al. Formation control algorithm of multi-uav-based network
infrastructure. Applied Sciences, 8(10), 2018.
[6] H. Bayerlein et al. Multi-uav path planning for wireless data harvestingwith deep reinforcement learning. IEEE Open Journal of the Communications Society, 2:1171–1187, 2021.
[7] C. Yu et al. The surprising effectiveness of ppo in cooperative multiagent games. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave,K. Cho, and A. Oh, editors, Advances in Neural Information Processing
Systems, volume 35, pages 24611–24624. Curran Associates, Inc., 2022.
[8] J. Schulman et al. Proximal policy optimization algorithms. CoRR,
abs/1707.06347, 2017.
[9] J. G. Kuba et al. Trust region policy optimisation in multi-agent
reinforcement learning. CoRR, abs/2109.11251, 2021.
[10] M. Wen et al. Multi-agent reinforcement learning is a sequence modelingproblem. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho,and A. Oh, editors, Advances in Neural Information Processing Systems,volume 35, pages 16509–16521. Curran Associates, Inc., 2022.
[11] A. Vaswani et al. Attention is all you need. In I. Guyon, U. Von Luxburg,
S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett,editors, Advances in Neural Information Processing Systems, volume 30.
Curran Associates, Inc., 2017.
[12] https://github.com/PKU-MARL/Multi-Agent-Transformer (Accessed on2023).
[13] https://github.com/openai/multiagent-particle-envs/ (Accessed on 2023).

IV. C ONCLUSION
In this study, we proposed a method for deriving deployment
strategies for UAV groups that form an ad hoc network, utilizing multi-agent reinforcement learning with MAT. Simulations
were conducted with reward structures designed to foster thedevelopment of an ad hoc network. The results indicatedthat, as learning progressed, the UAVs increasingly exhibitedcooperative flight behavior. Future work will involve adopting
an environmental model that more closely mirrors real-world
conditions, incorporating factors such as radio interference.
We aim to assess the effectiveness of our proposed method
further by exploring the optimization of routing paths througha routing protocol and comparing the performance of the ad
hoc network against alternative approaches.
ACKNOWLEDGMENT
This research was supported by the Strategic Information
and Communications R&D Promotion Programme (SCOPE)
by the Ministry of Internal Affairs and Communications, Japan(receipt number JP235003015).

732