# MARIO: Multi-Agent Resilience Framework for Network Slicing in Tactical Networks

Hnin Pann Phyu, Razvan Stanica, Diala Naboulsi

<figure><img src="image_1.png" /></figure>

## To cite this version:

Hnin Pann Phyu, Razvan Stanica, Diala Naboulsi. MARIO: Multi-Agent Resilience Framework for Network Slicing in Tactical Networks. DDINS 2025 - 7th International Workshop on Data Driven Intelligence for Networks and Systems, Jun 2025, Montreal, Canada. hal-05018410

HAL Id: hal-05018410

https://inria.hal.science/hal-05018410v1

Submitted on 3 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# MARIO: Multi-Agent ResIlience framewOrk for
Network Slicing in Tactical Networks

Hnin Pann Phyu

Département de Génie Logiciel et TI

École de Technologie Supérieure (ÉTS)

Montreal, Canada

hnin.pann-phyu.1@ens.etsmtl.ca

Razvan Stanica

INSA Lyon, Inria, CITI

Villeurbanne, France

razvan.stanica@insa-lyon.fr

Diala Naboulsi

Département de Génie Logiciel et TI

École de Technologie Supérieure (ÉTS)

Montreal, Canada

diala.naboulsi@etsmtl.ca

**Abstract**—Network slicing constitutes a paradigm shift as it transforms a 5G network into a set of versatile sub-networks for designated users, with specific requirements on security levels and quality of service (QoS) demands. Therefore, organizations with very high security and non-negotiable QoS requirements, such as the military, are leveraging the utilization of 5G network slicing for their operations. However, some network management challenges remain before constructing a resilient network with 99.999% reliability to different attacks, while providing isolation, high throughput and low latency. In this respect, we propose a reinforcement learning-based multi-agent resilience framework, which comprises centralized training using global information and decentralized decision-making by individual agents, each corresponding to an access point, to autonomously adapt network slicing configurations based on the evolving tactical landscape. The proposed framework continuously assesses network conditions and threat scenarios, to dynamically allocate the required resources and mitigate vulnerabilities. Numerical results exhibit that our proposed framework effectively defends against different adversarial actions and maintains operational continuity without compromising the QoS.

**Index Terms**—Network Slicing, Tactical Network, Security, Reliability

## I. INTRODUCTION

Network slicing is one of the breakthroughs of 5G technology, allowing various vertical services to utilize a common 5G infrastructure, while respecting performance isolation [1]. Because of these benefits, network slicing is embraced not only for commercial services, but also for tactical telecommunication services [2]. Indeed, the expectation for contemporary tactical networks is to enable a variety of network operations simultaneously [3]. Broadly, those operations can be categorized as follows [4]: i) Tactical Communication Slice (TCS) - a slice designed to support voice, data and video services in dynamic and challenging environments; ii) Command and Control Slice (CCS) - a slice for high-priority communication for command and control operations; and iii) Logistics and Supply Chain Slice (LSCS) - a slice focused on managing logistics and supply chain-related data, facilitating inventory management and transportation coordination. While TCS and

CCS demand high throughput and very low latency, LSCS requires best effort quality of service (QoS) and the ability to support massive connectivity.

Network slicing is expected to satisfy these heterogeneous demands through a logical end-to-end network dedicated for specific military services. However, deploying network slicing in tactical environments must satisfy stringent security criteria, with three categories of attacks defined for tactical networks [5]. Radio air interface attacks (i.e. jamming) involve individuals or groups maliciously disrupting communication signals within a specific area. Physical adversaries encompass threats aimed at destroying the physical infrastructure or hardware components that support network operations. Finally, cyber-attacks (i.e. spoofing, ransomware) involve malicious actions taken to gain unauthorized access, steal sensitive information, or disrupt services. These attacks could result in temporary or prolonged loss of connectivity, affecting the QoS for some or all connected users.

Moreover, tactical networks involve not only ground base stations (GBS), but also unmanned aerial vehicles (UAV) and satellite networks. These components exhibit varying degrees of robustness against different attack types, and an intelligent control framework is needed to efficiently coordinate users across various slice instances. Taking into account these points, our key contributions are twofold. First, we introduce a systematic reinforcement learning (RL) driven Multi-Agent ResIlience framewOrk (MARIO), which comprises centralized training and decentralized decision-making by individual agents, adjusting network slicing configurations across access points, an essential feature in a possibly malicious environment. Second, we conduct an extensive simulation to validate the performance of our proposed solution, comparing it against different baseline methods.

The remaining part of this paper is assembled as follows. Section II reviews related works on network slicing in tactical networks. Section III introduces the system model and problem statement. In Section IV, we provide the detailed design of our proposed MARIO solution. We discuss the results in Section V and conclude the paper in Section VI.

<footnote>This work was supported by the Mitacs/Ultra Intelligence & Communications project IT25839 and by the ANR DEMON (ANR-20-CE25-0006) project.</footnote>
## II. RELATED WORKS

From a network architecture development perspective, [2] outlines a network slicing deployment on 5G technology for the military use cases, based on the 5G-VINNI reference architecture. Moreover, the authors emphasize the needs of end-to-end security, the robustness against different types of attacks, benefits of having edge cloud and satellite for the high reliability and providing wide geographical coverage. Moreover, the study in [6] introduces a novel paradigm, the so-called dynamic data driven open radio access network (3D-O-RAN), which is composed of sensing, AI and data fusion blocks, thereby achieving any given mission objectives in the context of tactical RAN slicing.

For a network resource management and control aspect, the authors in [7] introduce an adversarial RL agent to confuse the learning process of a target gNodeB and propose a defense mechanism to safeguard the overall network from these RL-based attacks. On the one hand, the adversarial RL aims to jam specific resource blocks (RBs), thereby maximizing the number of failed network slicing requests by user equipment (UEs). Their results reveal that such RL-based attacks are more effective than random or myopic jamming. On the other hand, defensive RL agents based on Q-Learning, randomness in decision-making and manipulated feedback approaches are used to counter these attacks.

The authors in [8] investigate a deployment of tactical network slices in an ad hoc network with limited resources. In this regard, they design a non-cooperative multi-agent game theory framework to compete for shared resources, where the agent of each slice instance aims to maximize its average throughput. Recognizing the impact of processing security-sensitive traffic on delay, the authors take into account a penalty factor to carefully balance the trade-off between security and delay factors.

Similarly, the study in [9] investigates the user association problem in scenarios where networks are over-saturated. Their knapsack optimization problem maximizes the number of associated devices subject to QoS and resource constraints. The simulation of a heuristic-based method validates the effectiveness of their proposed solution, indicating its applicability for optimizing tactical network deployments.

From this literature review, we note that none of the contemporary works comprehensively addresses both the network performance and security aspects in tactical networking. In this light, we leverage an RL-based solution to adapt the slicing configuration based on dynamic network conditions and threat scenarios in the context of a tactical environment.

## III. SYSTEM MODEL AND PROBLEM FORMULATION

In this section, we lay out our system model and problem statement accordingly.

### A. System Model

We consider an UAV-and-satellite-aided slicing driven tactical network, which consists of a set of GBSs denoted as $b \in \mathcal{B}$,

<figure><img src="image_3.png" /><figcaption>Fig. 1: Heterogeneous tactical network environment.</figcaption></figure>

a set of UAVs denoted as $v \in \mathcal{V}$, and a LEO satellite denoted as e. In fact, UAVs, GBSs and the satellite act as access nodes, which can be defined as $\mathcal{N} = \mathcal{V} \cup \mathcal{B} \cup e$. Figure 1 depicts this heterogeneous tactical network environment.

We define $i_n \in \mathcal{I}_n$ as a set of slice instances attached to an access node $n$. A given time horizon $T$ is divided into $t$ time slots. We denote as $\xi_{i_n}^t$ the allocated bandwidth by node $n$ to the slice instance $i_n$ at time $t$. We consider $\xi_n$ as total bandwidth of access node $n$, i.e. $\sum_{i_n \in \mathcal{I}_n} \xi_{i_n}^t \le \xi_n$. In our system, users can be mapped to the different slice instances of the corresponding access nodes based on their demand and location, i.e. $u \in \mathcal{U}_{i_n}^t$ is a set of users associated to slice instance $i$ of access node $n \in \mathcal{N}$ at time $t$. Each user $u$ is characterized by a tuple $(l_u^t, \delta_u^t)$, where $l_u^t$ is the traffic demand in bits, and $\delta_u^t$ is the maximum tolerable delay in milliseconds.

Herein, we consider three different types of attacks in our tactical network, which correspond to the three types of attacks discussed in Section I: jamming, spoofing and physical attacks. Accordingly, let us define two binary variables $x_{i_n}^t$ and $y_{i_n}^t$ to reflect jamming and spoofing attacks at slice instance $i_n$ at time $t$, respectively, where $x_{i_n}^t = 1$ means there is a jamming attack and $x_{i_n}^t = 0$, otherwise. Similarly, $y_{i_n}^t = 1$ means there is a spoofing attack and $y_{i_n}^t = 0$, otherwise. Moreover, in a case of physical attack, this is not specific to a certain slice. Therefore, we define $z_n^t = 1$ if a physical attack happens at access node $n$, impacting all the slices. Note that the scope of this system model can be extended by incorporating various types of security threats.

Moreover, to counter cyber-attacks in a sliced architecture, we adopt the "invisible slice" concept (named like this because it is not advertised in the network). In this approach, when a suspicious user is detected, they may be migrated to the invisible slice, acting much like a honeypot. Upon confirmation of malicious activity, the user is no longer accepted on regular slices or allocated resources. In a different set-up, this invisible slice can also serve as a backup for some users when their allocated resources are being jammed, but parts of the spectrum are still usable.

Given the allocated bandwidth $\xi_{i_n}^t$ for each slice, the
maximum achievable user rate $r_{i_n,u}^t$ is computed as:

$$r_{i_n,u}^t = \frac{\xi_{i_n}^t}{\mathbf{u}_{i_n}^t} \log_2(1 + \mathrm{SINR}_{n,u}^t) \quad (1)$$

where $\mathbf{u}_{i_n}^t$ is the number of users associated to slice $i_n$ at access node $n$ at $t$, $\mathrm{SINR}_{n,u}$ is the signal to interference and noise ratio between user $u$ and access node $n$ during $t$. Here, $r_{i_n,u}^t$ includes both the received signal strength of user $u$ associated with slice $i_n$ of access node $n$, and the load level of slice $i_n$. We compute the SINR as below:

$$\mathrm{SINR}_{n,u}^t = \frac{P_{n,u}^t |g_{n,u}^t|^2}{\sum_{n' \in \hat{N}_n} P_{n'}^t |g_{n',u}|^2 + N_0} \quad (2)$$

where $P_{n,u}^t$ represents the transmit power allocated to $u$ at access node $n$. $g_{n,u}^t$ is a channel gain, $\hat{N}_n$ represents the access nodes possibly interfering with $n$, and $N_0$ is an additive white gaussian noise (AWGN). The empirical path loss model used in our study is given by: $PL(d_{u,n}) = 128.1 + 37.6\log(d_{u,n})$, where $d_{u,n}$ is the distance (in km) of user $u$ with respect to $n$. Accordingly, we obtain the channel gain as: $g_{n,u} = 10^{-PL(d_{u,n})/10}$.

The QoS in tactical networks is delay-oriented, with numerous services having strict temporal constraints. In our system model, the total communication delay $\tilde{\delta}_{i_n,u}^t$ is the sum between transmission and propagation delays:

$$\tilde{\delta}_{i_n,u}^t = \delta_{i_n,u}^t + \rho_{i_n,u}^t = \frac{l_u^t}{r_{i_n,u}^t} + \frac{d_{u,n}}{\zeta} \quad (3)$$

where $\zeta$ is the radio wave propagation speed in free space.

## B. Utility Model

With tactical networks proposing resilient services in complex scenarios, we consider QoS as the main metric to evaluate the performance of the overall network. We note that, in a tactical network, user access control is a reasonable assumption, which also implies that UEs are only accepted in the (well dimensioned) system if their requirements can be met. Therefore, a possible degradation in terms of QoS can only come from an attacker, or from the network management operations. In this regard, we define the overall QoS of access node $n$ as:

$$\eta_n^t = \frac{\sum_{i_n \in \mathcal{I}_n} \eta_{i_n}^t}{|\mathcal{I}_n|} \quad (4)$$

Accordingly, the average QoS of slice instance $i_n$ for all the corresponding users is:

$$\eta_{i_n}^t = \frac{\sum_{u \in \mathcal{U}_{i_n}^t} \eta_u^t}{|\mathcal{U}_{i_n}^t|} \quad (5)$$

with each user satisfaction being delay-oriented, as discussed above, and formalized as:

$$\eta_u^t = \frac{1}{1 + e^{-\theta(\delta_u^t - \delta_{i_n,u}^t)}} - \varpi^{HO} \quad (6)$$

where $\theta$ is a constant deciding the steepness of the satisfactory curve and $\varpi^{HO}$ is the penalty cost for handover. This is

because handovers incur a supplementary delay and, hence, QoS degradation. We consider three types of handover scenarios: i) switching slice instance only, in which handover cost $\varpi_{slice}^{HO}$ is incurred, ii) switching access node only, where $\varpi_{an}^{HO}$ is incurred and iii) switching access node and slice instance, thereby incurring $\varpi_{an-slice}^{HO}$. These handover costs affect user QoS differently (i.e. $\varpi_{an-slice}^{HO} > \varpi_{an}^{HO} > \varpi_{slice}^{HO}$), and they are applied when user association changes between two consecutive time slots.

## C. Problem Formulation for Multi-Agent Framework

As tactical networks need to be resilient and cope with very dynamic situations, in this work, we target a distributed cooperative multi-agent solution, where all the agents are trained with a shared global reward.

In this regard, we formulate our adaptive slice deployment problem as a decentralized partially observable Markov decision process (Dec-POMDP) and propose to solve it with learning-based solutions to achieve the slicing adaptation in near real-time. Technically, a Dec-POMDP model is defined by a tuple $(\mathcal{N}, \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{P}, \mathcal{R})$, where $\mathcal{N}$ is the total number of agents, $\mathcal{S}$ represents the (finite) overall network states of all the agents and $\mathcal{O} = \mathcal{O}_1 \times \mathcal{O}_2 \times \cdots \times \mathcal{O}_\mathcal{N}$ is a set of joint observations of agents. In particular, at each time step, $s \in \mathcal{S}$ is the global state of the network environment while the observation of each agent $n$ is $o_n \in \mathcal{O}_n$ is a subset of $s$, i.e. $o_n \subset s$. Simplifying matters, each agent can solely observe their own local environment $o_n$, i.e. $s = o_1 \cup o_2 \cup \cdots \cup o_\mathcal{N}$. We denote as $\mathcal{A} = \mathcal{A}_1 \times \mathcal{A}_2 \times \cdots \times \mathcal{A}_\mathcal{N}$ the set of joint actions, and $\mathcal{A}_n$ as the set of actions available to agent $n$, which can be different for each agent. In the following, $a_n \in \mathcal{A}_n$ represents the action of agent $n$ at any point, and $a_{-n}$ denotes the actions of the remaining agents except agent $n$, i.e. $a = a_n \cup a_{-n}$, $a \in \mathcal{A}$. Note that matrix $P$, representing the transition probabilities between different states, is unknown to an RL agent. However, it occurs that, if an agent knows the current state and the reward obtained in each iteration, it can still converge to the optimal solutions through RL approaches [10]. Similarly, $\mathcal{R} = \mathcal{R}_1 \times \mathcal{R}_2 \times \cdots \times \mathcal{R}_\mathcal{N}$ is the set of joint reward functions. At each time step, for agent $n$, $\mathcal{R}_n$ gives the immediate global reward $r_n$, i.e. $r_n = \mathcal{R}_n(s, a_n, a_{-n}, s')$.

Denoted by $\pi_n: \mathcal{O}_n \to \mathcal{A}_n$, the policy $\pi_n$ of agent $n$ specifies how to select the actions based on the given context. Needless to say, the ultimate goal of agent $n$ is to find the optimal policy $\pi_n^*$.

The detailed local observation space, action and reward of an access node in our tactical network, which we denote as agent $n$ at time $t$, are discussed in the following.

1) **Observation space:** the observation space of access node $n$ is composed of its jamming, spoofing and physical attacks status at current time $t$, together with its average QoS during the previous time $t-1$. It is expressed as:

$$o_n^t = \{ x_{i_n}^t, \dots, x_{|\mathcal{I}_n|}^t, y_{i_n}^t, \dots, y_{|\mathcal{I}_n|}^t, z_n^t, \eta_n^{t-1} \} \quad (7)$$
2) **Action space**: the action space of the agent (or node) $n$ is the following:

* Set up new UAV nodes in case of jamming or physical attacks against a GBS. In this case, for simplicity, we consider that the UAV position is pre-planned, and we do not try to optimize it.

* Increase or decrease of bandwidth by a factor $\alpha$, i.e. $\xi_{in}^t = \xi_{in}^{t-1} \cdot (1 + \alpha)$.

* Navigate suspicious users to an invisible slice to reduce their attack capabilities, in case of spoofing attack occur at specific slices.

* Navigate legit users to an invisible slice, in case of jamming occurring at specific slices.

* Navigate legit users to slice instances running on nearby nodes (including the satellite node). Indeed, in case a GBS is compromised by physical or cyber attacks, an agent might navigate the users of the victim GBS to the nearby nodes. This action can be also used a complement to the first action in this list, as setting up a new UAV base station is not instantaneous.

It is worth noting that we rely on a multi-discrete action space, which is a more intuitive representation of real-world scenarios. This approach allows agents to learn complex decisions spaces more effectively. Formally speaking, it allows agents to make multiple simultaneous decisions at each time step, and thus agents can have finer control over their actions.

3) **Reward** is synchronized with our main objective, which is maximizing the QoS $\eta_n^t$ while ensuring network reliability. Hence, the reward of agent $n$ can be expressed as:

$$r_n^t = \begin{cases} \eta_n^t & \text{if } \eta_n^t \ge \hat{\eta}_n^t \\ \eta_n^t - 1 & \text{otherwise} \end{cases}, n \in \mathcal{N}, t \in T, \quad (8)$$

where $\hat{\eta}_n^t$ is a predefined targeted QoS. We can simply obtain the global reward $r_{global}^t$ as the summation of all the local rewards $r_n^t$, i.e. $r_{global}^t = \sum_{n \in \mathcal{N}} r_n^t$. Essentially, $r_{global}^t$ promotes a collaborative approach among all the access node agents. As previously discussed, the singular objective of each agent is to determine a policy $\pi_n$ that maximizes the expected cumulative global reward. This can be formally expressed as:

$$\max_{\pi_n} \mathbb{E} \left[ \sum_{t \in T} r_{global}^t \right] \quad (9)$$

## IV. PROPOSED SOLUTION

In this section, we discuss the details of our proposed multi-agent resilience framework. Specifically, our proposed MARIO framework comprises a set of agents denoted by $n \in \mathcal{N}$, where each agent $n$ makes a decision for its associated access nodes. We select two different RL methods and integrate them in MARIO: Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C). These two methods are mainly chosen due to their compatibility with the multi-discrete action space considered in our problem. Indeed, unlike other RL techniques, PPO and A2C can effectively learn policies for environments with a mix of discrete actions with

different dimensions. The detailed steps of PPO and A2C for agent $n$ are outlined in Algorithms 1 and 2, respectively.

The inputs of both algorithms are represented by some parameters of the neural networks used in the RL process: the initial policy parameters $\theta_0$, the initial value function parameters $\phi_0$ and the clipping threshold $\epsilon$. The algorithms produce as output the optimal parameters $\theta^*$ and $\phi^*$.

Algorithm 1 is executed for a number of epochs $K$ (as shown in Line 1). For simplicity, we drop the time notations and we use $\pi_k$ and $\theta_k$ instead of $\pi_n(k)$ and $\theta_n(k)$. After setting up the policy $\pi_k$ using the initial parameters, we run it for $T$ steps in order to collect a set of learning trajectories $D_k$ (Lines 2-3). This allows us to compute the corresponding rewards and advantage estimates based on a pre-defined value function $V_{\phi_k}$ (Lines 4-5). After that, in Lines 6-10, we update the policy parameters $\theta$ and $\phi$ by taking $J$ steps of minibatch sample from the learning trajectories set $D_k$. In Line 8, we update the policy by maximizing the PPO-Clip objective, typically via stochastic gradient ascent using the Adam optimizer. The PPO-Clip objective is defined as:

$$\mathcal{L}_{\theta_k}^{CLIP} = \sum_{t=1}^{T} \left[ \min \left( \rho_t(\theta) \hat{A}_t^{\pi_k}, \text{clip}(\rho_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t^{\pi_k} \right) \right] \quad (10)$$

where $\rho_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_\theta(a_t|s_t)}$ is the probability ratio between the new and old policies. Finally, the value function parameters $\phi$ are updated in Line 9, typically via gradient descent.

In Algorithm 2, we use a different RL method instead of PPO, namely A2C. The reasoning is similar with the one in Algorithm 1. The first difference comes from the computation of the expected reward $G_t$ in Line 4. Subsequently, the update of parameters $\theta$ and $\phi$ is different, by using their gradients $d\theta$ and $d\phi$, as indicated in Lines 5-10. Specifically, Line 7 involves updating the expected reward followed by the respective gradient updates (in Lines 8-9).

## V. EVALUATION

In this section, we evaluate our proposed solutions by comparing them with baseline methods. We begin by describing the simulation environment, followed by an overview of the benchmark approaches and implementations, and conclude with a discussion of the results.

### A. Experimental Setup

We consider each slice instance serves [15-35] users. The location of a user is defined randomly (uniformly) distributed in an area of 0.6 km around the access nodes. Needless to say, the LEO satellite covers the whole network area. The total time frame is 300 minutes (i.e. $T=18000$ s), with one minute intervals (i.e. $t=60$ s). Each type of attack occurs randomly with 20% probability over the 300 minutes time-frame. Additionally, each network slice instance is allocated different bandwidth resources. The detailed parameter considerations are outlined in Table I.
**Algorithm 1:** MARIO-PPO agent *n* learning procedure

**Input:** Initialize policy $\theta_0, \phi_0$, clipping threshold $\epsilon$

**Output:** $\theta^*, \phi^*$

1 **for** $k = 0, 1, 2, ..., K$ **do**

2 $\pi_k = \pi(\theta_k)$

3 Run policy $\pi_k$ in environment for $T$ time steps to collect set of trajectories $\mathcal{D}_k$

4 Compute reward-to-go: $G_1, \dots, G_T$, i.e.

$$G_t = \sum_{i=t}^{T} \gamma^{i-t} r_t$$

5 Compute advantage estimates: $\hat{A}_1^{\pi_k}, \dots, \hat{A}_T^{\pi_k}$, i.e.

$$\hat{A}_t^{\pi_k} = G_t - V_{\phi_k}(s_t)$$

6 **for** $j = 0, 1, 2, ..., J$ **do**

7 Sample a mini-batch experience from $\mathcal{D}_k$

8 $\theta_{k+1} = \operatorname*{argmax}_{\theta} \frac{1}{T} \mathcal{L}_{\theta_k}^{CLIP}$

9 $\phi_{k+1} = \operatorname*{argmin}_{\phi} \frac{1}{T} \sum_{t=1}^{T} (V_{\phi}(s_t) - G_t)^2$

10 **end for**

11 **end for**

**Algorithm 2:** MARIO-A2C Learning agent *n* learning procedure

**Input:** Initialize policy $\theta_0, \phi_0$, clipping threshold $\epsilon$

**Output:** $\theta^*, \phi^*$

1 **for** $k = 0, 1, 2, ..., K$ **do**

2 $\pi_k = \pi(\theta_k)$

3 Run policy $\pi_k$ in environment for $T$ time steps to collect set of trajectories $\mathcal{D}_k$

4 $G_t = \begin{cases} 0 & \text{for terminal } s_t, \\ V_{\phi_k}(s_t) & \text{for non-terminal } s_t \end{cases}$

5 Reset gradients: $d\theta \leftarrow 0$ and $d\phi \leftarrow 0$

6 **for** $t = T - 1, ..., 0$ **do**

7 $G_t = r(s_t) + \gamma G(s_{t+1})$

8 Accumulate the policy gradient w.r.t $\theta$:
$$d\theta \leftarrow d\theta + \nabla_{\theta} \log \pi_{\theta}(s_t, a_t)(G_t - V_{\phi_t}(s_t))$$

9 Accumulate the critic gradient w.r.t $\phi$:
$$d\phi \leftarrow d\phi + \nabla_{\phi} (G_t - V_{\phi_t}(s_t))^2$$

10 **end for**

11 Perform update of $\theta$ using $d\theta$ and of $\phi$ using $d\phi$, thus $\theta_{k+1} = \theta$ and $\phi_{k+1} = \phi$

12 **end for**

## B. Benchmark and Implementation Setup

We evaluate the performance of our proposed MARIO-A2C and MARIO-PPO techniques against a baseline and a static solution. The baseline approach involves no responses to each attack, while in the static solution, an agent only decides to deploy a UAV in the event of a physical attack.

All the models are implemented using a Python environment. The two MARIO agents are implemented using the PyTorch framework. Our network slicing-enabled tactical network environment is constructed using the OpenAI Gym library, providing the environment for interactions and learning for the MARIO agents. Finally, these models were trained on the high-performance Linux server provided by the Digital Research Alliance of Canada.

<table><caption>TABLE I: LIST OF PARAMETERS</caption><thead><tr><th>Parameter</th><th>Value</th></tr></thead><tbody><tr><td>Number of users per slice</td><td>[15-35]</td></tr><tr><td>δ<sup>τ</sup><sub>u</sub></td><td>TCS: [5-10] ms<br>CCS: [1-5] ms<br>LSCS: [10-15] ms</td></tr><tr><td>ξ<sub>in</sub> [TCS, CCS, LSCS]</td><td>[30,40,15] MHz</td></tr><tr><td>Total Bandwidth ξ<sub>n</sub></td><td>100 MHz</td></tr><tr><td>α</td><td>[-0.1, ..., 0.1]</td></tr><tr><td>Invisible slice bandwidth</td><td>10 MHz</td></tr><tr><td>N<sub>0</sub></td><td>-174 dBm</td></tr><tr><td>P<sub>n,u</sub></td><td>75-105 dBm</td></tr><tr><td>LEO satellite distance</td><td>3600 km</td></tr><tr><td>Wave propagation speed</td><td>3 · 10<sup>5</sup> km/s</td></tr><tr><td>ω<sup>HO</sup><sub>slice</sub></td><td>-0.00005</td></tr><tr><td>ω<sup>HO</sup></td><td>-0.0001</td></tr><tr><td>ω<sup>HO</sup><sub>an-slice</sub></td><td>-0.0002</td></tr><tr><td>Learning rate</td><td>0.001</td></tr><tr><td>Batch size</td><td>64 samples</td></tr><tr><td>Maximum episodes</td><td>40000</td></tr></tbody></table>

## C. Results Discussion

To gain a comprehensive understanding of our agents' performance, we analyze the reward trends over the training periods. In these initial findings, for simplicity, we limit ourselves to scenarios where at maximum one slice instance per access node is under attack in each time slot. Figure 2 depicts the average reward trend of our agents compared to baseline and static approaches. The curves have been smoothed by averaging data over a rolling window of 200 iterations.

As we can see in Figure 2, MARIO-PPO and MARIO-A2C exhibit significant superiority over baseline and static approaches. Despite employing a robust network design facilitated by UAVs and satellite, the baseline approach lacks intelligent decision making capabilities, thereby failing to meet the strict demands of the tactical network. The static solution effectively defends against physical attacks, but it lacks the capability to handle the other two types of attacks (jamming and spoofing) considered in our problem. On the other hand, with the aid of multi-discrete action spaces, MARIO agents can try different combinations of actions efficiently to learn more about the ever-changing environment and discover near-optimal policies. Upon close investigation, we notice that MARIO-A2C exhibits greater stability and converges faster compared to MARIO-PPO. Specifically, MARIO-A2C agents start to converge at 17,000 epochs, whereas MARIO-PPO agents only achieve convergence after 22000 epochs.

To further appreciate the performance of our proposed solution, we consider a more complex scenario, where two slice instances are under attack per access node simultaneously. Technically, increasing the level of complexity within the problem might challenge the performance of the agents to a certain extent. In this regard, we evaluate the performance of our proposed solution under the conditions of a maximum
<figure><img src="image_4.png" /><figcaption>Fig. 2: Comparison of average reward with maximum one attack per slice.</figcaption></figure>

of one slice instance under attack versus a maximum of two slice instances under attack per time slot, as illustrated in Figure 3. As expected, agents dealing with a network environment having two attacks simultaneously converge slower than agents dealing with one attack. However, they attain a similar reward as in the single attack scenarios, demonstrating that our RL-based strategy can adapt to more complex attacks. Specifically, MARIO-A2C (with 2 attacks) and MARIO-PPO (with 2 attacks) show a delayed convergence (approximately around 20000 epochs and 25000 epochs respectively) with respect to their behavior in scenarios with only one attack. Notably, MARIO-A2C (with 2 attacks) achieves convergence at a faster rate compared to MARIO-PPO, even when the latter deals with only one attack. This underlines the efficient learning procedure in MARIO-A2C. Needless to say, the performance of the baseline and static approaches deteriorate when confronted with increasingly complex situations.

<figure><img src="image_5.png" /><figcaption>Fig. 3: Comparison of average reward with attacks on two slices simultaneously.</figcaption></figure>

However, to reach its superiority, MARIO-A2C consumes more computation power than MARIO-PPO, as it can be seen in Figure 4. Therefore, in summary, RL-based solutions designed in the MARIO framework can adapt to different attack scenarios in tactical networks, preserving a high user QoS level. Among the two studied approaches, MARIO-A2C exhibits simplicity and stability, outperforming MARIO-PPO in more complex environments, but requiring more intensive

computations.

<figure><img src="image_6.png" /><figcaption>Fig. 4: Computing time comparison.</figcaption></figure>

## VI. CONCLUSION

In this study, we explore the deployment of slice instances in tactical networks in adversarial environments, where different types of attacks may occur. We follow a Dec-POMDP formulation to accurately model our problem and rely on two RL-based solutions (i.e. MARIO-PPO and MARIO-A2C) to deal with it. In this regard, the agents of our proposed MARIO framework aim to autonomously adapt network slicing configurations, providing an intelligent and dynamic response to attack scenarios, while respecting stringent QoS requirements. Numerical results demonstrate the ability of MARIO to effectively defend against adversarial actions while maintaining operational continuity, without compromising QoS compared to baseline and static approaches.

## REFERENCES

[1] H. P. Phyu, D. Naboulsi, and R. Stanica, "Machine learning in network slicing—a survey," *IEEE Access*, vol. 11, pp. 39 123–39 153, 2023.

[2] P. Gronsund, A. Gonzalez, K. Mahmood, K. Nomeland, J. Pitter, A. Dimitriadis, T.-K. Berg, and S. Gerladi, "5G Service and Slice Implementation for a Military Use Case," *Proc. IEEE ICC Workshops*, 2020.

[3] J. L. Burbank, P. F. Chimento, B. K. Haberman, and W. T. Kasch, "Key challenges of military tactical networking and the elusive promise of manet technology," *IEEE Communications Magazine*, vol. 44, 2006.

[4] L. Concha Salor and V. Monzon Baeza, "Harnessing the Potential of Emerging Technologies to Break down Barriers in Tactical Communications," *MDPI Telecom*, vol. 4, no. 4, pp. 709–731, 2023.

[5] J. Suomalainen, J. Julku, M. Vehkapera, and H. Posti, "Securing Public Safety Communications on Commercial and Tactical 5G Networks: A Survey and Future Research Directions," *IEEE Open Journal of the Communications Society*, vol. 2, no. July, pp. 1590–1615, 2021.

[6] F. Restuccia, E. Blasch, A. Ashdown, J. Ashdown, and K. Turck, "3D-O-RAN: Dynamic Data Driven Open Radio Access Network Systems," *Proc. IEEE MILCOM*, pp. 19–24, 2022.

[7] Y. Shi, Y. E. Sagduyu, T. Erpek, and M. C. Gursoy, "How to Attack and Defend NextG Radio Access Network Slicing With Reinforcement Learning," *IEEE Open Journal of Vehicular Technology*, vol. 4, pp. 181–192, 2023.

[8] F. Yang, C. Comaniciu, P. Krishnan, K. Sabnani, and L. Greenwald, "A Game Theoretic Framework for Distributed Mission Slice Allocation and Management for Tactical Networks," *Journal of Network and Systems Management*, vol. 31, no. 1, pp. 1–28, 2023.

[9] A. Castanares and D. K. Tosh, "Tactical Topology Optimization Methodology for Slice Aware and Reconfigurable Battlefield Networks," *Proc. IEEE MILCOM*, pp. 273–278, 2022.

[10] Y. Shoham and K. Leyton-Brown, *Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations*. Cambridge University Press, 2009, pp. 215–216.