# SNOW: A Split Reinforcement Learning Approach for Energy Efficiency in Tactical Network Slicing

Hnin Pann Phyu, Razvan Stanica, Diala Naboulsi

► To cite this version:

Hnin Pann Phyu, Razvan Stanica, Diala Naboulsi. SNOW: A Split Reinforcement Learning Approach for Energy Efficiency in Tactical Network Slicing. WoWMoM 2025 - 26th IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks, May 2025, Fort Worth (TX), United States. hal-05095915

HAL Id: hal-05095915

https://inria.hal.science/hal-05095915v1

Submitted on 3 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# SNOW: A Split Reinforcement Learning Approach for Energy Efficiency in Tactical Network Slicing

Hnin Pann Phyu

Département de Génie Logiciel et TI
École de Technologie Supérieure (ÉTS)

Montreal, Canada

hnin.pann-phyu.1@ens.etsmtl.ca

Razvan Stanica

INSA Lyon, Inria, CITI
Villeurbanne, France

razvan.stanica@insa-lyon.fr

Diala Naboulsi

Département de Génie Logiciel et TI
École de Technologie Supérieure (ÉTS)

Montreal, Canada

diala.naboulsi@etsmtl.ca

**Abstract**—Thanks to its ability to provide secure and efficient communication capabilities, network slicing has been adopted as a key technology, not just in commercial networks but also in tactical networks. Based on multiple virtual networks, each dedicated to a specific service, a sliced architecture meets the diverse requirements of highly heterogeneous tactical services. However, tactical networks operate in challenging environments with inherent power constraints, where the need for energy-efficient network slicing management solutions is paramount. In this direction, we tackle a joint slice activation/deactivation and user association problem with the aim of studying trade-offs between energy efficiency and user quality of service. To solve the problem, we introduce our original approach: a split reinforcement learning-based energy-efficient slicing deployment algorithm, namely SNOW. SNOW divides the deep neural network into multiple sections, where the front-end part of the model is trained over multiple user devices and then the back-end part of the model is trained by the central nodes (i.e. base stations in this case), without sensitive data sharing. Extensive simulation results reveal that the proposed scheme is superior to the considered benchmarks in improving energy efficiency while maintaining network performance.

**Index Terms**—Network Slicing, Tactical Network, Split Learning, Energy Efficiency, QoS

## I. INTRODUCTION

The advancement of next-generation wireless networks is revolutionizing the landscape of digital communications by enabling unprecedented throughput, reliability, and versatility to different industry segments. A key driver of this transformation is network slicing, a groundbreaking concept that allows a single physical network to be divided into multiple virtual networks, each dedicated to a specific industry [1]. The concept of network slicing has been embraced by diverse industries due to its ability to provide personalized, efficient, and secure network services. Specifically, it has gained particular interest from organizations with high security and stringent quality of service (QoS) requirements, such as the military for their tactical network operations [2].

Yet, military operations require energy-efficient networking solutions that enable communication for extended periods on limited power resources. Despite its benefits, network slicing introduces new challenges in developing sustainable

tactical network operations, as it leads to higher energy consumption compared to non-sliced networks. Needless to say, energy-efficient network operation is indeed not trivial task in a military operation. Moreover, modern military operations involve different types of devices, ranging from sensors and surveillance cameras to drones. These devices work together to create a robust and flexible tactical network that supports various military operations. But this also leads to higher complexity of the network operations, which implies higher energy consumption. In short, the dynamic and complex nature of the tactical network makes it tedious to build the most energy-efficient and guaranteed QoS network operation.

In this regard, machine learning (ML) driven solutions are expected to bring significant gains in very complex scenarios, such as in tactical networks, in contrast to simple scenarios where heuristics can be used to provide satisfactory network performance. Oftentimes, in commercial networks, operators use sensitive user data to feed ML solutions to gain insights and to achieve complex decisions driven by multiple objectives with potential trade-offs. In tactical networks, in many situations, for example in joint military operations, we need to collaborate with entities that do not want to share their data due to operational concerns. Precisely, user data like location and usage of each application are all sensitive data in tactical networks, and they are not supposed to be shared, especially in military combat operations. However, they are also relevant and important features for an ML model to make an informed decision. In addition to this, military operations are usually executed over resource-constrained networks, and the need for intelligent, yet light-weighted ML solutions is inevitable. Simply put, we need a solution that is capable of preserving privacy, optimizing decisions, and can rapidly converge with few computation resources.

This can be achieved by embracing the split learning concept. Split learning-based frameworks represent promising solutions, which divide a deep neural network (DNN) used in the ML process into multiple sections, where each section is trained by different learners (i.e. user devices and base stations) [3]. Concisely, the partial front-ends of the model are trained by several user devices with their local observation vectors, and the back-end of the model is trained

<footnote>This work was supported by the Mitacs/Ultra Intelligence & Communications project IT25839.</footnote>
by the associated base station. The entire model training, between the user device and base stations, is carried out by sending the weights of the last layer (i.e. cut layer) of the user device model to the base station. We highlight that instead of sending the real data to other users or a centralized entity, only the cut layer is sent, hence providing better communication efficiency and enhancing privacy. None of the user devices nor access nodes involved in this collective training can see each other's data. The access node then trains the remaining model by combining the cut layers of its associated user devices, which then achieves the complete model training. At its core, split learning promises three aspects: i) scalability, accommodating increasing device numbers without overloading the central node; ii) reduced bandwidth, as only the cut layer transverses the network, instead of the whole data (as in centralized approaches) or model (as in federated learning (FL) approaches), and iii) improved privacy, as no data is shared across entities.

The novel contributions of this paper are the following:

* Firstly, we formulate a joint slice activation and user association problem within the realm of tactical networks, aiming to minimize energy consumption while ensuring network performance in terms of QoS.

* We introduce our original approach to solve the problem: a split reinforcement learning framework (SNOW), where user devices collectively participate in training a model for their associated access node, without requiring the sharing of their local datasets or observations. Most prominently, these privacy-preserving aspects set SNOW apart, offering a cutting-edge solution for energy-efficient and intelligent network slicing management, especially in military operations with very strict security and privacy constraints. This is the first time in the literature (not only network-related) when split learning is integrated in a reinforcement learning (RL) setup.

* Furthermore, we utilize zero-padding to ensure the model performance remains versatile and consistent when the number of user devices varies during the training. To our knowledge, this is the very first solution that allows dynamic behaviors in split learning-based approaches.

* Lastly, we conduct an extensive simulation to validate the performance of our proposed solution. We rigorously prove the benefits of our proposed approach from different perspectives, essentially comparing it against decentralized and centralized approaches.

The rest of this paper is structured as follows: Section II reviews related work on energy-efficient network slicing in tactical networks. Section III presents the system model and performance metrics. We then cast our problem as a decentralized partially observable Markov decision process (Dec-POMDP) in Section IV. In Section V, we give a background of split learning and detail the design of our proposed framework. Section VI discusses the obtained results and

Section VII concludes the paper.

## II. RELATED WORKS

In this section, we first examine notable studies focused on improving network operation efficiency in tactical networks. Then, we review existing works on split learning to better contextualize our contribution.

### A. Efficient Tactical Network

Research efforts aimed at enhancing the efficiency of tactical network operations cover a diverse array of topics, including the deployment of unmanned vehicles, offloading users tasks, and the activation of network slice instances.

On the deployment of unmanned vehicles, [4] targets the optimization of drone-aided last mile communications, by determining the optimal location for the central controller and the number of drones to be deployed. They propose a computationally efficient mixed integer linear problem (MILP) model which scales up to 75 users. Similarly, in [5], the emphasis is on energy-efficient unmanned vehicle systems (UVS) in tactical mobile ad-hoc networks (MANETs). With this, the authors advocate a centralized time division multiple access (TDMA) and power scheduling scheme aimed at maximizing energy efficiency while considering QoS requirements and real-time multimedia data transmission in tactical MANETs.

In the case of user task offloading, the authors in [6] focus on efficiently allocating computational resources in individual user devices for tactical tasks and FL model training. Based on the incentives received by training the FL model, each user device makes the decision to offload (or not) their tactical-related tasks to the mobile edge computing (MEC) server. Meanwhile, [7] considers a framework allowing tactical edge nodes to offload computational intensive tasks to MEC servers residing on the ground, on unmanned aerial vehicles, and on low earth orbit satellites, to reduce delay and energy consumption. They design a hierarchical distributed algorithm to minimize energy consumption, delay, and overhead.

Regarding network slicing, in [8], the authors explore the problem of slice allocation in tactical networks with the goal of achieving higher throughput per slice. They propose a hierarchical game theory-based approach, where the first stage optimizes the computation server and the second stage focuses on optimizing packet routing through inter-slice cooperation.

As observed, research on efficient network operation in modern tactical networks is still in its early stages, with many areas yet to be explored and developed. Notably, existing studies on tactical networks do not comprehensively address the energy efficiency of the overall network while optimizing network performance, as we aim to do in this work.

### B. Split Learning Use Cases and Deployment

Cooperative and decentralized ML approaches bring tremendous benefits when addressing problems with privacy
and computation resource constraints. Together with FL, split learning has recently gained considerable interest in both research and industry. The works in [9]–[12] further justify that split learning performs better than federated schemes when it comes to imbalanced datasets and large scenarios for classification and prediction problems. Specifically, the authors in [9] explore a scenario where several entities hold different data of patients and local and centralized entities conduct collective model training based on split learning for patient diagnosis. For a similar reason, [10] utilizes different biomedical and electronic health record (EHR) data to train deep learning models to predict if a patient can be discharged, training the model in a split fashion. In the context of the Internet of Things (IoT), [11] employs image and speech datasets to train a one-dimensional convolutional neural network (1D-CNN) model, which is split between the IoT devices and server for classification problems, demonstrating the advantages of split learning in resource-constrained environments. The recent work in [12] designs a split learning-based cooperative approach among base stations (i.e. the base station holds the local dataset) to accurately predict mobile network traffic.

Furthermore, the studies in [13]–[15] target the optimal configuration of split learning on mobile edge networks. Specifically, [13] relies on a Multi-Armed-Bandit (MAB) model to dynamically split the DNN layers on a mobile edge. In [14], the authors propose the round-robin method for client selection to take part in split model training to ensure that all clients equally participate in the task. Picturing the 6G edge network, [15] advocates the hierarchical split learning model by considering the sweet spot between edge storage and communication and computing cost. For instance, a smaller split model can be placed in a very resource-constrained device (i.e. on the user side) while a larger split model can be placed in more powerful entities (i.e. on the edge side) and a medium split layer can be deployed at the base stations.

In a nutshell, in contemporary research, split learning has primarily been applied to classification and prediction problems. However, its potential in reinforcement learning methods remains unexplored. Having said that, we take the first step towards a new horizon, introducing a split reinforcement learning-based framework to achieve both energy and network efficiency in tactical slicing, while ensuring privacy.

### III. SYSTEM MODEL AND PERFORMANCE METRICS

In this section, we lay out the considered network model, the energy consumption model, and the user QoS model, all required as part of our target problem.

#### A. Network Model

In our tactical network model, we consider multiple base stations, each associated with multiple slice instances. We denote $\mathcal{B}$ as the base station set, and $b$ as a base station in $\mathcal{B}$. We then define $\mathcal{N}_b$ as the set of neighboring base stations capable of hosting users served by $b$. This is exhibited as:

$$ \mathcal{N}_b = \{b' \in \mathcal{B} \mid d_{b,b'} \le d_{thr}, b \ne b'\} \quad (1) $$

where $d_{b,b'}$ represents the distance between base stations $b$ and $b'$, and $d_{thr}$ is a predefined distance threshold for two base stations to be considered neighbors. We consider time as divided into slots. We use $t$ to refer to a single time interval and $T$ to represent a set of time intervals of interest.

In the realm of a tactical network, we consider four distinct types of slices in this model: i) Tactical Communication Slice (TCS) - a slice designed to provide voice, data, and video services, and hence requiring high throughput; ii) Command and Control Slice (CCS) - a slice for high-priority communication for command and control operations, where ultra-low latency is necessary; iii) Logistics and Supply Chain Slice (LSCS) - a slice focused on managing logistics and supply chain-related data which need an infrastructure to support massive connectivity; and iv) Eco slice - a minimalistic slice, supporting basic network functions and a limited number of users in an energy efficient way. The first three slices offer services with varying QoS levels and can be activated or deactivated as needed, while the Eco slice is always active. Note that our problem can be extended to include more slice instances beyond those mentioned above. Consequently, we define $\mathcal{I}$ as the set of slice instances. Each slice instance $i \in \mathcal{I}$ is characterized by a specific QoS class identifier (QCI). In the following, we focus on delay as a QoS metric, therefore $\delta_i$ is a predefined guaranteed delay of slice instance $i$.

The bandwidth allocated to each slice $i$ at base station $b$ is represented as $BW_{i,b}$. Therefore, the total bandwidth $BW_b$ is calculated as $BW_b = \sum_{i \in \mathcal{I}} BW_{i,b}$. Furthermore, we assume the base station $b$ has a set of possible configurations denoted as $\mathcal{K}_b$. Each configuration $k \in \mathcal{K}_b$ represents the activation or deactivation decisions for the slices. Specifically, $k = \{c_{i,b}^t | i \in \mathcal{I}\}$, where $c_{i,b}^t = 1$ if slice instance $i$ at base station $b$ is active at time $t$, and 0 otherwise. As explained, we consider the Eco slice to be active all the time.

We denote the set of users served by slice $i$ of base station $b$ at time $t$ as $\mathcal{U}_{i,b}^t$. Then, the total set of users of $b$ can be defined as $\mathcal{U}_b^t = \bigcup_{i \in \mathcal{I}} \mathcal{U}_{i,b}^t$. Note that users are randomly distributed within the coverage area of base station $b$, which is a reasonable assumption in a tactical network. We consider user mobility such that users can be moving around randomly at each time slot. It is important to note that, in our system, users can simultaneously access more than one type of slice instances, reflecting realistic tactical network use cases [16].

The service requirement of user $u$ at time $t$ is characterized by $(l_{u,i}^t | i \in \mathcal{I})$, where $l_{u,i}^t$ is the traffic load demand in bits with regard to slice instance $i$. Given the bandwidth $BW_{i,b}$, the achievable data rate $r_{b,u,i}^t$ for user $u$ requesting slice $i$ and served by base station $b$ at time $t$ can be computed as:

$$ r_{b,u,i}^t = \frac{BW_{i,b}}{|\mathcal{U}_{i,b}^t|} \log_2(1 + SINR_{b,u,i}^t) \quad (2) $$

where $SINR_{b,u,i}^t$ is the signal to interference and noise ratio (SINR) for user $u$ at base station $b$ at time $t$. In fact, $r_{b,u,i}^t$ accounts for not only the received signal strength by user
u, but also for the load level of slice i. The SINR can be computed as follows:

$$SINR_{b,u}^{t} = \frac{P_{b,u}^{t} |g_{b,u}^{t}|^2}{\sum_{b' \in N_b} P_{b',u}^{t} |g_{b',u}^{t}|^2 + N_0} \quad (3)$$

in which $P_{b,u}^t$ is the transmit power allocated to user u by base station b at time t and $g_{b,u}^t$ is the corresponding channel gain. The additive white Gaussian noise (AWGN) is denoted by $N_0$. In our model, the path loss is given by $PL(d) = 128.1 + 37.6 \log(d)$, where d is the distance between the transmitter and the receiver. At this stage, the channel gain can be calculated as $h_d = 10^{-PL(d)/10}$.

Finally, the transmission delay $\delta_{b,u,i}^t$ for user u, served by slice i of base station b at time t can be calculated as:

$$\delta_{b,u,i}^{t} = \frac{l_{u,i}^{t}}{r_{b,u,i}^{t}} \quad (4)$$

## B. Energy Consumption Model

Generally, the overall base station energy consumption $\chi_b^t$ includes the combined energy consumption of all associated slice instances along with the static power value of the base station, $P_b^{static}$ (i.e., energy used for cooling and circuit power):

$$\chi_b^t = \sum_{i \in \mathcal{I}} c_{i,b}^t \cdot E_{i,b}^t + P_b^{static} \quad (5)$$

where $E_{i,b}^t$ is the energy consumption of slice i of base station b during time slot t, which includes a load-dependent power consumption component and a load-independent power consumption component. Mathematically, it can be written as:

$$E_{i,b}^{t} = \psi_{i} \cdot \rho_{i,b}^{t} \cdot P_{b}^{load} + \psi_{i} \cdot P_{b}^{fixed} \quad (6)$$

For the first part of Eq. 6, the load-dependent component is obtained using a static power transmission value per quantity of information, denoted as $P_b^{load}$, and $\rho_{i,b}^t$, the traffic load ratio of the associated slice instance i at base station b during t. This normalization parameter $\rho_{i,b}^t$ is obtained by dividing the traffic load observed by slice $l_{i,b}$ by the maximum traffic load $\mathcal{L}_{max,b}$ registered by base station b, as below:

$$\rho_{i,b}^{t} = \frac{l_{i,b}^{t}}{\mathcal{L}_{max,b}} \quad (7)$$

We consider $\mathcal{L}_{max,b}$ to be a predefined parameter representing the capacity of a base station, known to the operator.

In fact, the energy consumption of each slice instance is different [17]. As different network functions are activated for each slice, even under the same amount of traffic load, the energy consumption of each slice is different. Again, this explains the use of the power consumption impact factor $\psi_i$ of slice i in Eq. (6). The value of $\psi_i$ is different for each slice instance.

As for the load-independent component in Eq. (6), it is important to stress that a slice instance consumes some power even at zero load traffic, in order to instantiate its corresponding network functions. In this respect, $P_b^{fixed}$ is

independent of the traffic load, but still specific to the slice type, hence still regulated by the parameter $\psi_i$.

## C. User QoS Model

We define an aggregated metric, the overall QoS of a base station $Q_b^t$, as follows:

$$Q_b^t = \frac{\sum_{i \in \mathcal{I}} Q_{i,b}^t}{|\mathcal{I}|} \quad (8)$$

where the average QoS for all users of slice instance i of base station b at t, $Q_{i,b}^t$ can be obtained by:

$$Q_{i,b}^{t} = \frac{\sum_{u \in \mathcal{U}_{i,b}^{t}} Q_{u}^{t}}{|\mathcal{U}_{i,b}^{t}|} \quad (9)$$

In what follows, the QoS measured by a user u at time t, in terms of delay, is defined as:

$$Q_u^t = \frac{1}{1 + e^{-\theta(\delta_i - \delta_{b,u,i})}} - HO \quad (10)$$

Here, $\theta$ is a constant that determines the steepness of the user satisfaction curve, and $HO$ represents the penalty cost for handovers, which cause additional delays and degrade QoS. We hereby consider three types of handover scenarios: i) switching between slice instances only, incurring a handover cost $HO_{slice}$, ii) switching between base stations only, incurring a penalty $HO_b$, and iii) switching both base stations and slice instances, thereby incurring a cost $HO_{b-slice}$. Needless to say, these handover costs impact user QoS differently, in such a way that $HO_{b-slice} > HO_b > HO_{slice}$. In our model, handover costs are applied when user association changes between two consecutive time slots.

# IV. PROBLEM FORMULATION AS DEC-POMDP

Our work targets a cooperative multi-agent approach and this setting naturally aligns with a Dec-POMDP framework. In this light, we formulate our joint slice activation/deactivation and user association problem as a Dec-POMDP model, described below.

A Dec-POMDP is defined by $(\mathcal{M}, \mathcal{S}, \mathcal{A}, \mathcal{O}, P, \Omega, \mathcal{R})$, a tuple where $\mathcal{M} = \{1, \dots, m\}$ is the set of m agents, $\mathcal{S}$ represents a finite set of states, $\mathcal{A}$ is a finite set of joint actions, i.e. $\mathcal{A} = \mathcal{A}_1 \times \mathcal{A}_2 \times \dots \times \mathcal{A}_m$. At each time step t, each agent j takes an action $a_{j,t}$, forming one joint action $a = (a_1, \dots, a_m)$. Similar to the set of joint actions, $\mathcal{O} = \mathcal{O}_1 \times \mathcal{O}_2 \times \dots \times \mathcal{O}_m$ is the set of joint observations, where $\mathcal{O}_j$ is a set of observation available to agent j. Again, at each time step, the environment emits one joint observation $o = (o_1, \dots, o_m)$, from which each agent j can only observe its own component $o_{j,t}$. In Dec-POMDP, P is a transition probability function defined as $P: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to [0, 1]$ which captures the probability function to transition to $s'$ from current state s based on action a. We denote $\Omega$ as an observation function defined as $\Omega: \mathcal{S} \times \mathcal{A} \times \mathcal{O} \to [0, 1]$ and $\mathcal{R}$ is a reward function defined as $\mathcal{R}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$ that maps states and joint actions to real numbers. In general, P and $\Omega$ are unknown to an agent. However, RL can be used
to learn optimal policies without prior knowledge of these probabilities and functions [18].

Here, we conceive that the joint slice activation/deactivation and user association problem can be viewed as a simplified instance of an MDP, aligning naturally with the fundamental concepts of MAB problems. Therefore, we present an equivalent formulation as a state-aware MAB in the following, which we aim to solve using a split learning-based approach.

**State and observation:** In this model, the observations of user $u$ at base station $b$ consist of highly private information, with a potential impact on the success of a mission. More precisely, we consider that traffic load demand and locations of user $u$ at current time $t$ and next time step $t+1$ are known, which can be expressed as:

$$o_u^t = \{l_{u,i}^t, l_{u,i}^{t+1}, loc_u^t, loc_u^{t+1} | i \in \mathcal{I}\} \quad (11)$$

In our proposed framework, neither the entire observations nor each user's local observations are directly accessible to any individual agent $b$. However, for future use, we still describe the combined observations of all users as:

$$o_b^t = o_1^t \cup o_2^t \cup \cdots \cup o_{|\mathcal{U}_b^t|}^t \quad (12)$$

**Action:** For a base station $b$, a set of possible actions $\mathcal{A}_b$ includes a set of configurations $\mathcal{K}$, as defined in Section III-A, and a set of user association decisions. Each action can be expressed as below:

$$a_b^t = \{ \{c_{i,b}^t \mid i \in \mathcal{I}\}_{k \in \mathcal{K}_b}, \{x_{u,1}^t, \cdots, x_{u,|\mathcal{B}|}^t \mid u \in \mathcal{U}_b^t\} \} \quad (13)$$

where $x_{u,b}^t = 1$ if user $u$ is associated to base station $b$ at time $t$ and 0, otherwise. Note that the user can be associated with only one base station at a time, i.e. $\sum_{b \in \mathcal{B}} x_{u,b}^t = 1$.

**Reward:** We define the reward function to reflect the primary objectives of the problem. The goal of the base station agent is to maximize energy efficiency while maintaining the QoS for the users. Consequently, the immediate global reward function for the base station agent is formulated as follows:

$$r_b^{\text{global}}(t) = \sum_{b \in \mathcal{B}} r_b^{\text{local}}(o_b^t, a_b^t) \quad (14)$$

with

$$r_b^{\text{local}}(o_b^t, a_b^t) = \frac{1}{\chi_b^t} + \beta \cdot Q_b^t \quad (15)$$

where $\beta$ is the QoS impact factor, with a larger $\beta$ assigning more importance to the QoS criteria. The global reward, $r_b^{\text{global}}(.)$, is the sum of all local rewards $r_b^{\text{local}}(.)$ from the base station agents. In fact, this global reward encourages cooperation among the base station agents. Ultimately, the objective of the base station agent $b$ is to find a policy $\pi_b$ that maximizes the expected cumulative global reward. Mathematically, this can be expressed as:

$$\max_{\pi_b} \mathbb{E} \left[ \sum_{t \in T} r_b^{\text{global}}(t) \right] \quad (16)$$

<figure><img src="image_2.png" /><figcaption>Fig. 1: Three types of split learning.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2: SNOW architecture for energy efficiency in RAN slicing. For illustration purposes, we consider only one base station and $N$ number of users.</figcaption></figure>

## V. SNOW: SPLIT REINFORCEMENT LEARNING-BASED FRAMEWORK WITH ZERO PADDING

In this section, we begin with some preliminaries on split learning and zero-padding, followed by the detailed design of our proposed solution, SNOW.

### A. Split Learning and Zero-Padding Preliminaries

Split learning is a variation of FL, a technique that enables a local client and a server to collaboratively train a global model using decentralized data. Although FL is designed to prevent the sharing of the learners' data, the exchange of whole model updates during training can inadvertently disclose private information. Of course, it is much less likely to recover the original data when the weights and structure of the model are not shared [19]. In this light, split learning was introduced in 2018 [3]. As its name suggests, split learning divides a DNN into multiple sections which are trained on several devices (i.e. client and server) collectively, by exchanging the so-called cut layers, without necessarily sharing the data. Figure 1 shows the three common types of split learning [9]: i) Vanilla Split Learning - the simplest split learning model structure involves a single client and a single server sharing a label, ii) U-Shaped Configurations for Split Learning (as in the middle of Figure 1) the model
is shared between only one client and one server, with the label not being shared, and iii) Vertically Partitioned Data for Split Learning - this method allows for training a split model across multiple devices, with label sharing.

Our method draws inspiration from the third approach (on the right side of Figure 1), with two key modifications. First, no labels need to be shared because, in our setting, the back-end server model can independently calculate the true value (i.e. the label) without relying on the clients. Second, we employ a zero-padding technique to preserve the input sequence length of the back-end model when the number of clients vary. Zero-padding, commonly used in Convolutional Neural Networks (CNNs) to maintain the spatial dimensions (height and width) of an image as it passes through convolutional layers [20]. This involves adding zeros around the edges of the image, hence the term zero-padding. In split learning, zero-padding allows for the dynamic participation of clients while ensuring the consistent input at the server, thus stabilizing the learning process. The self-explained example of our proposed paradigm, SNOW is illustrated in Figure 2 for a single base station with multiple user devices.

In Figure 2, user devices train their partial front-end models with their local observation data, up to the cut layers. The cut layers are then forwarded to their associated base station to complete the learning of the remaining back-end model. The back-end model employs zero-padding to accommodate the dynamic participation of users in the training process, ensuring stability. This is very important for our tactical network use-case, where the number of users associated to the network is not fixed. Subsequently, the back-end model estimates the reward distribution as an output function. The loss value is calculated between the predicted reward and the true rewards. This loss value is then used to perform backpropagation to update the parameters of the model. Only the gradients of the back-end model are then sent to the front-end models for updating their weights. Notably, since no entire model weights are shared across agents, split learning provides better privacy, computational and communication efficiency compared to FL [15].

## B. Proposed Model

In what follows, we lay out the detailed learning steps of the agent at base station b. The inputs of the algorithm include the probability of selecting a random action $\epsilon$, the learning rate $\eta$, and the maximum number of time steps $T$ (or epochs) to train the model. The output is a trained model that can accurately predict the reward distribution $\hat{R}_{W_b}^*$ of all the possible actions.

Algorithm 1 begins by initialization with random values of the weights $W_b$ for the back-end model and $W_u$ for the front-end models of all the associated users of base station b (Line 1). At each time step $t$, each user $u$ observes its local observation $o_u^t$ (Line 4). For forward propagation, the front-end DNN function denoted as $\mathcal{F}\mathcal{E}(\cdot)$ processes the input observations up to the cut layer, where $X_u^c$ and $W_u$ represent

<table><caption>**Algorithm 1:** SNOW: Base station b agent learning strategy</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Input:</td><td>Probability of selecting a random action $\epsilon$, learning rate $\eta$, maximum time steps $T$</td></tr><tr><td>Output:</td><td>$\hat{R}_{W_b}^*$</td></tr><tr><td>1</td><td>Initialize $W_b$ and $W_u$ randomly</td></tr><tr><td>2</td><td>for $t \in T$ do</td></tr><tr><td>3</td><td>for $u \in U_b^t$ do</td></tr><tr><td>4</td><td>Each user observes partial observation:<br>$o_u^t = \{l_{u,i}^t, l_{u,i}^{t+1}, \text{loc}_u^t, \text{loc}_u^{t+1} | i \in \mathcal{I}\}$</td></tr><tr><td>5</td><td>Front-end model process till cut-layer:<br>$X_u^c = \mathcal{F}\mathcal{E}(o_u^t; W_u)$</td></tr><tr><td>6</td><td>Cut Layer is transmitted to the back-end</td></tr><tr><td>7</td><td>end for</td></tr><tr><td>8</td><td>Concatenate the cut layers:<br>$X^c = \{X_1^c, X_2^c, \dots, X_{|U_b^t|}^c\}$</td></tr><tr><td>9</td><td>Ensure the dimensional consistency with zero padding: $X_{\text{padded}}^c = \text{pad}(X^c)$</td></tr><tr><td>10</td><td>Back-end model complete the model training:<br>$X_b = \mathcal{B}\mathcal{E}(X_{\text{padded}}^c; W_b)$</td></tr><tr><td>11</td><td>Predict reward distribution for each action:<br>$\hat{R}_{W_b}(t)$</td></tr><tr><td>12</td><td>if $\text{rand()} < \epsilon$ then</td></tr><tr><td>13</td><td>Choose a random action $a_b^t$</td></tr><tr><td>14</td><td>else</td></tr><tr><td>15</td><td>Choose $a_b^t = \text{argmax}_{a_b^t} [\hat{R}_{W_b}(t)]$</td></tr><tr><td>16</td><td>end if</td></tr><tr><td>17</td><td>Obtain true reward $r_b^{\text{local}}(o_b^t, a_b^t)$ according to Equation (15)</td></tr><tr><td>18</td><td>Evaluate global reward $r_b^{\text{global}}(t)$ according to Equation (14)</td></tr><tr><td>19</td><td>Calculate the loss:<br>$\mathcal{L}(W_b) \triangleq (r_b^{\text{global}}(t) - [\hat{R}_{W_b}(t)]_{a_b^t})^2$</td></tr><tr><td>20</td><td>Compute the gradients and update the weights:<br>$W_b \leftarrow W_b - \eta \nabla \mathcal{L}(W_b)$</td></tr><tr><td>21</td><td>Only the gradients of cut layer are sent back to the users in $U_b^t$</td></tr><tr><td>22</td><td>for $u \in U_b^t$ do</td></tr><tr><td>23</td><td>$W_u \leftarrow W_u - \eta \nabla \mathcal{L}(W_u)$</td></tr><tr><td>24</td><td>end for</td></tr><tr><td>25</td><td>end for</td></tr></tbody></table>

the output of the front-end (i.e. the cut layer) and the weight matrix of the front-end model at user $u$ (Line 5).

Each user sends their cut layer to their respective base station b where the back-end model is located (Line 6), where the cut layers are concatenated (Line 8). At this stage, in case the number of participating users fluctuates, zero padding is applied to ensure dimensional consistency. Concisely, if the concatenated cut layer $X^c$ has different dimensions during training, the structure is padded with zeros
(i.e. $X_{padded}^c = pad(X^c)$) to keep the same dimension before being processed by the back end model (Line 9). With the resulting padded layer, the back-end model completes the model training until reaching the output layer $X_b$, with $\mathcal{BE}(\cdot)$ being the function of the back-end DNN (Line 10).

Thereafter, the back-end model generates the predicted reward distribution for all the available actions (Line 11). An action is chosen based on the exploration and exploitation trade-offs (Line 12 - Line 16). Based on the chosen action, the actual local reward is obtained (Line 17), and yet the global reward is calculated (Line 18). Then, the loss is calculated by comparing it against the true reward value (Line 19), followed by the computation of gradients and the update of the weight matrix of the back-end DNN (Line 20). Next, only the gradients of the cut layer are sent back to the front-end agents, and associated model weights are updated accordingly (Line 21 - Line 24). This process is repeated until the whole model is fully trained.

# VI. PERFORMANCE EVALUATION

This section begins with an explanation of our experimental setup, followed by a description of benchmark approaches. Eventually, we discuss the overall results.

## A. Experimental Setup

We consider a tactical network with 5 base stations, where each base station serves [6 – 10] users. Within a given base station coverage area, the location of a user is randomly distributed. If base stations are at a distance below 2 km, they are considered to be neighbors. We define 4 different slice instances as TCS, CCS, LSCS, and Eco slices, each of which has its own guaranteed delay (except the Eco slice) and allotted bandwidth. We set the duration of time slots in our model as $t = 1$ minute.

When it comes to splitting the DNN, we consider the front-end models for all the users to be the same size and the back-end models for all the base stations have the same size as well. Note that we split the DNN based on trial and error, rather than using another specific solution to split it, as in [13]–[15]. More precisely, the front-end model at the user side is composed of two fully connected layers of 50 neurons, followed by a RELU activation function for each. The back-end model at the base station side is composed of two fully connected layers of 500 neurons in the first layer and 250 layers in the second layer, followed by a RELU activation function for each. For brevity, the remaining parameters are given in Table I.

## B. Benchmark and Implementation Setup

To thoroughly understand the potential benefits of SNOW, we evaluate it against different benchmarks, both decentralized (i.e. AllActive, AllActive (no agent), Random) and centralized (i.e. CentralizedSP, CentralizedDIFF) approaches. A description of these benchmarks is provided below:

* AllActive: All the slice instances are active, and only the user association decision is made by SNOW.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Global rewards comparison with decentralized approaches.</figcaption></figure>

* AllActive (no agent): All the slice instances are active and users are associated with the nearest base station.

* Random: Slice activation/deactivation and user association decisions are made randomly.

* CentralizedSP: This is a centralized approach at base station $b$, without split learning and with the same partial observation as SNOW. This implies that the partial state of the central agent $b$, as shown in Eq. 11, is known. It is important to note that, in practice, user locations and their per-application usage data are not available to the central node, and they represent sensitive data. However, this benchmark is selected intuitively for the sake of comparison and to better understand the impact of the quality of state/observation information.

* CentralizedDIFF: This is a centralized approach at base station $b$, without split learning, with a different state definition than SNOW. Basically, in this case, we consider a realistic centralized agent, without access to sensitive data. Hence, we define the partial state of the central agent $b$ as $o_b^t = \{|U_b^t|, l_{i,b}^t, |i \in \mathcal{I}\}$, which represents the total user count connected to base station $b$, and the total load demand per slice instance at time $t$. This aggregated state information is normally available at the central node, without exposing any privacy problem.

In our study, all the models are implemented using Python and trained Linux servers on a high-performance national computation platform. The detailed parameters of the DNN models used in SNOW are also summarized in Table I.

## C. Results Discussion

For better visibility, all the line curves are smoothed by averaging within a rolling window of 100 iterations.

**Comparison with decentralized approaches:** Figure 3 reveals the trend of the global rewards for RL-based approaches: SNOW and AllActive. We notice that the global rewards of SNOW and AllActive are somewhat similar, and they both show good convergence properties. How-
<figure><img src="image_5.png" /><figcaption>Fig. 4: QoS and energy comparisons with decentralized approaches for the 5 base stations.</figcaption></figure>

<table><caption>TABLE I: LIST OF PARAMETERS</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Number of users per base station</td><td>[6-10]</td></tr><tr><td>Guaranteed delay δ<sub>i</sub> [TCS, CCS, LSCS]</td><td>[5,3,10] ms</td></tr><tr><td>Allotted bandwidth BW<sub>i,b</sub> [TCS, CCS, LSCS,Eco]</td><td>[25,35,20,10] MHz</td></tr><tr><td>P<sub>b</sub><sup>static</sup></td><td>18 Watts</td></tr><tr><td>P<sub>b</sub><sup>fixed</sup></td><td>139 Watts</td></tr><tr><td>P<sub>b</sub><sup>load</sup></td><td>742 Watts</td></tr><tr><td>AGWN N<sub>0</sub></td><td>-174 dBm</td></tr><tr><td>Transmit Power P<sub>b,u</sub></td><td>75 dBm</td></tr><tr><td>ψ<sub>i</sub> [TCS, CCS, LSCS, Eco]</td><td>[1.5, 1.8, 1.4, 0.8]</td></tr><tr><td>HO<sub>b</sub><sup>slice</sup></td><td>0.00005</td></tr><tr><td>HO<sub>b</sub><sup>fixed</sup></td><td>0.0001</td></tr><tr><td>HO<sub>b</sub><sup>load</sup></td><td>0.0002</td></tr><tr><td>QoS impact factor β</td><td>5</td></tr><tr><td>Exploration rate ε</td><td>0.1</td></tr><tr><td>Learning rate η</td><td>0.01</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Fig. 5: Overall energy comparison with decentralized approaches.</figcaption></figure>

ever, a complete comparison of their performance requires evaluating how well each approach meets all the targeted performance metrics (i.e. QoS and energy consumption).

For this reason, we first compare the different decentralized approaches: SNOW, AllActive, AllActive (no agent) and Random, in terms of QoS and energy consumption for individual base stations, as depicted in Figure 4. Seen

<figure><img src="image_7.png" /><figcaption>Fig. 6: Global rewards comparison with centralized approaches.</figcaption></figure>

through the lens of QoS in Figure 4a, both AllActive and AllActive (no agent) sustain a high level of QoS for all the base stations. Of course, this is expected, as all the slice instances are always active in these approaches. Interestingly, SNOW also maintains a similar level of QoS for all the scenarios despite dynamically activating and deactivating slice instances. However, we can see that the QoS of AllActive (no agent) is slightly better than SNOW and AllActive. It is worth explaining that in AllActive (no agent) the users remain continuously connected to their requested slice without any energy-saving measures. The only cost incurred is the handover cost between base stations. However, it all comes at the expense of increasing energy consumption (as we will see in Figure 5 shortly). As an insight, in the SNOW and AllActive solutions, various handover costs may arise, including handovers between slices, between base stations, and between both slices and base stations. This is the price we pay for finding the trade-off between QoS and energy. The QoS of Random is nowhere closer to its peers across
<figure><img src="image_8.png" /><figcaption>Fig. 7: QoS and energy comparisons with centralized approaches for base station 3 and base station 4.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 8: Overall energy comparison with centralized approaches.</figcaption></figure>

all base stations.

In Figure 4b, the energy consumption trends for base stations 1, 3, and 5 show that AllActive (no agent) has the highest energy consumption, while SNOW and AllActive are relatively lower and similar. AllActive (no agent) leads to unbalanced loads, causing higher consumption at specific base stations. At base station 4, AllActive (no agent) consumes less energy compared to SNOW and AllActive. At base station 2, SNOW performs well, maintaining low energy consumption and comparable QoS to AllActive and AllActive (no agent). Overall, SNOW appears to maintain moderate energy consumption levels across all base stations.

Note that SNOW takes a collaborative approach (i.e. it targets to optimize overall network efficiency), meaning that the objective is the overall energy consumption of all the base stations, shown in Figure 5, rather than focusing on individual base stations. Jointly considering the user association and slice activation/deactivation, SNOW achieves 40% energy improvement over AllActive (no agent). Although all slice instances are active in both AllActive and AllActive (no agent), AllActive directs users to the optimal slice instance of the base station with energy saving in mind, ultimately achieving 22% better energy utilization than AllActive (no agent). The Random approach deactivates some slice instances without considering QoS measures, and thus it sometimes happens to consume less energy compared to its peers.

Given these results, we outline that our proposed approach,

SNOW, ensures a more balanced load distribution and results in better energy use. This intelligent management reduces unnecessary energy expenditure while ensuring network performance. The comparison with the AllActive schemes shows that this improvement comes from two different elements: i) deactivating slices whenever this action does not have a significant impact on QoS, and ii) associating users with the most appropriate base station and slice instance, without accounting for only signal quality.

**Comparison with centralized approaches:** To further sharpen the benefits of SNOW, we hereby compare it with centralized approaches. Here, we conceive to shed light on the impact of the quality of the state and observation information used by the different techniques. We can see in Figure 6 that SNOW, CentralizedSP and CentralizedDIFF agents can optimize their actions to maximize global rewards and convergence time are more or less a few hundred epochs different from each other, SNOW being the fastest one.

As in the prior section, we evaluate the performance of the agents by looking at each factor (i.e. QoS and energy consumption) of the reward function separately, to reveal whether the agents have indeed learned to optimize both factors. In this part, we only show results for two base stations (out of 5), selecting those with the most different behaviors, which are also representative for the other base stations. Figure 7 depicts the QoS and energy comparisons between SNOW and centralized approaches for base stations 3 and 4. As we can see in the right two plots of Figure 7, SNOW, CentralizedSP and CentralizedDIFF achieve the required user QoS, meeting the delay requirements of all the users in both base stations. For the energy consumption comparisons, same as in Figure 4b, each agent acts differently for each base station, as shown in the left two plots of Figure 7. SNOW consumes more energy than both centralized approaches for base station 3, while the highest energy consumption comes from CentralizedDFF in base station 4. Going beyond these examples, we look into the overall energy consumption of all 5 base stations, as shown in Figure 8, where all the agents are trained by one global reward. In this case, SNOW consumes the lowest energy, and CentralizedSP outperforms CentralizedDIFF. This highlights the advantages of using sensitive user data in the defined state/observation information, as discussed in Section IV. This sensitive information clearly allows the agent to learn
more effectively, but it can hardly be shared in operational
settings without great risks for the users (in general), and for
the mission (in the specific case of tactical networks).

VII. CONCLUSION

Our research demonstrates that energy-efficient network slicing is pivotal for enhancing the operational efficiency of tactical networks, and, more generally, in environments with stringent power constraints. By addressing the complex problem of joint slice activation, deactivation, and user association, we design a novel ML-based solution that balances energy consumption and QoS. Most importantly, our proposed SNOW framework, the first to integrate split learning and RL, effectively utilizes network resources and navigates users by collectively training partial DNNs without sharing sensitive user data. Extensive simulations validate that SNOW outperforms existing benchmarks, achieving superior energy efficiency while ensuring a high QoS level. Last but not least, it is fair to say that SNOW offers energy efficiency, the required network performance, and enhanced privacy in resource-constrained and adversarial environments.

REFERENCES

[1] H. P. Phyu, D. Naboulsi, and R. Stanica, "Machine Learning in Network Slicing - A Survey," *IEEE Access*, vol. 11, no. April, pp. 39 123–39 153, 2023.

[2] A. Castanares, D. K. Tosh, and C. A. Kamhoua, "Slice Aware Framework for Intelligent and Reconfigurable Battlefield Networks," *Proceedings - IEEE Military Communications Conference MILCOM*, vol. 2021-Novem, pp. 489–494, 2021.

[3] O. Gupta and R. Raskar, “Distributed learning of deep neural network over multiple agents,” 2018. [Online]. Available: https://arxiv.org/abs/1810.06060

[4] M. E. Bruni, S. Khodaparasti, and G. Perboli, “Energy Efficient UAV-Based Last-Mile Delivery: A Tactical-Operational Model With Shared Depots and Non-Linear Energy Consumption,” *IEEE Access*, vol. 11, no. February, pp. 18 560–18 570, 2023.

[5] J. S. Lee, Y. S. Yoo, H. S. Choi, T. Kim, and J. K. Choi, "Energy-Efficient TDMA Scheduling for UVS Tactical MANET," *IEEE Communications Letters*, vol. 23, no. 11, pp. 2126-2129, 2019.

[6] S. Kim, “Cooperative Federated Learning-Based Task Offloading Scheme for Tactical Edge Networks,” *IEEE Access*, vol. 9, pp. 145 739–145 747, 2021.

[7] X. Lin, A. Liu, C. Han, X. Liang, K. Pan, and Z. Gao, “LEO Satellite and UAVs Assisted Mobile Edge Computing for Tactical Ad-Hoc Network: A Game Theory Approach,” *IEEE Internet of Things Journal*, vol. PP, p. 1, 2023.

[8] F. Yang, C. Comaniciu, P. Krishnan, K. Sabnani, and L. Greenwald, "A Game Theoretic Framework for Distributed Mission Slice Allocation and Management for Tactical Networks," *Journal of Network and Systems Management*, vol. 31, no. 1, pp. 1–28, 2023.
[Online]. Available: https://doi.org/10.1007/s10922-022-09709-w

[9] P. Vepakomma, "Split learning for health : Distributed deep learning without sharing raw patient data," 2018.

[10] Z. Li, C. Yan, X. Zhang, G. Gharibi, Z. Yin, X. Jiang, and B. A. Malin, "Split Learning for Distributed Collaborative Training of Deep Learning Models in Health Informatics," *AMIA ... Annual Symposium proceedings. AMIA Symposium*, vol. 2023, pp. 1047–1056, 2023.

[11] Y. Gao, M. Kim, S. Abuadbba, Y. Kim, C. Thapa, K. Kim, S. A. Camtep, H. Kim, and S. Nepal, "End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things," *Proceedings of the IEEE Symposium on Reliable Distributed Systems*, vol. 2020-Septe, pp. 91–100, 2020.

[12] P. Li, C. Guo, Y. Xing, Y. Shi, L. Feng, and F. Zhou, "Core network traffic prediction based on vertical federated learning and split learning," *Scientific Reports*, vol. 14, no. 1, pp. 1–11, 2024.
[Online]. Available: https://doi.org/10.1038/s41598-024-53193-y

[13] S. Tuli, "SplitPlace : Intelligent Placement of Split Neural Nets in Mobile Edge Environments."

[14] T. A. Khoa, D.-v. Nguyen, M.-s. Dao, and K. Zettsu, "splitDyn : Federated Split Neural Network for Distributed Edge AI Applications," *2022 IEEE International Conference on Big Data (Big Data)*, pp. 6066–6073, 2022.

[15] Z. Lin, G. Qu, X. Chen, and K. Huang, "Split learning in 6g edge networks," 2024. [Online]. Available: https://arxiv.org/abs/2306.12194

[16] J. Mei, X. Wang, K. Zheng, G. Boudreau, A. B. Sediq, and H. Abou-Zeid, "Intelligent Radio Access Network Slicing for Service Provisioning in 6G: A Hierarchical Deep Reinforcement Learning Approach," *IEEE Transactions on Communications*, vol. 69, no. 9, pp. 6063–6078, 2021.

[17] H. P. Phyu, D. Naboulsi, R. Stanica, and G. Poitau, "Towards energy efficiency in ran network slicing," in *2023 IEEE 48th Conference on Local Computer Networks (LCN)*, 2023, pp. 1–9.

[18] M. T. Spaan, "Partially observable markov decision processes," *Adaptation, Learning, and Optimization*, vol. 12, pp. 387–414, 2012.

[19] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, "Learning differentially private language models without losing accuracy," CoRR, vol. abs/1710.06963, 2017. [Online]. Available: http://arxiv.org/abs/1710.06963

[20] Z. Han, B. Liu, S.-B. Lin, and D.-X. Zhou, "Deep Convolutional Neural Networks with Zero-Padding: Feature Extraction and Learning," *arXiv preprint arXiv:2307.16203*, pp. 1–18, 2023.