# GNN-Based Multi-Agent DRL for Energy-Efficient Multi-Domain 6G Resource Orchestration

Abdelmounaim Bouroudi, Abdelkader Outtagarts, Yassine Hadjadj-Aoul

► To cite this version:

Abdelmounaim Bouroudi, Abdelkader Outtagarts, Yassine Hadjadj-Aoul. GNN-Based Multi-Agent DRL for Energy-Efficient Multi-Domain 6G Resource Orchestration. GLOBECOM 2025 - Annual IEEE Global Communications Conference, IEEE, Dec 2025, Taipei, Taiwan. hal-05455207

HAL Id: hal-05455207

https://inria.hal.science/hal-05455207v1

Submitted on 13 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# GNN-Based Multi-Agent DRL for Energy-Efficient Multi-Domain 6G Resource Orchestration

Abdelmounaim Bouroudi

Bell Labs

Nokia Networks France

Paris, France

abdelmounaim.bouroudi@gmail.com

Abdelkader Outtagarts

Bell Labs

Nokia Networks France

Paris, France

abdelkader.outtagarts@nokia-bell-labs.com

Yassine Hadjadj-Aoul

INRIA

Univ Rennes, CNRS, IRISA

Rennes, France

yassine.hadjadj-aoul@irisa.fr

**Abstract**—The rapid evolution towards 6G networks introduces new challenges in orchestrating services across distributed domains while ensuring sustainability goals, such as energy efficiency. Traditional scaling strategies focus on the number of network function instances without optimizing their placement based on energy consumption or resource usage. Addressing this gap, we propose a distributed and energy-efficient placement framework for scaled Network Function (NF) instances across multi-domain 6G infrastructures. Building upon a refined energy consumption model that accounts for computational and network-level power usage, we design a Graph Neural Network (GNN)-enhanced Deep Reinforcement Learning (DRL) agent to optimize placement decisions. The agent encodes the substrate topology and resource states to guide the selection of energy-efficient nodes during scaling operations. We implement and evaluate the framework in a realistic multi-domain scenario featuring fluctuating traffic patterns and heterogeneous node energy profiles. Results show that our approach reduces infrastructure energy consumption compared to a round-robin heuristic, while maintaining high placement success and efficient resource utilization. Among the DRL methods explored, Proximal Policy Optimization (PPO) achieved the best trade-off between placement stability, adaptability, and energy performance. These findings demonstrate the potential of GNN-based DRL agents for sustainable orchestration in future 6G networks.

**Index Terms**—6G networks, Multi-Domain Orchestration, AI/ML, DRL, GNN, NF Placement, Energy Efficiency, Sustainability.

## I. INTRODUCTION

The evolution of mobile communication systems towards beyond-5G (B5G) and 6G networks brings unprecedented demands for scalability, flexibility, and sustainability. Cloud-native architectures, end-to-end network slicing, and distributed edge computing are now central to network design, requiring efficient orchestration across multiple administrative domains [1]. However, the rapid growth in traffic and the deployment of dense network infrastructures have made energy consumption a critical bottleneck, impacting both operational expenditure (OPEX) and environmental objectives, such as carbon neutrality [2].

While horizontal scaling strategies, often driven by AI-based traffic forecasting or Deep Reinforcement Learning (DRL) methods, can adapt the number of Network Function (NF) instances to dynamic traffic variations [3], [4], these approaches typically overlook the placement problem, i.e.,

deciding where to deploy scaled instances to minimize energy consumption and optimize resource utilization. In multi-domain environments, this placement challenge is further complicated by partial observability, heterogeneous hardware profiles, and conflicting local policies [5]. Without intelligent placement strategies, the energy and resource savings achieved by scaling can be rapidly lost.

Recent advances in Artificial Intelligence (AI) and Machine Learning (ML) techniques, particularly the combination of GNN with DRL, have shown promise for resource allocation problems in networking [6], [7]. GNNs provide efficient representations of complex topologies, while DRL offers adaptive decision-making capabilities. Nevertheless, most existing works focus either on single-domain orchestration or on resource optimization without explicitly considering energy consumption as a key optimization objective [8], [9]. Moreover, scalable, energy-aware solutions for cross-domain placement remain largely underexplored.

In this context, we propose a distributed, energy-efficient placement framework for scaled NF instances in multi-domain 6G networks. This work follows a Nokia Unified Networking Experience (UNEXT) architecture [10] for a multi-domain end-to-end system scenario, providing a realistic baseline for orchestration in cloud-native environments. It extends our previous contribution presented in [11], where we developed a distributed scaling and orchestration framework. Specifically, our proposed solution is structured around four main components. First, we design a refined energy consumption model that accurately represents both compute nodes and networking elements at the domain level, capturing both static and dynamic power behaviors, inspired by work in [12]. Second, we introduce a GNN-based encoder to model the substrate topology and resource states, enabling a scalable and efficient representation of large multi-domain infrastructures. Third, we develop a DRL-based placement agent, enhanced with GNN embeddings, that learns energy-aware placement strategies favoring both resource utilization and placement success. We explore three DRL algorithms in this context: Deep Q-Learning (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO). Finally, we implement and evaluate the complete framework in an extended version of the OMNeT++-based multi-domain
orchestration testbed introduced in [13]. Evaluation results under a realistic multi-domain scenario with heterogeneous node energy profiles demonstrate that the proposed framework significantly reduces infrastructure energy consumption compared to a round-robin heuristic, while maintaining high placement success rates and efficient resource utilization.

The remainder of this paper is organized as follows. Section II presents an overview of the related work. Section III presents the system model, including the energy consumption model and the network representation. Section IV details the proposed GNN-enhanced DRL placement architecture and the proposed placement algorithms. Section V presents the experimental evaluation and discussion. Finally, Section VI concludes the paper.

## II. RELATED WORK

Several research efforts have addressed energy-efficient resource allocation and orchestration in cloud and network environments. In [12], the authors proposed an energy model for fog and edge computing environments, capturing static and dynamic power usage in compute and networking elements. However, this model was not originally designed for orchestrated service placement decisions in dynamic 6G networks.

Multi-domain orchestration has been explored in [5], where resource management across administrative domains is addressed through centralized optimization methods. Although effective for simple scenarios, centralized approaches struggle with scalability and partial observability, critical issues in distributed 6G infrastructures. Similarly, resource allocation frameworks such as [14] and [15] emphasize the importance of optimizing slice placement and scaling but lack explicit integration of energy-aware placement strategies.

AI-based techniques have been increasingly proposed to enhance orchestration automation. In [3] and [4], DRL was applied to VNF placement and resource scheduling, achieving adaptability to network dynamics. However, these works focus mainly on service quality optimization without prioritizing energy consumption. Recent studies like [8] and [9] specifically target energy efficiency through DRL for power control and resource allocation, but are limited to single-domain or wireless-only contexts.

Graph Neural Networks have emerged as powerful tools for topology modeling [6]. Works like [7] and [16] leverage GNNs to enhance resource allocation or virtual network embedding tasks. Nevertheless, these solutions typically assume full network visibility and do not tackle the decentralized, multi-domain orchestration problem with energy constraints. In [17], a federated learning-based orchestration was proposed for energy efficiency, but without addressing NF placement across multiple domains. Overall, while DRL and GNNs have demonstrated strong potential individually, an integrated framework that jointly models multi-domain topologies, optimizes energy consumption, and adapts placement decisions using DRL remains largely unexplored.

## III. DISTRIBUTED ENERGY-AWARE PLACEMENT FRAMEWORK

In this section, we describe our proposed framework for energy-efficient NF placement across multiple domains in a 6G environment, along with the energy consumption model used to guide placement decisions.

### A. Distributed Placement Architecture

The proposed system is based on a distributed orchestration architecture designed for energy-efficient Network Function (NF) placement across multiple domains. As illustrated in Figure 1, each domain $K := \{1, \dots, K\}$ independently hosts a set of compute nodes and interconnecting routers, and manages a collection of slices $s := \{1, \dots, S\}$. Each slice consists of a chain of NFs deployed over one or multiple domains. For each slice in the domain, local Scaling Agents monitor slice-level KPIs, such as CPU utilization, and dynamically decide how many NF instances to add or remove. These scaling decisions are forwarded to the slice Placement Agent, which determines the optimal compute nodes for deploying the new instances, while considering both resource availability and energy consumption objectives.

Inter-domain coordination is achieved through a lightweight information exchange protocol, where scaling agents share aggregated slice metrics (like CPU usage, memory, throughput, etc) to enhance the global decision-making process without requiring full topology disclosure. Once the scaling and placement decisions are generated by each domain's agents, they are executed independently on the corresponding slice clusters. This distributed and modular design enables scalability, resilience, and energy-aware orchestration across heterogeneous and partially observable network domains.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Multi-domain orchestration framework with per-domain scaling and placement agents.</figcaption></figure>

### B. Energy Consumption Model

To enable energy-aware orchestration, we implement a comprehensive Energy Consumption (EC) model adapted from [12]. This model accounts for both compute nodes and network elements (switches or routers). For a compute node $n_i \in N_K$ over a period $T$, the EC is given by:
$$EC_{\text{total}}^{n_i} = (P_{\text{static}}^{n_i} + P_{\text{dyn}}^{n_i}) \times T \quad (1)$$

where $P_{\text{static}}^{n_i}$ is the static idle consumption of the node and $P_{\text{dyn}}^{n_i}$ is the dynamic consumption.

The dynamic part is given by:

$$P_{\text{dyn}}^{n_i} = P_{\text{epc}}^{n_i}(U_{\text{cpu}}^{n_i} + U_{\text{H}}^{n_i}) + P_{\text{nf-c}}^{n_i}|nf_{\text{created}}| \quad (2)$$

where $P_{\text{epc}}^{n_i}$ is the energy per CPU cycle, $U_{\text{cpu}}^{n_i}$ is the NF CPU load, $U_{\text{H}}^{n_i}$ is the hypervisor overhead, $P_{\text{nf-c}}^{n_i}$ is the energy for NF creation, and $|nf_{\text{created}}|$ is the number of NFs created during T. Similarly, the energy consumption of a network element $s_j \in \mathcal{S}_K$ is expressed as:

$$EC_{\text{total}}^{s_j} = (P_{\text{static}}^{s_j} + P_{\text{dyn}}^{s_j}) \times T \quad (3)$$

where $P_{\text{static}}^{s_j}$ is the static switch power, and the dynamic component is:

$$P_{\text{dyn}}^{s_j} = P_{\text{pkt}}^{s_j} N_{\text{pkt}}^{s_j} + P_{\text{sf}}^{s_j} N_{\text{pkt}}^{s_j} L_{\text{pkt}} \quad (4)$$

where $P_{\text{pkt}}^{s_j}$ is the per-packet processing energy, $N_{\text{pkt}}^{s_j}$ the number of processed packets, $P_{\text{sf}}^{s_j}$ the per-byte forwarding cost, and $L_{\text{pkt}}$ the average packet size. At the domain level, the total energy consumption is aggregated as:

$$E_{\text{total}}^{K} = P U E_{K} \left( \sum_{n_i \in \mathcal{N}_{K}} E_{\text{total}}^{n_i} + \sum_{s_j \in \mathcal{S}_{K}} E_{\text{total}}^{s_j} \right) \quad (5)$$

where $P U E_K$ is the Power Usage Effectiveness representing the cooling and overhead factor for domain K. This energy modeling framework enables the placement agent to prioritize decisions that minimize domain-wide energy expenditure while respecting resource and performance constraints.

## IV. GNN-ENHANCED DRL-BASED PLACEMENT AGENT

### A. GNN-Enhanced DRL Placement Agent

<figure><img src="image_3.png" /><figcaption>Fig. 2: GNN-Enhanced DRL Placement Agent Architecture.</figcaption></figure>

Building upon the energy modeling, we design a DRL-based placement agent capable of optimizing NF placement decisions with energy awareness. The overall architecture is depicted in Figure 2. It consists of two main components: a Graph Neural Network (GNN) encoder and a Deep Reinforcement Learning (DRL) policy. The substrate network is modeled as a graph $G = (\mathcal{V}, \mathcal{E})$, where each node $v_i \in \mathcal{V}$ encodes its resources, energy parameters, and consumption

values. The GNN extracts the embeddings $h_i$ of the substrate cluster nodes, which are aggregated into a global state $z$ via mean pooling. The DRL agent receives as input the state $s_t = (z, \mathcal{R}_t)$, where $\mathcal{R}_t$ contains the scaling request, and outputs a placement action $a_t$ aimed at minimizing energy consumption while satisfying resource requirements.

### B. DRL Agent Definition

To apply DRL to the placement problem, we formulate it as a Markov Decision Process (MDP) by explicitly defining the state, action, and reward:

**State:** At time $t$, the state $s_t$ is:

$$s_t = (z_t, \mathcal{R}_t) \quad (6)$$

where $z_t$ is the GNN-encoded substrate graph embedding and $\mathcal{R}_t$ is the placement request, including the number of instances and resource demands.

**Action:** The action $a_t$ is a vector:

$$a_t = [a_t^{(1)}, \dots, a_t^{(N)}] \quad \text{with} \quad a_t^{(i)} \in [0, \phi_{\max}] \quad (7)$$

where $a_t^{(i)}$ indicates the number of instances placed on node $i$ and $\phi_{\max}$ represents the maximum instances that can be placed per node. The action vector must satisfy the constraint that the total assigned instances equals the number requested, ensuring resource feasibility.

**Reward:** The global reward $R_t$ is defined as:

$$R_t = \begin{cases} R_{\text{eff}}, & \text{valid placement} \\ R_{\text{penal}}, & \text{violation} \end{cases} \quad (8)$$

The efficiency reward $R_{\text{eff}}$ aggregates resource efficiency, energy savings, and placement success:

$$R_{\text{eff}} = \beta_1 R_{\text{rue}} + \beta_2 R_{\text{ec}} + \beta_3 R_{\text{placement}} \quad (9)$$

where:

$$R_{\text{rue}} = \frac{1}{N} \sum_{i=1}^{N} \left( 1 - \frac{\text{free\_cpu}_i}{\text{max\_cpu}_i} \right), \quad (10)$$

$$R_{\text{ec}} = - \min \left( \frac{EC_{\text{cluster}}}{EC_{\text{max}}}, 1 \right), \quad (11)$$

$$R_{\text{placement}} = \frac{|\phi_{\text{placed}}|}{|\phi_{\text{req}}|}. \quad (12)$$

Here, $R_{\text{rue}}$ promotes efficient CPU resource utilization, $R_{\text{ec}}$ minimizes energy consumption normalized by the cluster maximum, and $R_{\text{placement}}$ ensures successful completion of placement requests.

The penalty reward $R_{\text{penal}}$ captures two types of violations: CPU oversubscription and over-provisioning (i.e., placing more instances than requested).

**Let:**

* $\phi_{\text{excess}} = \max(0, |\phi_{\text{placed}}| - |\phi_{\text{req}}|)$: the number of excess instances placed beyond the request,

* $\delta_{\text{cpu}}$: indicator function equal to 1 if any node exceeds its maximum CPU capacity, and 0 otherwise,

* $\delta_{\text{over}}$: an indicator function equal to 1 if more instances are placed than requested, i.e., $|\phi_{\text{placed}}| > |\phi_{\text{req}}|$,
Then:

$$R_{\text{penal}} = -\delta_{\text{cpu}} - \delta_{\text{over}} \left( \frac{\phi_{\text{excess}}}{|\phi_{\text{req}}|} + \epsilon \right) \quad (13)$$

where $\epsilon$ is a small constant to avoid division by zero.

### C. Placement Algorithms

Building on the DRL agent definition, we evaluate three algorithms for energy-aware placement: Deep Q-Learning (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO). Each uses the same GNN encoder to extract state embeddings, and the corresponding loss is back-propagated through the entire architecture to enable end-to-end learning.

1) **DQN:** DQN is a value-based method where the agent learns a Q-function $Q(s, a)$ that estimates the expected return from taking action $a$ in state $s$: $Q(s, a) = E[\sum_{k=0}^{\infty} \gamma^k r_{t+k}]$. To reduce overestimation, we adopt the Double DQN variant with two networks: an online actor $\theta$ and a target network $\theta^-$. The target is computed as:

$$y_t = r_t + \gamma Q(s_{t+1}, \arg\max_{a'} Q(s_{t+1}, a'; \theta); \theta^-) \quad (14)$$

and the minimized loss is: $\mathcal{L}_{\text{DQN}} = E[(y_t - Q(s_t, a_t))^2]$. The action space is discrete, where each element represents the number of instances allocated to a node, and we vary the per-node upper bound $\phi_{\max} \in \{1, 5, 10\}$ to analyze action space dimensionality effects.

2) **DDPG:** To overcome DQN's limitations in large discrete spaces, we adopt DDPG, an off-policy continuous control algorithm. It learns a deterministic policy $\mu_{\theta}(s)$ and a critic $Q_{\phi}(s, a)$. The critic minimizes:

$$\mathcal{L}_{\text{critic}} = E[(Q_{\phi}(s_t, a_t) - y_t)^2], \quad (15)$$

$$y_t = r_t + \gamma Q_{\phi'}(s_{t+1}, \mu_{\theta'}(s_{t+1})), \quad (16)$$

while the actor minimizes:

$$\mathcal{L}_{\text{actor}} = -E[Q_{\phi}(s_t, \mu_{\theta}(s_t))] \quad (17)$$

Since placement requires discrete allocations, we apply a rounding heuristic based on softmax probabilities to convert the continuous action to valid discrete action:

#### Algorithm 1: DDPG Placement Heuristic

**Input:** State embedding $s$, instances $K$, actor $\mu_{\theta}$

1. $\text{logits} \leftarrow \mu_{\theta}(s) + \mathcal{N}(0, \sigma)$;

2. $\pi(a|s) \leftarrow \text{Softmax}(\text{logits})$;

3. $\text{allocation} \leftarrow \text{Round}(\pi(a|s) \cdot K)$;

**Output:** Discrete allocation vector $a$

3) **PPO:** While DDPG handles continuous outputs, it lacks robustness in high-dimensional discrete spaces. PPO improves learning stability using a clipped surrogate objective:

$$\mathcal{L}^{\text{PPO}} = E \left[ \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1 \pm \epsilon) \hat{A}_t \right) \right] \quad (18)$$

where $r_t(\theta) = \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta}^{\text{kl}}(a_t|s_t)}$ and $\hat{A}_t$ is the estimated advantage. The value network is optimized via

$$\mathcal{L}^{\text{value}} = \frac{1}{2}(V(s_t) - R_t)^2 \quad (19)$$

where $V(s_t)$ denotes the critic's estimate of the expected discounted return starting from state $s_t$. To match PPO's continuous action outputs with our discrete placement space, we reuse the softmax-rounding heuristic:

#### Algorithm 2: PPO Placement Heuristic

**Input:** State embedding $s$, instances $K$, actor $\mu_{\theta}$

1. $\mu, \sigma \leftarrow \mu_{\theta}(s)$;

2. $a \sim \mathcal{N}(\mu, \sigma)$;

3. $\pi(a|s) \leftarrow \text{Softmax}(a)$;

4. $\text{allocation} \leftarrow \text{Round}(\pi(a|s) \cdot K)$;

**Output:** Discrete allocation vector $a$

This ensures compatibility with our environment while preserving PPO's exploration flexibility.

### V. PERFORMANCE EVALUATION

To assess the effectiveness of our proposed placement strategies, we consider a realistic multi-domain orchestration scenario reflecting an end-to-end B5G architecture. The scenario comprises two domains: a Radio Access Network (RAN) and a Core Network (CN), interconnected via dedicated links. Traffic follows a weekly smart city pattern, where usage peaks during daytime and drops at night, with a higher peak during the weekend. The deployed slice spans both domains, with two NFs in the RAN (CU and DU) and one NF in the Core (UPF). Scaling events are mainly triggered at the Core level, where the UPF must dynamically adapt to traffic load. Traffic dynamics are captured through an LSTM-based forecasting module that predicts future data rates in the RAN domain. These predictions are sent to the Core domain via the inter-domain messaging bus, allowing proactive scaling decisions. The underlying testbed is based on OMNeT++ and implements the distributed orchestration framework described above.

The Core domain cluster consists of six heterogeneous nodes, each with distinct compute capacities and energy profiles. Nodes N11–N13 are modeled after energy-efficient Taurus servers, while N14–N16 reflect higher-capacity but more energy-hungry Parasilo servers. The energy-related parameters were calibrated based on empirical measurements reported in [12] and are summarized in Table I.

<table><caption>TABLE I: CPU and Energy Parameters for the Core Domain Nodes.</caption><thead><tr><th>Node</th><th>cpu<sup>n<sub>i</sub></sup></th><th>P<sup>n<sub>i</sub></sup><sub>static</sub> (W)</th><th>P<sup>s<sub>i</sub></sup><sub>static</sub> (W)</th><th>epc (J/cycle)</th><th>P<sup>s<sub>i</sub></sup><sub>pkt</sub> 10<sup>-3</sup> (J/pkt)</th><th>P<sup>s<sub>i</sub></sup><sub>sf</sub> (W)</th><th>U<sup>n<sub>i</sub></sup><sub>H</sub> (%)</th></tr></thead><tbody><tr><td>N11-N13</td><td>5000.0</td><td>80.0</td><td>4.0</td><td>0.01</td><td>1.2</td><td>0.1</td><td>0.1</td></tr><tr><td>N12</td><td>3000.0</td><td>80.0</td><td>4.0</td><td>0.01</td><td>1.2</td><td>0.1</td><td>0.1</td></tr><tr><td>N14-N16</td><td>5000.0</td><td>120.0</td><td>4.0</td><td>0.5</td><td>1.2</td><td>0.1</td><td>0.1</td></tr><tr><td>N15</td><td>3000.0</td><td>120.0</td><td>4.0</td><td>0.5</td><td>1.2</td><td>0.1</td><td>0.1</td></tr></tbody></table>

Static node power consumption ($P_{\text{static}}^{n_i}$) and energy per CPU cycle ($epc$) differentiate efficient and less efficient nodes. The first three nodes (N11-N13) are more energy efficient, while the last three consume more energy. CPU resources are varied to prevent selection being based on a single parameter. In addition, we consider network elements consumption as
homogeneous across the cluster. The PUE factor is set to 1.5, reflecting typical compute-heavy data center conditions [18]. A good placement strategy should prioritize energy-efficient nodes first to minimize overall infrastructure consumption without sacrificing resource availability. The performance of the placement strategies is evaluated along the three key dimensions of the reward equations (9) with $\beta_{1,2,3} = 1/3$.

Figure 3a shows the evolution of the global reward for the five evaluated agents. PPO rapidly reaches a stable and high reward close to 0.38, outperforming all other agents. DDPG converges more slowly and stabilizes around 0.13, showing improvements over the DQN variants. Among DQN agents, DQN-1 (smallest action space) achieves the best performance, reaching around 0.2, while DQN-5 and DQN-10 remain lower, struggling around 0.1 and 0.08, respectively. These results highlight the difficulty DQN faces when the action space grows, whereas continuous control (DDPG, PPO) significantly enhances learning feasibility and reward quality. These results can be explained by the individual reward components.

Figure 3b presents the placement success reward across agents. PPO achieves almost 100% placement success within the first 100 epochs and maintains it consistently, whereas DDPG converges more slowly and plateaus around 80% success. DQN-1 adapts better to placement constraints compared to DQN-5 and DQN-10, but even with a small action space, DQN remains inferior to PPO. For larger DQN action spaces (DQN-5, DQN-10), placement errors increase, significantly harming their performance during early training stages.

Figure 3c compares the Resource Utilization Efficiency (RUE) reward. DDPG slightly surpasses PPO after 80 epochs, reflecting its tendency to compact instances more tightly across fewer nodes, thus improving node CPU usage. Among DQN agents, DQN-1 achieves the best RUE, confirming that a simpler action space facilitates more efficient resource usage. However, as action space size increases (DQN-5 and DQN-10), the agent struggles to improve RUE, staying at lower levels.

Finally, Figure 3d illustrates the energy efficiency reward. PPO consistently achieves the best energy savings, outperforming DDPG and all DQN variants. DDPG initially exhibits higher energy consumption due to exploration noise, but gradually improves with training. DQN-1 achieves moderate energy savings, whereas DQN-5 and DQN-10 suffer more, as their larger action spaces make it harder to target energy-efficient nodes early on. These results emphasize that continuous control strategies (particularly PPO) are better suited for sustainable, energy-efficient placement in complex orchestration scenarios.

Figure 4a and Figure 4b present the energy consumption comparison between the three DRL agents — DQN ($M=1$), DDPG, and PPO — and the Round-Robin (RR) heuristic over a five-week simulation. Figure 4a shows the energy consumed per placement step, measured in Joules. All DRL agents significantly outperform RR, achieving lower energy per decision. PPO consistently yields the best results, partic-

<figure><img src="image_4.png" /><figcaption>(a) Global reward comparison.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(b) Placement success rate comparison.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(c) RUE reward comparison.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(d) Energy efficiency reward comparison.</figcaption></figure>

<caption>Fig. 3: Training performance evolution of DQN ($M=1, 5, 10$), DDPG, and PPO agents across different reward components.</caption>

ularly under high-load conditions (weekends), demonstrating better generalization. DDPG and DQN also improve energy
<figure><img src="image_8.png" /><figcaption>(a) Energy consumption per placement step (Joules).</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(b) Total energy consumption per weekday (kWh).</figcaption></figure>

<caption>Fig. 4: Energy consumption comparison across five weeks for DRL agents and the heuristic baseline.</caption>

efficiency compared to RR, but DQN shows higher variability, especially during traffic peaks. In Figure 4b, we report the cumulative energy consumption per weekday, obtained by trapezoidal integration and expressed in kilowatt-hours (kWh). Again, all DRL methods show lower overall energy usage compared to RR. PPO achieves the best cumulative savings across the entire week. DDPG also outperforms DQN, especially under weekend loads. Notably, DQN's energy consumption exceeds RR during peak periods, suggesting its limited capacity to handle large scaling actions due to its restricted action space. Overall, these results demonstrate that PPO not only optimizes placement decisions per action but also maintains high energy efficiency over extended operational periods, highlighting the importance of policy-based approaches for sustainable network orchestration.

## VI. CONCLUSION

In this paper, we proposed an energy-efficient placement framework for multi-domain 6G networks based on GNN-enhanced Deep Reinforcement Learning (DRL). Building upon our previous work on distributed scaling, we introduced a realistic energy consumption model adapted from cloud-native environments to account for both static and dynamic power behaviors of compute nodes and network elements. We formulated the placement problem as a Markov Decision Process (MDP) and evaluated different DRL algorithms—DQN, DDPG, and PPO—augmented with heuristics for discrete action transformation. Experimental results demonstrated that all DRL agents significantly outperform a Round-Robin baseline in terms of energy consumption and resource utilization, with PPO achieving the best performance by combining high placement success with energy-aware decision-making. These

findings highlight the potential of combining DRL-based approaches with GNN state encoding to enable intelligent, sustainable orchestration in future 6G networks.

## REFERENCES

[1] T. Huang, W. Yang, J. Wu, J. Ma, X. Zhang, and D. Zhang, "A survey on green 6g network: Architecture and technologies," *IEEE access*, vol. 7, pp. 175 758-175 768, 2019.

[2] M. A. Uusitalo, P. Rugeland, M. R. Boldi, E. C. Strinati, P. Demestichas, M. Ericson, G. P. Fettweis, M. C. Filippou, A. Gati, M.-H. Hamon et al., "6g vision, value, use cases and technologies from european 6g flagship project hexa-x," *IEEE access*, vol. 9, pp. 160 004-160 020, 2021.

[3] P. T. A. Quang, A. Bradai, K. D. Singh, and Y. Hadjadj-Aoul, "Multi-domain non-cooperative vnf-fg embedding: A deep reinforcement learning approach," in *IEEE INFOCOM 2019-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)*. IEEE, 2019, pp. 886-891.

[4] D.-H. Luong, H.-T. Thieu, A. Outtagarts, and Y. Ghamri-Doudane, "Predictive autoscaling orchestration for cloud-native telecom microservices," in *2018 IEEE 5G World Forum (5GWF)*. IEEE, 2018, pp. 153-158.

[5] A. Bozorgchenani, D. Tarchi, and G. E. Corazza, "Centralized and distributed architectures for energy and delay efficient fog network-based edge computing services," *IEEE Transactions on Green Communications and Networking*, vol. 3, no. 1, pp. 250-263, 2018.

[6] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, "Graph neural networks: A review of methods and applications," *AI open*, vol. 1, pp. 57-81, 2020.

[7] Z. He, L. Wang, H. Ye, G. Y. Li, and B.-H. F. Juang, "Resource allocation based on graph neural networks in vehicular communications," in *GLOBECOM 2020-2020 IEEE Global Communications Conference*. IEEE, 2020, pp. 1-5.

[8] X. Zhang, H. Tian, W. Ni, Z. Yang, and M. Sun, "Deep reinforcement learning for energy efficiency maximization in swipt-based over-the-air federated learning," *IEEE Transactions on Green Communications and Networking*, vol. 8, no. 1, pp. 525-541, 2023.

[9] J. Peng, J. Zheng, L. Zhang, and M. Xiao, "Deep reinforcement learning for energy-efficient power control in heterogeneous networks," in *ICC 2022-IEEE International Conference on Communications*. IEEE, 2022, pp. 141-146.

[10] A. Sefidcon, C. Vulkan, and M. Gruber, "UNEXT: A Unified Network Working Experience," https://onestore.nokia.com/asset/213573, 2021, accessed: 2024-04-28.

[11] A. Bouroudi, A. Outtagarts, and Y. Hadjadj-Aoul, "A novel drl framework for cross-domain network scaling in 6g networks," in *2024 IEEE 25th International Conference on High Performance Switching and Routing (HPSR)*. IEEE, 2024, pp. 118-123.

[12] E. Ahvar, A.-C. Orgerie, and A. Lebre, "Estimating energy consumption of cloud, fog, and edge computing infrastructures," *IEEE Transactions on Sustainable Computing*, vol. 7, no. 2, pp. 277-288, 2019.

[13] A. Bouroudi, A. Outtagarts, and Y. Hadjadj-Aoul, "Multi-domain scaling algorithm with inter-orchestrator communication for beyond 5g/6g networks," in *2024 International Wireless Communications and Mobile Computing (IWCMC)*. IEEE, 2024, pp. 1054-1061.

[14] J. Lorincz, A. Kukuruzović, and Z. Blažević, "A comprehensive overview of network slicing for improving the energy efficiency of fifth-generation networks," *Sensors*, vol. 24, no. 10, p. 3242, 2024.

[15] D. chandra Bikkasani, "Ai-driven 5g network optimization: A comprehensive review of resource allocation, traffic management, and dynamic network slicing," *American Journal of Artificial Intelligence*, vol. 8, no. 2, pp. 55-62, 2024. [Online]. Available: https://doi.org/10.11648/j.ajai.20240802.14

[16] A. Rkhami, T. A. Q. Pham, Y. Hadjadj-Aoul, A. Outtagarts, and G. Rubino, "On the use of graph neural networks for virtual network embedding," in *2020 International Symposium on Networks, Computers and Communications (ISNCC)*. IEEE, 2020, pp. 1-6.

[17] H. Chergui, L. Blanco, L. A. Garrido, K. Ramantas, S. Kukliński, A. Ksentini, and C. Verikoukis, "Zero-touch ai-driven distributed management for energy-efficient 6g massive network slicing," *Ieee Network*, vol. 35, no. 6, pp. 43-49, 2022.

[18] "Google data centers efficiency," https://datacenters.google/efficiency/, accessed: March 2025.