# LSTM-Based Traffic Load Balancing and Resource Allocation for an Edge System

Authors: Thembelihle Dlamini; Sifiso Vilakati
Journal: Wireless Communications and Mobile Computing
 (2020)

---

## Abstract

The massive deployment of small cell Base Stations (SBSs) empowered with computing capabilities presents one of the most ingenious solutions adopted for 5G cellular networks towards meeting the foreseen data explosion and the ultralow latency demanded by mobile applications. This empowerment of SBSs withMulti-access Edge Computing (MEC) has emerged as a tentative solution to overcome the latency demands and bandwidth consumption required by mobile applications at the network edge. The MEC paradigm offers a limited amount of resources to support computation, thus mandating the use of intelligence mechanisms for resource allocation. The use of green energy for powering the network apparatuses (e.g., Base Stations (BSs), MEC servers) has attracted attention towards minimizing the carbon footprint and network operational costs. However, due to their high intermittency and unpredictability, the adoption of learning methods is a requisite. Towards intelligent edge system management, this paper proposes a Green-based Edge Network Management (GENM) algorithm, which is an online edge system management algorithm for enabling green-based load balancing in BSs and energy savings within the MEC server. The main goal is to minimize the overall energy consumption and guarantee the Quality of Service (QoS) within the network. To achieve this, the GENM algorithm performs dynamic management of BSs, autoscaling and reconfiguration of the computing resources, and on/off switching of the fast tunable laser drivers coupled with location-aware traffic scheduling in the MEC server. The obtained simulation results validate our analysis and demonstrate the superior performance of GENM compared to a benchmark algorithm.


---

## Body

## 1. Introduction

The foreseendense deployment of BS empowered with computing capabilities in order to meet the ultralow latency demanded by mobile users raises concerns related to energy consumption. Apart from the fact that BS energy costs account for a large part of the operating expenses of MN operators, there are also increasing concerns regarding their environmental impact in terms of high carbon dioxide (CO2) emissions. In an effort to minimize energy consumption and energy costs in 5G cellular networks within the MEC paradigm, this paper advocates for the integration of EH systems into network apparatuses and the use of container-based virtualization within computing platforms (i.e., MEC servers). The use of green energy mitigates the negative environmental impact of MN and enables cost-saving for mobile operators in terms of lowering operational energy costs. The motivation towards green energy is due to the fact that current trends in battery and solar module costs show a reduction in prices. The benefits of container-based virtualization is the reduction in energy drained in the computing platform due to their lower overheads when compared with VM [1, 2]. For a qualitative comparison of different virtualization techniques, interested readers are referred to [1].In this paper, a group of EH BS placed in proximity to a EH-MEC server is considered as anedge system. The EH-MEC server manages the BS system, deciding upon the allocation of transmission resources and also handling the computing and communication processes. In general, renewable energy systems are dimensioned to guarantee the autonomous operation of the edge system. Thus, it is desirable that the utilization of green energy be made one of the performance metrics when designing load-balancing strategies [3, 4], instead of the network impact [5, 6]. As a result, the green-based load balancing is a promising technique for optimizing MEC performance since it exploits the spatial diversity of the available green energy to reshape the network load among the BSs [4]. In this case, MN can prioritize the utilization of BS with sufficient green energy to serve more traffic while reducing the traffic loads for those BS consuming on-grid power. For instance, in the MEC server, a trade-off between the green energy utilization and the amount of workload that can be computed locally should be carefully evaluated.In this regard, it is worth noting that the energy consumption within the virtualized computing platform is due to (i) the active computing resources, i.e., VM or containers [7–9], and (ii) the network communications, communication related to transmission drivers [10, 11], and the intracommunications [12]. In order to alleviate this, this paper assumes that the container-based virtualization be deployed in the MEC server as containers are lightweight, i.e., demand less memory space, have shorter start-up time, and offer software portability. At each time instance, the containers are provisioned based on the forecasted server workloads, a technique referred to as autoscaling. In addition, the transmission drivers used for data transfers within the server are tuned by taking into account the mobile user’s location. The server is also capable of caching the frequently requested contents locally.In densely deployed BS, the energy drained is due to the always-on design approach [5, 6, 13], yet traffic load varies during the day, e.g., the demand is low during the night. Therefore, in order to intelligently manage the BS towards minimizing the energy consumption, the green-based load balancing technique is employed, i.e., BS sleep modes are enabled in some BS using green energy as a performance metric.


### 1.1. Paper Contributions

This paper considers an energy cost model that takes into account the computing, caching and communication processes within the MEC server, and transmission-related energy consumption in BS. Here, the GENM algorithm is proposed for enablinggreen-based traffic load balancing, i.e., the BS are dynamically switched on/off based on their harvested energy, autoscaling, and reconfiguring the computing resources, and the tuning of transmission drivers. This entails using a minimum number of optical drivers for real-time data transfers, over a short-term horizon. In order to solve the energy consumption minimization problem, the GENM algorithm performs online supervisory control, utilizing the learned traffic load and the harvested energy patterns. Then, the output is utilized within a LLC policy [14] to obtain the optimal system control actions that yield the desired energy savings that guarantee the required QoS.This work is an extension of [11], where energy savings and QoS guarantee were considered only within a virtualized computing platform placed in proximity to a cluster of BS. In [11], the role of the MEC server is to handle the offloaded computational workload only, which means that the energy cost model lacks the consideration of the BS management procedures, caching process, and the use of containers.The summary of contributions are listed as follows:(i)
The use of container-based virtualization is introduced as they are lightweight, i.e., demand less memory space, have shorter start-up time, and offer software portability(ii)
The proposed GENM algorithm, which is an online edge management system, makes use of predictive optimization, specifically using the LLC, where green-based load balancing, container provisioning, and the tuning of the transmission drivers is performed based on the learned information(iii)
The numerical results, obtained with real-world harvested energy and traffic load traces, show that the proposed optimization strategy is able to efficiently manage the edge network resources in order to minimize the energy drained under the guidance of the intelligent online-based resource manager and the energy-saving proceduresIn order to achieve these, the remainder of the paper is organized as follows: Section2 describes the related work. Section 3 explains the system model. In Section 4, the design and the implementation of the online algorithm are presented. Simulated results are discussed in Section 5. Lastly, the work is concluded in Section 6.


## 1.1. Paper Contributions

This paper considers an energy cost model that takes into account the computing, caching and communication processes within the MEC server, and transmission-related energy consumption in BS. Here, the GENM algorithm is proposed for enablinggreen-based traffic load balancing, i.e., the BS are dynamically switched on/off based on their harvested energy, autoscaling, and reconfiguring the computing resources, and the tuning of transmission drivers. This entails using a minimum number of optical drivers for real-time data transfers, over a short-term horizon. In order to solve the energy consumption minimization problem, the GENM algorithm performs online supervisory control, utilizing the learned traffic load and the harvested energy patterns. Then, the output is utilized within a LLC policy [14] to obtain the optimal system control actions that yield the desired energy savings that guarantee the required QoS.This work is an extension of [11], where energy savings and QoS guarantee were considered only within a virtualized computing platform placed in proximity to a cluster of BS. In [11], the role of the MEC server is to handle the offloaded computational workload only, which means that the energy cost model lacks the consideration of the BS management procedures, caching process, and the use of containers.The summary of contributions are listed as follows:(i)
The use of container-based virtualization is introduced as they are lightweight, i.e., demand less memory space, have shorter start-up time, and offer software portability(ii)
The proposed GENM algorithm, which is an online edge management system, makes use of predictive optimization, specifically using the LLC, where green-based load balancing, container provisioning, and the tuning of the transmission drivers is performed based on the learned information(iii)
The numerical results, obtained with real-world harvested energy and traffic load traces, show that the proposed optimization strategy is able to efficiently manage the edge network resources in order to minimize the energy drained under the guidance of the intelligent online-based resource manager and the energy-saving proceduresIn order to achieve these, the remainder of the paper is organized as follows: Section2 describes the related work. Section 3 explains the system model. In Section 4, the design and the implementation of the online algorithm are presented. Simulated results are discussed in Section 5. Lastly, the work is concluded in Section 6.


## 2. Related Work

### 2.1. Methods for Load Balancing in MN

Load balancing has been studied towards data center management whereby the data center servers employ temporal dependency strategies, i.e., the servers are turned on/off depending on the arrival rates of workloads. This significantly differs from our considered problem as we consider load balancing in SBSs powered by green energy. Towards load balancing, the dynamic BS switching on/off strategies have been used. However, this may have an impact on the network due to the load that is offloaded to the neighboring BS(s). To avoid this, the BS to be switched off is carefully identified within the BS cluster. In [5, 6], the network impact is used to identify the BS to be switched off, one at a time, with no significant network performance degradation. Taking into account daily traffic load variation, strategies for opportunistic utilization of the unexploited third-party small cell (SC) capacity is exploited towards energy savings in [15], in order to enable the switching off of some BSs. Here, an offloading mechanism is introduced, where the operators lease the capacity of a SC network owned by a third party in order to switch off their BS (Macro BS) and maximize their energy efficiency, when the traffic demand is low.The use of green energy as a performance metric has been explored within the Radio Access Network (RAN) [3, 16]. Along the lines of MN softwarization, a distributed user association scheme that makes use of the SoftRAN concept for traffic load balancing via the RAN Controller (RANC) is proposed in [3]. Here, the user association algorithm runs on the RANC, and the users report their downlink data rates via the associated BS to the RANC, where the traffic loads from individual users and BS are measured. The algorithm enhances the network performance by reducing the average traffic delivery latency in BS as well as to reduce the on-grid power consumption by optimizing the green energy usage. Then, the authors in [16] proposed to optimize the utilization of green energy for cellular networks by optimizing the BS transmission power. The proposed scheme achieves significant on-grid power savings by scheduling the green energy consumption along the time domain for individual BS and balancing the green energy consumption among the BSs.Along the lines of MEC, the authors in [4] proposed a framework for jointly performing load balancing, admission control, and energy purchase within a network of EH-powered BS with the goal of minimizing the computation delay and data traffic drops (i.e., increasing the locally computed workloads). This work uses green energy as a performance metric. To solve this problem, an online and distributed algorithm is proposed leveraging the Lyapunov optimization with the perturbation technique. Here, the algorithm makes the traffic load decisions without forecasting the future traffic load and harvested energy. In contrast, the work presented in this paper considers the short-term future traffic load and the harvested energy for decision making. Then, in our previous work [6], a supervisory online control algorithm that makes use of clustering and the network impact metric towards load balancing in MN is proposed. Here, the BSs are empowered with computation capabilities (with VM as computing resources), the LSTM neural network is used for forecasting, and the LLC policy handles foresighted optimization. Even though these works perform load balancing, the problem and scenario considered in this paper are different. Here, a MEC server manages the SBSs powered by green energy. Similar to [11], forecasting and foresighted optimization are used for edge system management.Load balancing that follows the energy routing, i.e., more computational workload is offloaded to where more energy is available, is presented in [17]. To handle spatial uneven computation workloads experienced by the MEC-enabled BSs, the authors proposed a peer offloading scheme. Here, the BSs share their computing resources and energy costs.


### 2.2. Methods for Energy Saving within Computing Platforms

Green computing over data centers is an emerging paradigm that aims at performing the dynamic energy-saving management of data center infrastructures. Here, procedures for the dynamic on/off switching of servers have been proposed as a way of minimizing energy consumption in computing platforms. A novel postdecision state-based learning algorithm for server provisioning at the network edge is presented in [18]. This work incorporates green energy. At the beginning of the time slot, the servers are consolidated, i.e., the number of turned on physical servers are minimized, using the learned optimal policy for dynamic workload offloading and the autoscaling (or right-sizing). Then, in our previous works [6, 11, 19], VM soft-scaling (i.e., the reduction of computing resources per time instance) is employed towards energy saving in virtualized platforms either energized by only renewable energy or hybrid supplies (solar and power grid). This is achieved by forecasting the traffic load and harvested energy and then employing foresighted optimization to obtain the system control inputs. The work of [20] uses an iterative algorithm to obtain the number of computing resources (VMs) to be provisioned within a node that transmits to clients wireless. Then, the work of [21] considers a vehicular scenario where vehicles connect wireless to fog nodes and then develop an adaptive scheduler, which computes on-the-fly the solutions of both the resource reconfiguration and consolidation problems. For this purpose, the primal-dual algorithm is used.In computing platforms, computation offloading strategies can be jointly exploited together with delay constraints towards energy savings. The authors in [22] proposed an offloading policy to find the optimal place where to offload and the amount of offloaded task data. In this work, the time taken for processing the offloaded task is reduced, at the same time consuming less energy. Then, in [23], an efficient scheduling for latency-sensitive applications is proposed towards energy and response time minimization. The achieved results show a reduction in delay and network usage and the energy consumption. In addition, the works of [24] jointly optimize the computing and communication resources, taking into account the local task execution delay and transmission delay. To meet the task delay requirements, in [25], the heterogeneous clouds, i.e., edge and remote cloud, are coordinated. Here, different policies are employed in the clouds. In this, the edge cloud handles tasks with loose delay bounds and drops tasks with stringent delay bounds when the traffic load is heavy.Towards minimizing energy consumption induced by communication activities within a computing node, the idea of tuning transmission drivers, as one of the energy-saving strategies within the MN infrastructure, is first conceived in [10, 26] where a computing node (router) is considered. Here, it is observed that having the least number of data transmission drivers (fast tunable lasers) can yield a significant amount of energy savings. Motivated by the aforementioned works, within the MEC paradigm, the authors in [11] put forward a traffic engineering- and MEC LS-based algorithm that uses a location-aware procedure for provisioning the transmission drivers for data transfer towards target BS. Here, the MEC LS API is employed for retrieving the UE’s location and then passing the information to the authorized applications within the MEC platform, for decision-making.Overall, these works numerically analyze and test the energy performance of some state-of-the-art schedulers for computing platforms, but do not attempt to optimize it through the dynamic joint scaling of the available communication-plus-computing resources. The joint analysis of the computing-plus-communication energy consumption within the MEC paradigm is still an open research topic.


### 2.3. Methods for Guaranteeing Quality of Service and Enabling Energy Savings (within the MEC Paradigm)

The mobile operator’s goal is to provide QoS Internet services for large populations of clients, while minimizing the overall computing-plus-communication energy consumption. Hence, a trade-off is required between QoS and energy savings. Future MNs are expected to learn the diverse characteristics of user behavior, as well as renewable energy variations, in order to autonomously determine good system configurations. Towards this goal, online forecasting using ML techniques and the LLC method can yield the desired system behavior when taking into account the environmental inputs, i.e., BS traffic load, server workloads, and energy to be harvested. Next, the mathematical tools that are used in this research work are reviewed, namely, the LLC method [14, 27, 28] and LSTM neural networks [29, 30].The LLC has been used in [27] to address a resource provision problem within virtualized environments. The optimization problem is posed as a profit maximization problem under uncertainty and the LLC formulation models the cost of control. To address the uncertainty over the workload arrival, the Kalman filter is used. Then, in [31], an online supervisory control scheme based on LLC policies is proposed. Here, after the occurrence of an event, the next control action is determined by estimating the system behavior a few steps into the future using the currently available information as inputs. The control action exploration is performed using a search tree assuming that the controller knows all future possible states of the process over the prediction horizon. An online control framework for resource management in switching hybrid systems is proposed in [14], where the system’s control inputs are finite. The relevant parameters of the operating environment, e.g., workload arrival, are estimated and then used by the system to forecast future behavior over a look-ahead horizon. From this, the controller optimizes the predicted system behavior following the specified QoS through the selection of the system controls. In [6, 19], a LLC-based supervisory algorithm is proposed to obtain the system control actions yielding the desired trade-off between energy consumption and QoS. Here, the traffic load and harvested energy are forecasted and then used as input in the algorithm. The BS are densely deployed in [6], and each BS is empowered with computation capabilities. Then, in [19], a remote site powered by only green energy is considered.LSTM can be used for multistep time series forecasting as it is able to handle the long-term dependencies due to its inherent capability of storing past information and then recalling it. The forecasting method is utilized in [6, 19] within an LLC-based algorithm to obtain the system control actions yielding the desired trade-off between energy consumption and QoS. The application of the LSTM network is extended to include ITS in [32]. A new ITS edge analytics architecture that makes use of deep learning techniques that run either on the mobile devices or on the intravehicle processors for data analytics is presented. A combination of LSTM networks and deep CNN is adopted, i.e., CNN-LSTM network, for path selection in autonomous vehicles, whereby the CNN is used for feature extraction, and then, the extracted information is fed into LSTM networks for driving path selection. Forecasting server workloads using the LSTM network can be beneficial for dynamic resource scaling and power consumption in cloud computing data centers. In [33], a forecasting model using the LSTM network for predicting future data center workloads is proposed, and then, the results are fed into the resource manager for decision-making, which involves either scaling up or down the computing resources (servers in this case).


## 2.1. Methods for Load Balancing in MN

Load balancing has been studied towards data center management whereby the data center servers employ temporal dependency strategies, i.e., the servers are turned on/off depending on the arrival rates of workloads. This significantly differs from our considered problem as we consider load balancing in SBSs powered by green energy. Towards load balancing, the dynamic BS switching on/off strategies have been used. However, this may have an impact on the network due to the load that is offloaded to the neighboring BS(s). To avoid this, the BS to be switched off is carefully identified within the BS cluster. In [5, 6], the network impact is used to identify the BS to be switched off, one at a time, with no significant network performance degradation. Taking into account daily traffic load variation, strategies for opportunistic utilization of the unexploited third-party small cell (SC) capacity is exploited towards energy savings in [15], in order to enable the switching off of some BSs. Here, an offloading mechanism is introduced, where the operators lease the capacity of a SC network owned by a third party in order to switch off their BS (Macro BS) and maximize their energy efficiency, when the traffic demand is low.The use of green energy as a performance metric has been explored within the Radio Access Network (RAN) [3, 16]. Along the lines of MN softwarization, a distributed user association scheme that makes use of the SoftRAN concept for traffic load balancing via the RAN Controller (RANC) is proposed in [3]. Here, the user association algorithm runs on the RANC, and the users report their downlink data rates via the associated BS to the RANC, where the traffic loads from individual users and BS are measured. The algorithm enhances the network performance by reducing the average traffic delivery latency in BS as well as to reduce the on-grid power consumption by optimizing the green energy usage. Then, the authors in [16] proposed to optimize the utilization of green energy for cellular networks by optimizing the BS transmission power. The proposed scheme achieves significant on-grid power savings by scheduling the green energy consumption along the time domain for individual BS and balancing the green energy consumption among the BSs.Along the lines of MEC, the authors in [4] proposed a framework for jointly performing load balancing, admission control, and energy purchase within a network of EH-powered BS with the goal of minimizing the computation delay and data traffic drops (i.e., increasing the locally computed workloads). This work uses green energy as a performance metric. To solve this problem, an online and distributed algorithm is proposed leveraging the Lyapunov optimization with the perturbation technique. Here, the algorithm makes the traffic load decisions without forecasting the future traffic load and harvested energy. In contrast, the work presented in this paper considers the short-term future traffic load and the harvested energy for decision making. Then, in our previous work [6], a supervisory online control algorithm that makes use of clustering and the network impact metric towards load balancing in MN is proposed. Here, the BSs are empowered with computation capabilities (with VM as computing resources), the LSTM neural network is used for forecasting, and the LLC policy handles foresighted optimization. Even though these works perform load balancing, the problem and scenario considered in this paper are different. Here, a MEC server manages the SBSs powered by green energy. Similar to [11], forecasting and foresighted optimization are used for edge system management.Load balancing that follows the energy routing, i.e., more computational workload is offloaded to where more energy is available, is presented in [17]. To handle spatial uneven computation workloads experienced by the MEC-enabled BSs, the authors proposed a peer offloading scheme. Here, the BSs share their computing resources and energy costs.


## 2.2. Methods for Energy Saving within Computing Platforms

Green computing over data centers is an emerging paradigm that aims at performing the dynamic energy-saving management of data center infrastructures. Here, procedures for the dynamic on/off switching of servers have been proposed as a way of minimizing energy consumption in computing platforms. A novel postdecision state-based learning algorithm for server provisioning at the network edge is presented in [18]. This work incorporates green energy. At the beginning of the time slot, the servers are consolidated, i.e., the number of turned on physical servers are minimized, using the learned optimal policy for dynamic workload offloading and the autoscaling (or right-sizing). Then, in our previous works [6, 11, 19], VM soft-scaling (i.e., the reduction of computing resources per time instance) is employed towards energy saving in virtualized platforms either energized by only renewable energy or hybrid supplies (solar and power grid). This is achieved by forecasting the traffic load and harvested energy and then employing foresighted optimization to obtain the system control inputs. The work of [20] uses an iterative algorithm to obtain the number of computing resources (VMs) to be provisioned within a node that transmits to clients wireless. Then, the work of [21] considers a vehicular scenario where vehicles connect wireless to fog nodes and then develop an adaptive scheduler, which computes on-the-fly the solutions of both the resource reconfiguration and consolidation problems. For this purpose, the primal-dual algorithm is used.In computing platforms, computation offloading strategies can be jointly exploited together with delay constraints towards energy savings. The authors in [22] proposed an offloading policy to find the optimal place where to offload and the amount of offloaded task data. In this work, the time taken for processing the offloaded task is reduced, at the same time consuming less energy. Then, in [23], an efficient scheduling for latency-sensitive applications is proposed towards energy and response time minimization. The achieved results show a reduction in delay and network usage and the energy consumption. In addition, the works of [24] jointly optimize the computing and communication resources, taking into account the local task execution delay and transmission delay. To meet the task delay requirements, in [25], the heterogeneous clouds, i.e., edge and remote cloud, are coordinated. Here, different policies are employed in the clouds. In this, the edge cloud handles tasks with loose delay bounds and drops tasks with stringent delay bounds when the traffic load is heavy.Towards minimizing energy consumption induced by communication activities within a computing node, the idea of tuning transmission drivers, as one of the energy-saving strategies within the MN infrastructure, is first conceived in [10, 26] where a computing node (router) is considered. Here, it is observed that having the least number of data transmission drivers (fast tunable lasers) can yield a significant amount of energy savings. Motivated by the aforementioned works, within the MEC paradigm, the authors in [11] put forward a traffic engineering- and MEC LS-based algorithm that uses a location-aware procedure for provisioning the transmission drivers for data transfer towards target BS. Here, the MEC LS API is employed for retrieving the UE’s location and then passing the information to the authorized applications within the MEC platform, for decision-making.Overall, these works numerically analyze and test the energy performance of some state-of-the-art schedulers for computing platforms, but do not attempt to optimize it through the dynamic joint scaling of the available communication-plus-computing resources. The joint analysis of the computing-plus-communication energy consumption within the MEC paradigm is still an open research topic.


## 2.3. Methods for Guaranteeing Quality of Service and Enabling Energy Savings (within the MEC Paradigm)

The mobile operator’s goal is to provide QoS Internet services for large populations of clients, while minimizing the overall computing-plus-communication energy consumption. Hence, a trade-off is required between QoS and energy savings. Future MNs are expected to learn the diverse characteristics of user behavior, as well as renewable energy variations, in order to autonomously determine good system configurations. Towards this goal, online forecasting using ML techniques and the LLC method can yield the desired system behavior when taking into account the environmental inputs, i.e., BS traffic load, server workloads, and energy to be harvested. Next, the mathematical tools that are used in this research work are reviewed, namely, the LLC method [14, 27, 28] and LSTM neural networks [29, 30].The LLC has been used in [27] to address a resource provision problem within virtualized environments. The optimization problem is posed as a profit maximization problem under uncertainty and the LLC formulation models the cost of control. To address the uncertainty over the workload arrival, the Kalman filter is used. Then, in [31], an online supervisory control scheme based on LLC policies is proposed. Here, after the occurrence of an event, the next control action is determined by estimating the system behavior a few steps into the future using the currently available information as inputs. The control action exploration is performed using a search tree assuming that the controller knows all future possible states of the process over the prediction horizon. An online control framework for resource management in switching hybrid systems is proposed in [14], where the system’s control inputs are finite. The relevant parameters of the operating environment, e.g., workload arrival, are estimated and then used by the system to forecast future behavior over a look-ahead horizon. From this, the controller optimizes the predicted system behavior following the specified QoS through the selection of the system controls. In [6, 19], a LLC-based supervisory algorithm is proposed to obtain the system control actions yielding the desired trade-off between energy consumption and QoS. Here, the traffic load and harvested energy are forecasted and then used as input in the algorithm. The BS are densely deployed in [6], and each BS is empowered with computation capabilities. Then, in [19], a remote site powered by only green energy is considered.LSTM can be used for multistep time series forecasting as it is able to handle the long-term dependencies due to its inherent capability of storing past information and then recalling it. The forecasting method is utilized in [6, 19] within an LLC-based algorithm to obtain the system control actions yielding the desired trade-off between energy consumption and QoS. The application of the LSTM network is extended to include ITS in [32]. A new ITS edge analytics architecture that makes use of deep learning techniques that run either on the mobile devices or on the intravehicle processors for data analytics is presented. A combination of LSTM networks and deep CNN is adopted, i.e., CNN-LSTM network, for path selection in autonomous vehicles, whereby the CNN is used for feature extraction, and then, the extracted information is fed into LSTM networks for driving path selection. Forecasting server workloads using the LSTM network can be beneficial for dynamic resource scaling and power consumption in cloud computing data centers. In [33], a forecasting model using the LSTM network for predicting future data center workloads is proposed, and then, the results are fed into the resource manager for decision-making, which involves either scaling up or down the computing resources (servers in this case).


## 3. System Model

In line with ETSI-proposed MEC deployment scenarios discussed in [34], the considered network scenario is illustrated in Figure 1 where the proposed model is cache-enabled, TCP/IP offload capable (i.e., enables partial offloading in the server’s NIC such as checksum computation [35]). The virtualized MEC server is assumed to be hosting C containers deployed at an aggregation point, which is in proximity to a cluster of N BS from the same MN operator. The BS are interconnected to the MEC server for computation workload offloading. Each network apparatus (BS, MEC server) is mainly powered by renewable energy harvested from wind and solar radiation, and it is equipped with an EB for energy storage. In this case, energy can only be purchased from the grid supply to supplement the renewable energy supplies. The BS coverage areas overlap so that green-based load balancing is possible. The EM is an entity responsible for selecting the appropriate energy source to fulfill the EB and also for monitoring the energy level of the EB. Then, the electromechanical switch (SW) aggregates the energy sources to fulfill the EB level. In the MEC server, there is the presence of a virtualized access control router which acts as an access gateway for admission control, responsible for local and remote routing, and it is locally hosted as an application. Also, the MEC platform is able to track user location via the MEC Location Service Application Programmable Interface (LS API). Lastly, a discrete-time model is considered whereby time is discretized as t=1,2,⋯ and each time slot t has a fixed duration τ=30. The list of symbols that are used in the paper is reported in Table 1.Figure 1
Edge system energized by hybrid energy sources: on-grid power and green energy (solar and wind).Table 1
Notation: list of symbols used in the analysis.
SymbolDescriptionInput parametersCMaximum number of containers hosted by the MEC server, indexed bycNNumber of BS, indexed bynξtAggregate computational workloadτTime slot durationLintAmount of aggregate workload at the input bufferLouttAmount of aggregate workload at the output bufferLoutmaxt,LinmaxtWorkload buffers maximum capacityfmaxMaximum processing rate for containercθidle,ctStatic energy consumed by containerc in the idle stateθmax,ctMaximum energy consumed by containerc at maximum processing ratezePer-container reconfiguration cost caused by a unit-size frequency switchingλctWorkload fraction to be computed by thec-th containerλmaxMaximum computation load per containerΔMaximum per-slot and per-container allowed processing timeθidleNICtEnergy drained by the NIC when powered, with no data transferMMaximum number of multiple fast tunable lasersβmaxMaximum energy buffer capacityβup,βlowUpper and lower energy buffer thresholdsVariablesθCOMMtTotal BS energy consumption at time slottθMECtServer’s energy consumption at time slottθCNTtEnergy drained due to the active containers, w.r.t CPU utilization, at time slottθSCtEnergy drained due to container switching the processing rates at time slottθOFFtEnergy induced by the TOE attθLNKtEnergy drained due to the virtual-link communication cost at time slottθDRtEnergy drained due to the number of active transmission drivers at time slottθCCtTotal energy cost incurred by the content caching processCtNumber of containers to be active in time slottfctInstantaneous processing rateψctLoad-dependent factorrctc-th virtual link communication rate at slot tζntBS switching status indicator attθmaxNICtMaximum energy drained by the TOE attχttThe expected processing timeMtNumber of active transmission drivers attbtEnergy buffer level in slottHtHarvested energy profile in slottEtPurchased grid energy in slott


### 3.1. Communication Traffic and Energy Consumption

From a networking perspective, the understanding and characterization of energy consumption within the MN can pave the way towards more efficient and user-oriented networking solutions. This can be achieved through the use of historical mobile traffic traces such as Call Detail Records (CDRs) obtained from mobile operators, specifically in the EPC network. Due to the difficulties in obtaining relevant open-source datasets containing computing requests, real MN traffic load traces obtained from the TIM network (availed through the Big Data Challenge [36]) are used to emulate the computational load. In order to understand the daily traffic load patterns, the clustering algorithm X-means [37] has been applied to classify the load profiles into several categories. Here, each BS n is assumed to have a related load profile Lnt which is picked at random as one of the four clusters in Figure 2. In addition, it is assumed that Lnt consists of 80% delay sensitive workloads γnt and the remainder is delay-tolerant workloads. The total aggregate delay-sensitive workload per time instance is ξt=∑n=1Nγnt.Figure 2
Normalized BS traffic load behavior represented as clusters. The data from [39] has been split into four representative clusters.The virtualized router in the MEC server of Figure1 determines the amount of workload that can be accepted by the input buffer at slot t, and the aggregated (or admitted) workload is denoted by Lint∈0,Linmax (measured in [Mbits]). Linmax is the maximum input buffer size. In addition, it is assumed that the input/output (I/O) queues of the MEC server are loss-free, and they implement the First-In First-Out (FIFO) service discipline, thus Lint=Loutt, where Loutt is the amount of the aggregate computed workload at the output buffer.The total energy consumption ([J]) for the edge system at time slott is formulated as follows, inspired by [11, 12]:
(1)θEDGEt=θCOMMt+θMECt,where θCOMMt is the energy consumption term induced by all BS communications and θMECt is the energy consumption term induced by the MEC server’s computing, caching, and communication processes.


#### 3.1.1. BS Energy Cost

The overall energy consumption within the coverage area is defined as the sum of all the BS components:(2)θCOMMt=∑n=1NθBS,nt=∑n=1Nδntθ0+θload,nt,where δnt∈0,1 is the BS n switching status indicator (1 for active mode and 0 for power saving mode); θ0 is a constant value (load independent) representing the operation energy which includes baseband processing, radio frequency power expenditures, etc.; θload,nt is the load-dependent BS transmission power to the served users that guarantees low latency at the edge. It is obtained by using the transmission model in [38].


#### 3.1.2. MEC Energy Cost

The energy drained due to the computing, caching, and communication processes is defined as:(3)θMECt=θCNTt+θSWTt+θOFFt+θLNKt+θDRt+θCCt,where θCNTt is the energy drained due to the active containers, w.r.t CPU utilization, and θSWTt is the energy drained due to containers adapting to new processing rates fct∈f0,fmaxMbit/s. The term f0 is the zero speed of the container, e.g., deep sleep or shutdown, and fmax is the maximum available processing rate for container c. It is worth noting that actual containers are generally instantiated atop physical computing cores which offer only a finite set of processing speeds. The term θOFFt is the energy induced by the TCP/IP offload on the NIC, and θLNKt is the energy drained due to the virtual-links (to-and-from containers) communication cost. Then, θDRt is the amount of energy consumed by the active transmission drivers, and θCCt is the total energy cost incurred by the content caching process.In this regard, it is assumed that real-time processing of computation workloads are performed in parallel over the containers interconnected by a rate-adaptive Virtual LAN (VLAN). In addition, it is also assumed that the CPU frequency is fixed at each user and may vary over users. The amount of energy consumed by the CPU is related to the provisioned computing resources, i.e., the CPU share allocated to each container, per time instancet, named Ct≤C, index by c. Thus, θCNTt is defined as [19]:
(4)θCNTt=∑c=1Ctθidle,ct+ψctθmax,ct−θidle,ct,where θidle,ct represents the static energy drained by container c in the idle state, ψct=fct/fmax2 is the utilization function of container c [28], and θmax,ct is the maximum energy that container c can consume. The quantity ψctθmax,ct−θidle,ct represents the dynamic energy component of container c.The intelligent resource manager implements a suitable frequency-scaling policy in real-time, in order to allow the containers to scale up/down their processing ratesfct at the minimum cost. In this regard, it should be noted that switching from the processing frequency fct−1 (the processing rate at the t−1 time instance) to the next processing frequency fct entails an energy cost, θSWTt. This depends on the absolute processing rate gap fct−fct−1; thus, θSWTt is defined as:
(5)θSWTt=∑c=1Ctzefct−fct−12,where ze is the per-container reconfiguration cost caused by a unit-size frequency switching. Typically,zeis limited to a few hundreds of millijoule per square megahertz.Before proceeding, it is worth noting the following: at the beginning of the time slott, the online algorithm adaptively allocates the available resources and then determine the containers that are demanded,Ct; the size of the workload allocated to the containerc, denoted byλct; andfctfor containercthat will yield the desired or expected processing time,χct=λct/fct.χct≤Δ, whereΔis the maximum per-slot and per-container processing time ([s]). Note that Lint=∑c=1Ctλct is the amount of computational workload admitted in the MEC server, by the router. The amount of the workload to be admitted per slot shall be decided at the beginning of each time slot depending on the forecasted green energy, grid power to be purchased, and the expected computational workloads L^int. Moreover, virtualization technologies specify the minimum and maximum amount of resources that can be allocated per container [40]; thus, the maximum amount is denoted by λmax. Lastly, the container(s) provisioning and workload allocation is discussed in Section 4.2, Remark 1, and fct=Δλct/Δ.By implementing a TCP Offload Engine (TOE) in high-speed computing environments, some TCP/IP processing is offloaded to the network adapter for the purpose of reducing the CPU utilization. To obtain the energy cost incurred, the performance measure for the Broadcom (Fibre) 10 Gbps NIC [35] is considered here as an example of a TCP/IP offload-capable device. Note that θOFFt is data volume-dependent and it is obtained as:
(6)θOFFt=ζtθidleNICt+θmaxNICt,where θidleNICt>0 is the energy drained by the TOE when powered, with all links connected without any data transfer. This provides an opportunity for switching off the network adapter if there is no data transfer, making the energy drained to be zero. For this, ζt=0,1 is the switching status indicator (1 for active state and 0 for idle state) and θmaxNICt=gt·Lint/η is the maximum energy drained, where gt is a fractional value representing the amount of load computed in the network adapter and η is the NIC best throughput performance, hereby obtained as a fixed value measured in [Gbit/].In order to keep the transmission delays from (to) the scheduler to (from) the connected containers at a minimum value, it is assumed that each containerc communicates with the resource scheduler through a dedicated reliable link that operates at the transmission rate of rct [(bit/s)]. Thus, the energy needed for sustaining the two-way cth link is defined, inspired by [41]:
(7)θLNKt=2∑c=1CtPcrctλct/rct,where Pcrct=Sc2rct/Wc−1 is the power drained by the cth communication link and Sc=Wc×N0c/gc. N0cW/Hz is the noise spectral power density, Wc is the bandwidth, and gc is the (nonnegative) gain of the cth link. In practical application scenarios, the maximum per-slot communication rate within the intra-VLAN is generally limited up to an assigned value rmax. Thus, the following hard constraint must hold: ∑c=1Ctrct≤rmax.In this regard, a two-way per-task execution delay is considered. Here, there is a total ofc=1,⋯,Ct link connection delays, each denoted by ρct=λct/rct, and χct≤Δ where Δ is the server’s response time, i.e., the maximum time allowed for processing the total computation load, and it is fixed in advance regardless of the task size allocated to container c. Since parallel real-time processing is assumed in this work, the overall communication equates to 2ρct+Δ. Therefore, the hard per-task delay constraint on the computation time is: max2ρct+Δ=τmax, where τmax is the maximum tolerable delay, which is fixed in advance.Edge distributed devices utilize low-level signaling for information sharing. Thus, edge computing systems receive information from mobile devices within the local access network to discover their location. In return, for everyclient who offloaded their task into the MEC server associated with the radio nodes, i.e., BS, its location and the computation result are known through the LS (which is a service that supports UE location retrieval mechanism, and then passing the information to the authorized applications within the server), thus enabling the location-aware traffic routing and obtaining the number of transmission drivers to be active for data transfers. The term θDRt depends on the number of active laser (optical) drivers, named Mt≤M, where M is the total number of drivers that are required for transferring ℓmt∈Loutt in time slot t (ℓmt is the downlink traffic volume ([bits] of the driver at slot t). Loutt is accumulated over a fixed period of time to form a batch at the output buffer. This means that a large number of drivers yield large transmission speed while at the same time resulting in high energy consumption [26]. Therefore, the energy consumption can be minimized by launching an optimal number of drivers for the data transfer.The energy drained during the data transmission process consists of the following: a constant energy for utilizing each fast tunable driver denoted bydmtJ/s, the target transmission rate r0 [bits/s], and Loutt. Thus, the energy is inspired by [11]:
(8)θDRt=∑m=1Mtdmtlmtr0,where the parameter Mt is obtained using the total number of target BS as Mt=1/u·ωt+1/ωt2, where ωt=ρ/σNBSt∙∈0,1 is a controllable factor that determines the delay constraint of optical networks, σ ([ms]) is the reconfiguration cost for tuning the transceivers, NBSt is an integer value representing the total number of target BS at time slot t, and ρ is the number of time slots at which the computed workload is accumulated at the output buffer. Thus, the terms σ and ρ are fixed values, and Loutt is equally distributed over the Mt drivers.The MEC server is able to cache contents from the Internet and store the contents closer to mobile users. The caching process also contributes to the energy consumption in the server. The caching process is restricted to only viral content. For example, when a video becomes viral, users watching it share and talk about that video, which will be then requested by other users after a response time. Taking into account the Internet users’ response timeλ¯t, this epidemic behavior can be modelled by the self-excited Hawkes condition Poisson process described in [42]: λ¯t=Vt+∑ti≤tΩikit−ti, where kit is the response time function and Ωi is the number of potential viewers who will be influenced after ti, which is the time when the user i shared the video. The term Vt is added as a component of the model to capture the views that are not triggered by the epidemic effect. The energy consumption is mainly contributed by content caching and data transmission processes, as such θCCt is defined as:
(9)θCCt=λ¯tθTRt+θCACHEt,where θTRt is the power consumption due to transmission and θCACHEt is the power consumption contributed by the caching process.


### 3.2. Energy Patterns and Storage

The rechargebale energy storage device is characterized by its finite energy storage capacitybmax. At each time instance, the energy level reports are pushed from the BS sites to the MEC server. Thus, the EB level bt is known, enabling the provisioning of the required computation and communication resources, i.e., the required containers, transmission drivers, and BSs to be active. In this paper, the amount of harvested energy Ht, per site BS site, in time slot t is obtained from open-source solar and wind traces from a farm located in Belgium [43] (see Figure 3 above). The data in the dataset matches our time slot duration (30). The dataset is the result of daily environmental records for a place assumed to be free from surrounding obstructions (e.g., buildings, shades).Figure 3
Example traces for harvested solar traces and wind traces from [43].The harvested energyHt is obtained by picking a day at random in the dataset and associating it with one site. Here, the wind energy is selected as a source during the solar energy off-peak periods. The available EB level bt+1 located at the BS site (BS n) or computing platform evolves according to the following dynamics:
(10)bt+1=minbt+Ht−θsitet−at+Et,bmax,where bt is the energy level in the battery at the beginning of the time slot t and θsitet represents either θBS,nt, the BS energy consumption of the communication site, or θMECt, the energy drained at the computing platform, over time slot t, see Eqs. (2) and (3). at is leakage energy, and Et≥0 is the amount of energy purchased from the power grid. It is worth noting that bt is updated at the beginning of time slot t, whereas Ht, θBS,nt, and θMECt are only known at the end of it. Thus, the energy constraint at the computing site must be satisfied for every time slot: θMECt≤bt.For decision-making in the GENM application, the received EB level reports are compared with the following thresholds:blow and bup, respectively, termed the lower and the upper energy threshold with 0<blow<bup<bmax. bup corresponds to the desired energy buffer level at the BS site or computing site, and blow is the lowest EB level that any site should ever reach. If bt<blow, then BS n or the computing site is said to be energy deficient. The suitable energy source at each time slot t is selected based on the forecast expectations, i.e., the expected harvested energy H^t. If H^t is enough to reach bup, no energy purchase is needed. Otherwise, the remaining amount up to bup, i.e., Et=bup−bt, is purchased from the electrical grid. Our optimization framework in Section 4.1 makes sure that bt never falls below blow and guarantees that bup is reached at every time slot.


## 3.1. Communication Traffic and Energy Consumption

From a networking perspective, the understanding and characterization of energy consumption within the MN can pave the way towards more efficient and user-oriented networking solutions. This can be achieved through the use of historical mobile traffic traces such as Call Detail Records (CDRs) obtained from mobile operators, specifically in the EPC network. Due to the difficulties in obtaining relevant open-source datasets containing computing requests, real MN traffic load traces obtained from the TIM network (availed through the Big Data Challenge [36]) are used to emulate the computational load. In order to understand the daily traffic load patterns, the clustering algorithm X-means [37] has been applied to classify the load profiles into several categories. Here, each BS n is assumed to have a related load profile Lnt which is picked at random as one of the four clusters in Figure 2. In addition, it is assumed that Lnt consists of 80% delay sensitive workloads γnt and the remainder is delay-tolerant workloads. The total aggregate delay-sensitive workload per time instance is ξt=∑n=1Nγnt.Figure 2
Normalized BS traffic load behavior represented as clusters. The data from [39] has been split into four representative clusters.The virtualized router in the MEC server of Figure1 determines the amount of workload that can be accepted by the input buffer at slot t, and the aggregated (or admitted) workload is denoted by Lint∈0,Linmax (measured in [Mbits]). Linmax is the maximum input buffer size. In addition, it is assumed that the input/output (I/O) queues of the MEC server are loss-free, and they implement the First-In First-Out (FIFO) service discipline, thus Lint=Loutt, where Loutt is the amount of the aggregate computed workload at the output buffer.The total energy consumption ([J]) for the edge system at time slott is formulated as follows, inspired by [11, 12]:
(1)θEDGEt=θCOMMt+θMECt,where θCOMMt is the energy consumption term induced by all BS communications and θMECt is the energy consumption term induced by the MEC server’s computing, caching, and communication processes.


### 3.1.1. BS Energy Cost

The overall energy consumption within the coverage area is defined as the sum of all the BS components:(2)θCOMMt=∑n=1NθBS,nt=∑n=1Nδntθ0+θload,nt,where δnt∈0,1 is the BS n switching status indicator (1 for active mode and 0 for power saving mode); θ0 is a constant value (load independent) representing the operation energy which includes baseband processing, radio frequency power expenditures, etc.; θload,nt is the load-dependent BS transmission power to the served users that guarantees low latency at the edge. It is obtained by using the transmission model in [38].


### 3.1.2. MEC Energy Cost

The energy drained due to the computing, caching, and communication processes is defined as:(3)θMECt=θCNTt+θSWTt+θOFFt+θLNKt+θDRt+θCCt,where θCNTt is the energy drained due to the active containers, w.r.t CPU utilization, and θSWTt is the energy drained due to containers adapting to new processing rates fct∈f0,fmaxMbit/s. The term f0 is the zero speed of the container, e.g., deep sleep or shutdown, and fmax is the maximum available processing rate for container c. It is worth noting that actual containers are generally instantiated atop physical computing cores which offer only a finite set of processing speeds. The term θOFFt is the energy induced by the TCP/IP offload on the NIC, and θLNKt is the energy drained due to the virtual-links (to-and-from containers) communication cost. Then, θDRt is the amount of energy consumed by the active transmission drivers, and θCCt is the total energy cost incurred by the content caching process.In this regard, it is assumed that real-time processing of computation workloads are performed in parallel over the containers interconnected by a rate-adaptive Virtual LAN (VLAN). In addition, it is also assumed that the CPU frequency is fixed at each user and may vary over users. The amount of energy consumed by the CPU is related to the provisioned computing resources, i.e., the CPU share allocated to each container, per time instancet, named Ct≤C, index by c. Thus, θCNTt is defined as [19]:
(4)θCNTt=∑c=1Ctθidle,ct+ψctθmax,ct−θidle,ct,where θidle,ct represents the static energy drained by container c in the idle state, ψct=fct/fmax2 is the utilization function of container c [28], and θmax,ct is the maximum energy that container c can consume. The quantity ψctθmax,ct−θidle,ct represents the dynamic energy component of container c.The intelligent resource manager implements a suitable frequency-scaling policy in real-time, in order to allow the containers to scale up/down their processing ratesfct at the minimum cost. In this regard, it should be noted that switching from the processing frequency fct−1 (the processing rate at the t−1 time instance) to the next processing frequency fct entails an energy cost, θSWTt. This depends on the absolute processing rate gap fct−fct−1; thus, θSWTt is defined as:
(5)θSWTt=∑c=1Ctzefct−fct−12,where ze is the per-container reconfiguration cost caused by a unit-size frequency switching. Typically,zeis limited to a few hundreds of millijoule per square megahertz.Before proceeding, it is worth noting the following: at the beginning of the time slott, the online algorithm adaptively allocates the available resources and then determine the containers that are demanded,Ct; the size of the workload allocated to the containerc, denoted byλct; andfctfor containercthat will yield the desired or expected processing time,χct=λct/fct.χct≤Δ, whereΔis the maximum per-slot and per-container processing time ([s]). Note that Lint=∑c=1Ctλct is the amount of computational workload admitted in the MEC server, by the router. The amount of the workload to be admitted per slot shall be decided at the beginning of each time slot depending on the forecasted green energy, grid power to be purchased, and the expected computational workloads L^int. Moreover, virtualization technologies specify the minimum and maximum amount of resources that can be allocated per container [40]; thus, the maximum amount is denoted by λmax. Lastly, the container(s) provisioning and workload allocation is discussed in Section 4.2, Remark 1, and fct=Δλct/Δ.By implementing a TCP Offload Engine (TOE) in high-speed computing environments, some TCP/IP processing is offloaded to the network adapter for the purpose of reducing the CPU utilization. To obtain the energy cost incurred, the performance measure for the Broadcom (Fibre) 10 Gbps NIC [35] is considered here as an example of a TCP/IP offload-capable device. Note that θOFFt is data volume-dependent and it is obtained as:
(6)θOFFt=ζtθidleNICt+θmaxNICt,where θidleNICt>0 is the energy drained by the TOE when powered, with all links connected without any data transfer. This provides an opportunity for switching off the network adapter if there is no data transfer, making the energy drained to be zero. For this, ζt=0,1 is the switching status indicator (1 for active state and 0 for idle state) and θmaxNICt=gt·Lint/η is the maximum energy drained, where gt is a fractional value representing the amount of load computed in the network adapter and η is the NIC best throughput performance, hereby obtained as a fixed value measured in [Gbit/].In order to keep the transmission delays from (to) the scheduler to (from) the connected containers at a minimum value, it is assumed that each containerc communicates with the resource scheduler through a dedicated reliable link that operates at the transmission rate of rct [(bit/s)]. Thus, the energy needed for sustaining the two-way cth link is defined, inspired by [41]:
(7)θLNKt=2∑c=1CtPcrctλct/rct,where Pcrct=Sc2rct/Wc−1 is the power drained by the cth communication link and Sc=Wc×N0c/gc. N0cW/Hz is the noise spectral power density, Wc is the bandwidth, and gc is the (nonnegative) gain of the cth link. In practical application scenarios, the maximum per-slot communication rate within the intra-VLAN is generally limited up to an assigned value rmax. Thus, the following hard constraint must hold: ∑c=1Ctrct≤rmax.In this regard, a two-way per-task execution delay is considered. Here, there is a total ofc=1,⋯,Ct link connection delays, each denoted by ρct=λct/rct, and χct≤Δ where Δ is the server’s response time, i.e., the maximum time allowed for processing the total computation load, and it is fixed in advance regardless of the task size allocated to container c. Since parallel real-time processing is assumed in this work, the overall communication equates to 2ρct+Δ. Therefore, the hard per-task delay constraint on the computation time is: max2ρct+Δ=τmax, where τmax is the maximum tolerable delay, which is fixed in advance.Edge distributed devices utilize low-level signaling for information sharing. Thus, edge computing systems receive information from mobile devices within the local access network to discover their location. In return, for everyclient who offloaded their task into the MEC server associated with the radio nodes, i.e., BS, its location and the computation result are known through the LS (which is a service that supports UE location retrieval mechanism, and then passing the information to the authorized applications within the server), thus enabling the location-aware traffic routing and obtaining the number of transmission drivers to be active for data transfers. The term θDRt depends on the number of active laser (optical) drivers, named Mt≤M, where M is the total number of drivers that are required for transferring ℓmt∈Loutt in time slot t (ℓmt is the downlink traffic volume ([bits] of the driver at slot t). Loutt is accumulated over a fixed period of time to form a batch at the output buffer. This means that a large number of drivers yield large transmission speed while at the same time resulting in high energy consumption [26]. Therefore, the energy consumption can be minimized by launching an optimal number of drivers for the data transfer.The energy drained during the data transmission process consists of the following: a constant energy for utilizing each fast tunable driver denoted bydmtJ/s, the target transmission rate r0 [bits/s], and Loutt. Thus, the energy is inspired by [11]:
(8)θDRt=∑m=1Mtdmtlmtr0,where the parameter Mt is obtained using the total number of target BS as Mt=1/u·ωt+1/ωt2, where ωt=ρ/σNBSt∙∈0,1 is a controllable factor that determines the delay constraint of optical networks, σ ([ms]) is the reconfiguration cost for tuning the transceivers, NBSt is an integer value representing the total number of target BS at time slot t, and ρ is the number of time slots at which the computed workload is accumulated at the output buffer. Thus, the terms σ and ρ are fixed values, and Loutt is equally distributed over the Mt drivers.The MEC server is able to cache contents from the Internet and store the contents closer to mobile users. The caching process also contributes to the energy consumption in the server. The caching process is restricted to only viral content. For example, when a video becomes viral, users watching it share and talk about that video, which will be then requested by other users after a response time. Taking into account the Internet users’ response timeλ¯t, this epidemic behavior can be modelled by the self-excited Hawkes condition Poisson process described in [42]: λ¯t=Vt+∑ti≤tΩikit−ti, where kit is the response time function and Ωi is the number of potential viewers who will be influenced after ti, which is the time when the user i shared the video. The term Vt is added as a component of the model to capture the views that are not triggered by the epidemic effect. The energy consumption is mainly contributed by content caching and data transmission processes, as such θCCt is defined as:
(9)θCCt=λ¯tθTRt+θCACHEt,where θTRt is the power consumption due to transmission and θCACHEt is the power consumption contributed by the caching process.


## 3.1.1. BS Energy Cost

The overall energy consumption within the coverage area is defined as the sum of all the BS components:(2)θCOMMt=∑n=1NθBS,nt=∑n=1Nδntθ0+θload,nt,where δnt∈0,1 is the BS n switching status indicator (1 for active mode and 0 for power saving mode); θ0 is a constant value (load independent) representing the operation energy which includes baseband processing, radio frequency power expenditures, etc.; θload,nt is the load-dependent BS transmission power to the served users that guarantees low latency at the edge. It is obtained by using the transmission model in [38].


## 3.1.2. MEC Energy Cost

The energy drained due to the computing, caching, and communication processes is defined as:(3)θMECt=θCNTt+θSWTt+θOFFt+θLNKt+θDRt+θCCt,where θCNTt is the energy drained due to the active containers, w.r.t CPU utilization, and θSWTt is the energy drained due to containers adapting to new processing rates fct∈f0,fmaxMbit/s. The term f0 is the zero speed of the container, e.g., deep sleep or shutdown, and fmax is the maximum available processing rate for container c. It is worth noting that actual containers are generally instantiated atop physical computing cores which offer only a finite set of processing speeds. The term θOFFt is the energy induced by the TCP/IP offload on the NIC, and θLNKt is the energy drained due to the virtual-links (to-and-from containers) communication cost. Then, θDRt is the amount of energy consumed by the active transmission drivers, and θCCt is the total energy cost incurred by the content caching process.In this regard, it is assumed that real-time processing of computation workloads are performed in parallel over the containers interconnected by a rate-adaptive Virtual LAN (VLAN). In addition, it is also assumed that the CPU frequency is fixed at each user and may vary over users. The amount of energy consumed by the CPU is related to the provisioned computing resources, i.e., the CPU share allocated to each container, per time instancet, named Ct≤C, index by c. Thus, θCNTt is defined as [19]:
(4)θCNTt=∑c=1Ctθidle,ct+ψctθmax,ct−θidle,ct,where θidle,ct represents the static energy drained by container c in the idle state, ψct=fct/fmax2 is the utilization function of container c [28], and θmax,ct is the maximum energy that container c can consume. The quantity ψctθmax,ct−θidle,ct represents the dynamic energy component of container c.The intelligent resource manager implements a suitable frequency-scaling policy in real-time, in order to allow the containers to scale up/down their processing ratesfct at the minimum cost. In this regard, it should be noted that switching from the processing frequency fct−1 (the processing rate at the t−1 time instance) to the next processing frequency fct entails an energy cost, θSWTt. This depends on the absolute processing rate gap fct−fct−1; thus, θSWTt is defined as:
(5)θSWTt=∑c=1Ctzefct−fct−12,where ze is the per-container reconfiguration cost caused by a unit-size frequency switching. Typically,zeis limited to a few hundreds of millijoule per square megahertz.Before proceeding, it is worth noting the following: at the beginning of the time slott, the online algorithm adaptively allocates the available resources and then determine the containers that are demanded,Ct; the size of the workload allocated to the containerc, denoted byλct; andfctfor containercthat will yield the desired or expected processing time,χct=λct/fct.χct≤Δ, whereΔis the maximum per-slot and per-container processing time ([s]). Note that Lint=∑c=1Ctλct is the amount of computational workload admitted in the MEC server, by the router. The amount of the workload to be admitted per slot shall be decided at the beginning of each time slot depending on the forecasted green energy, grid power to be purchased, and the expected computational workloads L^int. Moreover, virtualization technologies specify the minimum and maximum amount of resources that can be allocated per container [40]; thus, the maximum amount is denoted by λmax. Lastly, the container(s) provisioning and workload allocation is discussed in Section 4.2, Remark 1, and fct=Δλct/Δ.By implementing a TCP Offload Engine (TOE) in high-speed computing environments, some TCP/IP processing is offloaded to the network adapter for the purpose of reducing the CPU utilization. To obtain the energy cost incurred, the performance measure for the Broadcom (Fibre) 10 Gbps NIC [35] is considered here as an example of a TCP/IP offload-capable device. Note that θOFFt is data volume-dependent and it is obtained as:
(6)θOFFt=ζtθidleNICt+θmaxNICt,where θidleNICt>0 is the energy drained by the TOE when powered, with all links connected without any data transfer. This provides an opportunity for switching off the network adapter if there is no data transfer, making the energy drained to be zero. For this, ζt=0,1 is the switching status indicator (1 for active state and 0 for idle state) and θmaxNICt=gt·Lint/η is the maximum energy drained, where gt is a fractional value representing the amount of load computed in the network adapter and η is the NIC best throughput performance, hereby obtained as a fixed value measured in [Gbit/].In order to keep the transmission delays from (to) the scheduler to (from) the connected containers at a minimum value, it is assumed that each containerc communicates with the resource scheduler through a dedicated reliable link that operates at the transmission rate of rct [(bit/s)]. Thus, the energy needed for sustaining the two-way cth link is defined, inspired by [41]:
(7)θLNKt=2∑c=1CtPcrctλct/rct,where Pcrct=Sc2rct/Wc−1 is the power drained by the cth communication link and Sc=Wc×N0c/gc. N0cW/Hz is the noise spectral power density, Wc is the bandwidth, and gc is the (nonnegative) gain of the cth link. In practical application scenarios, the maximum per-slot communication rate within the intra-VLAN is generally limited up to an assigned value rmax. Thus, the following hard constraint must hold: ∑c=1Ctrct≤rmax.In this regard, a two-way per-task execution delay is considered. Here, there is a total ofc=1,⋯,Ct link connection delays, each denoted by ρct=λct/rct, and χct≤Δ where Δ is the server’s response time, i.e., the maximum time allowed for processing the total computation load, and it is fixed in advance regardless of the task size allocated to container c. Since parallel real-time processing is assumed in this work, the overall communication equates to 2ρct+Δ. Therefore, the hard per-task delay constraint on the computation time is: max2ρct+Δ=τmax, where τmax is the maximum tolerable delay, which is fixed in advance.Edge distributed devices utilize low-level signaling for information sharing. Thus, edge computing systems receive information from mobile devices within the local access network to discover their location. In return, for everyclient who offloaded their task into the MEC server associated with the radio nodes, i.e., BS, its location and the computation result are known through the LS (which is a service that supports UE location retrieval mechanism, and then passing the information to the authorized applications within the server), thus enabling the location-aware traffic routing and obtaining the number of transmission drivers to be active for data transfers. The term θDRt depends on the number of active laser (optical) drivers, named Mt≤M, where M is the total number of drivers that are required for transferring ℓmt∈Loutt in time slot t (ℓmt is the downlink traffic volume ([bits] of the driver at slot t). Loutt is accumulated over a fixed period of time to form a batch at the output buffer. This means that a large number of drivers yield large transmission speed while at the same time resulting in high energy consumption [26]. Therefore, the energy consumption can be minimized by launching an optimal number of drivers for the data transfer.The energy drained during the data transmission process consists of the following: a constant energy for utilizing each fast tunable driver denoted bydmtJ/s, the target transmission rate r0 [bits/s], and Loutt. Thus, the energy is inspired by [11]:
(8)θDRt=∑m=1Mtdmtlmtr0,where the parameter Mt is obtained using the total number of target BS as Mt=1/u·ωt+1/ωt2, where ωt=ρ/σNBSt∙∈0,1 is a controllable factor that determines the delay constraint of optical networks, σ ([ms]) is the reconfiguration cost for tuning the transceivers, NBSt is an integer value representing the total number of target BS at time slot t, and ρ is the number of time slots at which the computed workload is accumulated at the output buffer. Thus, the terms σ and ρ are fixed values, and Loutt is equally distributed over the Mt drivers.The MEC server is able to cache contents from the Internet and store the contents closer to mobile users. The caching process also contributes to the energy consumption in the server. The caching process is restricted to only viral content. For example, when a video becomes viral, users watching it share and talk about that video, which will be then requested by other users after a response time. Taking into account the Internet users’ response timeλ¯t, this epidemic behavior can be modelled by the self-excited Hawkes condition Poisson process described in [42]: λ¯t=Vt+∑ti≤tΩikit−ti, where kit is the response time function and Ωi is the number of potential viewers who will be influenced after ti, which is the time when the user i shared the video. The term Vt is added as a component of the model to capture the views that are not triggered by the epidemic effect. The energy consumption is mainly contributed by content caching and data transmission processes, as such θCCt is defined as:
(9)θCCt=λ¯tθTRt+θCACHEt,where θTRt is the power consumption due to transmission and θCACHEt is the power consumption contributed by the caching process.


## 3.2. Energy Patterns and Storage

The rechargebale energy storage device is characterized by its finite energy storage capacitybmax. At each time instance, the energy level reports are pushed from the BS sites to the MEC server. Thus, the EB level bt is known, enabling the provisioning of the required computation and communication resources, i.e., the required containers, transmission drivers, and BSs to be active. In this paper, the amount of harvested energy Ht, per site BS site, in time slot t is obtained from open-source solar and wind traces from a farm located in Belgium [43] (see Figure 3 above). The data in the dataset matches our time slot duration (30). The dataset is the result of daily environmental records for a place assumed to be free from surrounding obstructions (e.g., buildings, shades).Figure 3
Example traces for harvested solar traces and wind traces from [43].The harvested energyHt is obtained by picking a day at random in the dataset and associating it with one site. Here, the wind energy is selected as a source during the solar energy off-peak periods. The available EB level bt+1 located at the BS site (BS n) or computing platform evolves according to the following dynamics:
(10)bt+1=minbt+Ht−θsitet−at+Et,bmax,where bt is the energy level in the battery at the beginning of the time slot t and θsitet represents either θBS,nt, the BS energy consumption of the communication site, or θMECt, the energy drained at the computing platform, over time slot t, see Eqs. (2) and (3). at is leakage energy, and Et≥0 is the amount of energy purchased from the power grid. It is worth noting that bt is updated at the beginning of time slot t, whereas Ht, θBS,nt, and θMECt are only known at the end of it. Thus, the energy constraint at the computing site must be satisfied for every time slot: θMECt≤bt.For decision-making in the GENM application, the received EB level reports are compared with the following thresholds:blow and bup, respectively, termed the lower and the upper energy threshold with 0<blow<bup<bmax. bup corresponds to the desired energy buffer level at the BS site or computing site, and blow is the lowest EB level that any site should ever reach. If bt<blow, then BS n or the computing site is said to be energy deficient. The suitable energy source at each time slot t is selected based on the forecast expectations, i.e., the expected harvested energy H^t. If H^t is enough to reach bup, no energy purchase is needed. Otherwise, the remaining amount up to bup, i.e., Et=bup−bt, is purchased from the electrical grid. Our optimization framework in Section 4.1 makes sure that bt never falls below blow and guarantees that bup is reached at every time slot.


## 4. Problem Formulation

In this section, the optimization problem is formulated to obtain reduced energy consumption through short-term traffic load and harvested solar energy forecasting along with energy management procedures. The optimization problem is defined in Section4.1, and the edge system management procedures are presented in Section 4.2.


### 4.1. Optimization Problem

Our objective is to improve the overall energy savings of the edge system through BS power saving modes (i.e., green-based traffic load balancing), autoscaling of containers, contents caching, and tuning of the transmission drivers, and also to guarantee the QoS within the network. Note that at the end of each time slot, the EB states are updated depending on the harvested energy and the consumed energy, thereby linking per-time slot problems across time.To achieve our objective, two cost functions are defined, one captures the edge system energy consumption and the other handles the QoS. This is defined as (F1)θEDGEt, which weighs the energy consumption due to transmission in the BSs and the computing-plus-communication activities in the MEC server and (F2) a quadratic termξt−Lint2, which accounts for the QoS. In this regard, it is worth noting that F1 tends to push the system towards self-sustainability solutions and F2 favors solutions where the delay-sensitive load is entirely admitted in the MEC server by the router application, taking into account the expected energy to be harvested in the computing site. A weight Γ=0,1 is utilized to balance the two objectives F1 and F2. The corresponding (weighted) cost function is defined as:
(11)Jδ,ψ,M,t=ΔΓ¯θEDGEδnt,ψct,Mt,t+Γξt−Lint2,where Γ¯=Δ1−Γ. Hence, starting from t=1 (i.e., t=1,2,⋯,T) as the current time slot and the finite horizon T, the following optimization problem is formulated as:
(12)P1:minE∑t=1TJδ,ψ,M,t,subject to:A1:δnt∈ε,1,A2:β≤Ct≤C,A3:bt≥blow,A4:0≤fct≤fmax,A5:0≤λct≤λmax,A6:χct≤Δ,A7:∑c=1Ctrct≤rmax,A8:θMECt≤bt,A9:max2ρct+Δ=τmax,t=1,⋯,T,where the set of objective variables to be configured at slott in the BS system and MEC server is defined as E=Δδnt,Ct,ψct,Pct,λct,ζt,Mt. The setting handles the transmission and computing-plus-communication activities. Constraint A1 specifies the BS operation status (either power saving or active), and A2 forces the required number of containers, Ct, to be always greater than or equal to a minimum number β≥1: the purpose of this is to be always able to handle mission-critical communications. A3 makes sure that the EB level is always above or equal to a preset threshold βlow, to guarantee energy self-sustainability over time. Furthermore, A4 and A5 bound the maximum processing rate and workloads of each running container c, with c=1,⋯,Ct, respectively. Constraint A6 represents a hard-limit on the corresponding per-slot and per-VM processing time. A7 bounds the aggregate communication rate sustainable by the VLAN to rmax, and A8 ensures that the energy consumption at the computing site (due to the admitted computational workload) is bounded by the available energy in the EB. A9 forces the server to process the offloaded tasks within the set value τmax.From the optimization problem P1, it could be noted thatJζ,ψ,M,t consists of a nonconvex component defined in Eq. (7), while the others are convex and nondecreasing. In this case, Eq. (7) can be converted into a convex function using the GP concept [44], by introducing alternative variables and approximations. In this case, fixed parameters and approximations are introduced, i.e., μc,νc. In the sequel, the index t is dropped to improve readability. Thus, letting rc=2λc/τmax−Δ and then obtaining Pcrc in terms of λc by rearranging the Shannon-Hartley expression and substituting the value of rc:P^crc=2λc/τmax−Δ−νcWcln2/μcWc+lnN0c−lngc. From the Shannon-Hartley expression, the presence of the log-sum-exp function is observed as it has been proven to be convex in [45] and recall that Pcrc=expP^crc.To solve P1 in (12), the LLC principles [28, 31], GP technique [44], and heuristics is used towards obtaining the feasible system control inputs φt=δnt,Ct,ψct,Pct,λct,ζt,Mt for t=1,⋯,T. Note that (12) can iteratively be solved at any time slot t≥1, by just redefining the time horizon as t′=t,t+1,⋯,t+T−1.


### 4.2. Edge System Management

In this subsection, a traffic load and energy harvesting prediction method and an online management algorithm are proposed to solve the previously stated problem P1.


#### 4.2.1. Traffic Load and Energy Prediction

Given a time slot duration ofτ=30, the time series prediction is performed, i.e., the T=3 estimates of L^nt and H^t are obtained by using an LSTM developed in Python using Keras deep learning libraries (Sequential, Dense) where the network has a one-dimensional (1D) subsequence of data, single feature, and multistep for an output. The dataset is split as 70% for training and 30% for testing. The efficient Adam implementation of stochastic gradient descent and fit the model for 20 epochs with a batch size of 4 is used. As for the performance measure of the model, the RMSE is used.


#### 4.2.2. Edge System Dynamics

The system state vector at timet is denoted by qt=δt,Ct,Mt,bt, which contains the number of active BS, δt; number of active containers, Ct; transmission drivers for data transfers, Mt; and the EB level, bt. The input vector φt=δnt,Ct,ψct,Pct,λct,ζt,Mt drives the MEC server behavior (handles the joint switching on/off of BSs, autoscaling and reconfiguration of containers, and the tuning of transmission drivers) at time t. In this work, Pc∗t is obtained with CVXOPT toolbox, and λc∗t is obtained by following the procedure outlined in Remark 1 (M. Andersen and J. Dahl. CVXOPT: Python Software for Convex Programming, 2019. [Online]. Available: https://cvxopt.org/).The system behavior is described by the discrete-time state-space equation, adopting the LLC principles [14, 28]:
(13)qt+1=ϕqt,φt,where ϕ· is a behavioral model that captures the relationship between qt,φt and the next state qt+1. Note that this relationship accounts for the amount of energy drained θCOMMt,θMECt that harvested Ht and that purchased from the electrical grid Et, which together lead to the next buffer level βt+1 through Eq. (10). The GENM algorithm finds the best control action vector that yields the desired energy savings within the edge network. Specifically, for each time slot t, problem (12) is solved, obtaining control actions for the prediction horizon T. The control action that is applied at time t is φ∗t, which is the first one in the retrieved control sequence. This control amounts to setting the number of active BSs, δn∗t; number of instantiated containers, C∗t (along with their obtained ψc∗t, Pc∗t, λc∗t values); NIC status to either active or not, ζ∗t∈0,1; and the optimal transmission drivers, M∗t. The entire process is repeated every time slot t when the controller can adjust the behavior given the new state information.Stateqt and φt are, respectively, measured and applied at the beginning of the time slot t, whereas the offered load Lt and the harvested energy Ht are accumulated during the time slot and their value becomes known only by the end of it. This means that, being at the beginning of the time slot t, the system state at the next time slot t+1 can only be estimated, which is formally written as:
(14)q^t+1=ϕqt,φt.Remark 1 (container provisioning and load distribution).
For a fair provisioning of the computing resources,Ct, and the expected workload allocation, ξ^t+1, a remark is presented. Firstly, each container can only compute an amount of up to λmax, and to meet the latency requirements, Ct is obtained as Ct=ξ^t+1/λmax, where ∙ returns the nearest upper integer. Secondly, to distribute the workload among the Ct containers, a heuristic process splits the computational workload λct=λmax to the first Ct−1 containers, and the remaining workload λct=ξ^t+1−Ct−1λmax to the last one.


#### 4.2.3. Edge System Management Framework

In order to perform traffic load balancing using the green energy as a performance metric, a framework is defined that will identify the BS to be dynamically switched off and then steer the traffic load towards those BS with sufficient green energy. To do this, the available operating interval is defined as the ratio of the next time slot available green energy and the expected total power consumption (recall that the BS load is forecasted), per BS site, as(15)Int=bnt+1θBS,nt+1≥1.IfInt<1, the BS site will not have sufficient energy to handle the expected traffic and it becomes a potential BS to be switched off. In the case where Int≥1, the site energy will be sufficient to handle the expected traffic. The potential BS to be switched off, denoted by BS n, will offload its traffic load to a neighboring BS, denoted by BS nn′. For BS nn′ to be able to handle the offloaded traffic, the energy must be sufficient; thus, the green-based operating interval is defined as
(16)xnn′t=bnn′t+1θBS,nn′t+1,where θBS,nn′t+1 is the total energy consumption of the BS site when the traffic load from the neighboring BS is combined with the expected load of the BS, and bnn′t+1 is the next time slot energy. Next, the BS wake-up procedure is discussed.(1) BS Wake-Up Procedure. To support BS reactivation commands, the UE location fingerprints that are obtained from the LS API are considered. The UE trajectory is assumed to be sequential, i.e., from BS-to-BS along the same direction (while still associated with the MEC server), and this is represented as i1t→i2t→⋯→int, where int refers to the serving/target BS node n in association with user i, at time slot t. When a BS node is switched off, it goes into a discontinuous reception cycle and configures a timer to awake and listen. Here, the MEC server, as BSs manager, sends wake-up control information as wake-up signaling (the information is a single bit). The wake-up information is only sent during the listening period. A BS n can be woken up only if it meets the following conditions: (i) bnt+1>blow and (ii) a group of UEs that are associated with the MEC server is expected to receive the computed results via BS n (their trajectory is towards BS n as reported by the LS in the MEC).


#### 4.2.4. Green-Based Edge Network Management (GENM) Algorithm

In order for the algorithm to manage the BS system, deciding upon the allocation of their transmission resources and also handling the computing and communication process, the best control action,φt=δnt,Ct,ψct,Pct,λct,ζt,Mt, that will yield the expected system behavior shall be obtained.The edge network management algorithm pseudocode is outlined in Algorithm1, and it is based on the LLC principles from [14, 28]. Starting from the initial state, the controller constructs, in a breadth-first fashion, a tree comprising all possible future states up to the prediction depth T. The algorithm proceeds as follows: a search set S consisting of the current system state is initialized (line 01), and it is accumulated as the algorithm traverse through the tree (line 03), accounting for predictions, accumulated workloads at the output buffer, mobile devices trajectory int, past outputs and controls, and operating intervals. The set of states reached at every prediction depth t+k is referred to as St+k (line 02). Given qt, the workload L^int+k and harvested energy H^t+k are estimated first, then obtain the operating intervals Int+k,xnn′t+k (line 02), and generate the next set of reachable control actions by applying the accepted workload ξt+k, energy harvested, and green-based operating interval (line 03). The energy cost function corresponding to each generated state q^t+k is then computed (line 04), where q^t+k take into account of ηn as observed from Loutt. Once the prediction horizon is explored, a sequence of reachable states yielding minimum energy consumption is obtained (line 05). The control action φ∗t corresponding to q^t+k (the first state in this sequence) is provided as input to the system while the rest are discarded (line 06). The process is repeated at the beginning of each time slot t.Algorithm 1. GENM algorithm pseudocode.
Input:qt (current state)Output:φ∗t (control input vector)01: Parameter initializationSt=qt.02:for (k within the prediction horizon of depth T) do-L^int+k:= forecast the workload-H^nt+k:= forecast the energy-Int+k:= operating interval of each BS-xnn′t+k:= green-based operating interval-St+k=∅03:for (each qt in St+k) do- generate all reachable statesq^t+k-St+k=St+k∪q^t+k04:for (each q^t+k in St+k) do- calculate the correspondingθEDGEq^t+ktaking into account ofκn from Louttend forend forend for05: - obtain a sequence of reachable states yieldingminimum energy cost06:φ∗t≔ control leading from qt to q^min07:Returnφ∗t(1) Algorithm Complexity. The algorithm is executed at each time instance, and the corresponding time complexity is obtained as follows. The time complexity associated with the computation of the Int and xnn′ is linear with the size of the BS group ∣N∣ interconnected to the MEC server. Next, the complexity associated with updating the load allocation for the active BSs is ∣N∣−1, which leads to ON2. In the worst-case scenario (no BS has been switched off), the total complexity is ∣N∣qtφtT, which is linear in all variables, namely, number of BSs interconnected to the MEC server, number of system states, number of control actions, and time horizon T.


## 4.1. Optimization Problem

Our objective is to improve the overall energy savings of the edge system through BS power saving modes (i.e., green-based traffic load balancing), autoscaling of containers, contents caching, and tuning of the transmission drivers, and also to guarantee the QoS within the network. Note that at the end of each time slot, the EB states are updated depending on the harvested energy and the consumed energy, thereby linking per-time slot problems across time.To achieve our objective, two cost functions are defined, one captures the edge system energy consumption and the other handles the QoS. This is defined as (F1)θEDGEt, which weighs the energy consumption due to transmission in the BSs and the computing-plus-communication activities in the MEC server and (F2) a quadratic termξt−Lint2, which accounts for the QoS. In this regard, it is worth noting that F1 tends to push the system towards self-sustainability solutions and F2 favors solutions where the delay-sensitive load is entirely admitted in the MEC server by the router application, taking into account the expected energy to be harvested in the computing site. A weight Γ=0,1 is utilized to balance the two objectives F1 and F2. The corresponding (weighted) cost function is defined as:
(11)Jδ,ψ,M,t=ΔΓ¯θEDGEδnt,ψct,Mt,t+Γξt−Lint2,where Γ¯=Δ1−Γ. Hence, starting from t=1 (i.e., t=1,2,⋯,T) as the current time slot and the finite horizon T, the following optimization problem is formulated as:
(12)P1:minE∑t=1TJδ,ψ,M,t,subject to:A1:δnt∈ε,1,A2:β≤Ct≤C,A3:bt≥blow,A4:0≤fct≤fmax,A5:0≤λct≤λmax,A6:χct≤Δ,A7:∑c=1Ctrct≤rmax,A8:θMECt≤bt,A9:max2ρct+Δ=τmax,t=1,⋯,T,where the set of objective variables to be configured at slott in the BS system and MEC server is defined as E=Δδnt,Ct,ψct,Pct,λct,ζt,Mt. The setting handles the transmission and computing-plus-communication activities. Constraint A1 specifies the BS operation status (either power saving or active), and A2 forces the required number of containers, Ct, to be always greater than or equal to a minimum number β≥1: the purpose of this is to be always able to handle mission-critical communications. A3 makes sure that the EB level is always above or equal to a preset threshold βlow, to guarantee energy self-sustainability over time. Furthermore, A4 and A5 bound the maximum processing rate and workloads of each running container c, with c=1,⋯,Ct, respectively. Constraint A6 represents a hard-limit on the corresponding per-slot and per-VM processing time. A7 bounds the aggregate communication rate sustainable by the VLAN to rmax, and A8 ensures that the energy consumption at the computing site (due to the admitted computational workload) is bounded by the available energy in the EB. A9 forces the server to process the offloaded tasks within the set value τmax.From the optimization problem P1, it could be noted thatJζ,ψ,M,t consists of a nonconvex component defined in Eq. (7), while the others are convex and nondecreasing. In this case, Eq. (7) can be converted into a convex function using the GP concept [44], by introducing alternative variables and approximations. In this case, fixed parameters and approximations are introduced, i.e., μc,νc. In the sequel, the index t is dropped to improve readability. Thus, letting rc=2λc/τmax−Δ and then obtaining Pcrc in terms of λc by rearranging the Shannon-Hartley expression and substituting the value of rc:P^crc=2λc/τmax−Δ−νcWcln2/μcWc+lnN0c−lngc. From the Shannon-Hartley expression, the presence of the log-sum-exp function is observed as it has been proven to be convex in [45] and recall that Pcrc=expP^crc.To solve P1 in (12), the LLC principles [28, 31], GP technique [44], and heuristics is used towards obtaining the feasible system control inputs φt=δnt,Ct,ψct,Pct,λct,ζt,Mt for t=1,⋯,T. Note that (12) can iteratively be solved at any time slot t≥1, by just redefining the time horizon as t′=t,t+1,⋯,t+T−1.


## 4.2. Edge System Management

In this subsection, a traffic load and energy harvesting prediction method and an online management algorithm are proposed to solve the previously stated problem P1.


### 4.2.1. Traffic Load and Energy Prediction

Given a time slot duration ofτ=30, the time series prediction is performed, i.e., the T=3 estimates of L^nt and H^t are obtained by using an LSTM developed in Python using Keras deep learning libraries (Sequential, Dense) where the network has a one-dimensional (1D) subsequence of data, single feature, and multistep for an output. The dataset is split as 70% for training and 30% for testing. The efficient Adam implementation of stochastic gradient descent and fit the model for 20 epochs with a batch size of 4 is used. As for the performance measure of the model, the RMSE is used.


### 4.2.2. Edge System Dynamics

The system state vector at timet is denoted by qt=δt,Ct,Mt,bt, which contains the number of active BS, δt; number of active containers, Ct; transmission drivers for data transfers, Mt; and the EB level, bt. The input vector φt=δnt,Ct,ψct,Pct,λct,ζt,Mt drives the MEC server behavior (handles the joint switching on/off of BSs, autoscaling and reconfiguration of containers, and the tuning of transmission drivers) at time t. In this work, Pc∗t is obtained with CVXOPT toolbox, and λc∗t is obtained by following the procedure outlined in Remark 1 (M. Andersen and J. Dahl. CVXOPT: Python Software for Convex Programming, 2019. [Online]. Available: https://cvxopt.org/).The system behavior is described by the discrete-time state-space equation, adopting the LLC principles [14, 28]:
(13)qt+1=ϕqt,φt,where ϕ· is a behavioral model that captures the relationship between qt,φt and the next state qt+1. Note that this relationship accounts for the amount of energy drained θCOMMt,θMECt that harvested Ht and that purchased from the electrical grid Et, which together lead to the next buffer level βt+1 through Eq. (10). The GENM algorithm finds the best control action vector that yields the desired energy savings within the edge network. Specifically, for each time slot t, problem (12) is solved, obtaining control actions for the prediction horizon T. The control action that is applied at time t is φ∗t, which is the first one in the retrieved control sequence. This control amounts to setting the number of active BSs, δn∗t; number of instantiated containers, C∗t (along with their obtained ψc∗t, Pc∗t, λc∗t values); NIC status to either active or not, ζ∗t∈0,1; and the optimal transmission drivers, M∗t. The entire process is repeated every time slot t when the controller can adjust the behavior given the new state information.Stateqt and φt are, respectively, measured and applied at the beginning of the time slot t, whereas the offered load Lt and the harvested energy Ht are accumulated during the time slot and their value becomes known only by the end of it. This means that, being at the beginning of the time slot t, the system state at the next time slot t+1 can only be estimated, which is formally written as:
(14)q^t+1=ϕqt,φt.Remark 1 (container provisioning and load distribution).
For a fair provisioning of the computing resources,Ct, and the expected workload allocation, ξ^t+1, a remark is presented. Firstly, each container can only compute an amount of up to λmax, and to meet the latency requirements, Ct is obtained as Ct=ξ^t+1/λmax, where ∙ returns the nearest upper integer. Secondly, to distribute the workload among the Ct containers, a heuristic process splits the computational workload λct=λmax to the first Ct−1 containers, and the remaining workload λct=ξ^t+1−Ct−1λmax to the last one.


### 4.2.3. Edge System Management Framework

In order to perform traffic load balancing using the green energy as a performance metric, a framework is defined that will identify the BS to be dynamically switched off and then steer the traffic load towards those BS with sufficient green energy. To do this, the available operating interval is defined as the ratio of the next time slot available green energy and the expected total power consumption (recall that the BS load is forecasted), per BS site, as(15)Int=bnt+1θBS,nt+1≥1.IfInt<1, the BS site will not have sufficient energy to handle the expected traffic and it becomes a potential BS to be switched off. In the case where Int≥1, the site energy will be sufficient to handle the expected traffic. The potential BS to be switched off, denoted by BS n, will offload its traffic load to a neighboring BS, denoted by BS nn′. For BS nn′ to be able to handle the offloaded traffic, the energy must be sufficient; thus, the green-based operating interval is defined as
(16)xnn′t=bnn′t+1θBS,nn′t+1,where θBS,nn′t+1 is the total energy consumption of the BS site when the traffic load from the neighboring BS is combined with the expected load of the BS, and bnn′t+1 is the next time slot energy. Next, the BS wake-up procedure is discussed.(1) BS Wake-Up Procedure. To support BS reactivation commands, the UE location fingerprints that are obtained from the LS API are considered. The UE trajectory is assumed to be sequential, i.e., from BS-to-BS along the same direction (while still associated with the MEC server), and this is represented as i1t→i2t→⋯→int, where int refers to the serving/target BS node n in association with user i, at time slot t. When a BS node is switched off, it goes into a discontinuous reception cycle and configures a timer to awake and listen. Here, the MEC server, as BSs manager, sends wake-up control information as wake-up signaling (the information is a single bit). The wake-up information is only sent during the listening period. A BS n can be woken up only if it meets the following conditions: (i) bnt+1>blow and (ii) a group of UEs that are associated with the MEC server is expected to receive the computed results via BS n (their trajectory is towards BS n as reported by the LS in the MEC).


### 4.2.4. Green-Based Edge Network Management (GENM) Algorithm

In order for the algorithm to manage the BS system, deciding upon the allocation of their transmission resources and also handling the computing and communication process, the best control action,φt=δnt,Ct,ψct,Pct,λct,ζt,Mt, that will yield the expected system behavior shall be obtained.The edge network management algorithm pseudocode is outlined in Algorithm1, and it is based on the LLC principles from [14, 28]. Starting from the initial state, the controller constructs, in a breadth-first fashion, a tree comprising all possible future states up to the prediction depth T. The algorithm proceeds as follows: a search set S consisting of the current system state is initialized (line 01), and it is accumulated as the algorithm traverse through the tree (line 03), accounting for predictions, accumulated workloads at the output buffer, mobile devices trajectory int, past outputs and controls, and operating intervals. The set of states reached at every prediction depth t+k is referred to as St+k (line 02). Given qt, the workload L^int+k and harvested energy H^t+k are estimated first, then obtain the operating intervals Int+k,xnn′t+k (line 02), and generate the next set of reachable control actions by applying the accepted workload ξt+k, energy harvested, and green-based operating interval (line 03). The energy cost function corresponding to each generated state q^t+k is then computed (line 04), where q^t+k take into account of ηn as observed from Loutt. Once the prediction horizon is explored, a sequence of reachable states yielding minimum energy consumption is obtained (line 05). The control action φ∗t corresponding to q^t+k (the first state in this sequence) is provided as input to the system while the rest are discarded (line 06). The process is repeated at the beginning of each time slot t.Algorithm 1. GENM algorithm pseudocode.
Input:qt (current state)Output:φ∗t (control input vector)01: Parameter initializationSt=qt.02:for (k within the prediction horizon of depth T) do-L^int+k:= forecast the workload-H^nt+k:= forecast the energy-Int+k:= operating interval of each BS-xnn′t+k:= green-based operating interval-St+k=∅03:for (each qt in St+k) do- generate all reachable statesq^t+k-St+k=St+k∪q^t+k04:for (each q^t+k in St+k) do- calculate the correspondingθEDGEq^t+ktaking into account ofκn from Louttend forend forend for05: - obtain a sequence of reachable states yieldingminimum energy cost06:φ∗t≔ control leading from qt to q^min07:Returnφ∗t(1) Algorithm Complexity. The algorithm is executed at each time instance, and the corresponding time complexity is obtained as follows. The time complexity associated with the computation of the Int and xnn′ is linear with the size of the BS group ∣N∣ interconnected to the MEC server. Next, the complexity associated with updating the load allocation for the active BSs is ∣N∣−1, which leads to ON2. In the worst-case scenario (no BS has been switched off), the total complexity is ∣N∣qtφtT, which is linear in all variables, namely, number of BSs interconnected to the MEC server, number of system states, number of control actions, and time horizon T.


## 4.2.1. Traffic Load and Energy Prediction

Given a time slot duration ofτ=30, the time series prediction is performed, i.e., the T=3 estimates of L^nt and H^t are obtained by using an LSTM developed in Python using Keras deep learning libraries (Sequential, Dense) where the network has a one-dimensional (1D) subsequence of data, single feature, and multistep for an output. The dataset is split as 70% for training and 30% for testing. The efficient Adam implementation of stochastic gradient descent and fit the model for 20 epochs with a batch size of 4 is used. As for the performance measure of the model, the RMSE is used.


## 4.2.2. Edge System Dynamics

The system state vector at timet is denoted by qt=δt,Ct,Mt,bt, which contains the number of active BS, δt; number of active containers, Ct; transmission drivers for data transfers, Mt; and the EB level, bt. The input vector φt=δnt,Ct,ψct,Pct,λct,ζt,Mt drives the MEC server behavior (handles the joint switching on/off of BSs, autoscaling and reconfiguration of containers, and the tuning of transmission drivers) at time t. In this work, Pc∗t is obtained with CVXOPT toolbox, and λc∗t is obtained by following the procedure outlined in Remark 1 (M. Andersen and J. Dahl. CVXOPT: Python Software for Convex Programming, 2019. [Online]. Available: https://cvxopt.org/).The system behavior is described by the discrete-time state-space equation, adopting the LLC principles [14, 28]:
(13)qt+1=ϕqt,φt,where ϕ· is a behavioral model that captures the relationship between qt,φt and the next state qt+1. Note that this relationship accounts for the amount of energy drained θCOMMt,θMECt that harvested Ht and that purchased from the electrical grid Et, which together lead to the next buffer level βt+1 through Eq. (10). The GENM algorithm finds the best control action vector that yields the desired energy savings within the edge network. Specifically, for each time slot t, problem (12) is solved, obtaining control actions for the prediction horizon T. The control action that is applied at time t is φ∗t, which is the first one in the retrieved control sequence. This control amounts to setting the number of active BSs, δn∗t; number of instantiated containers, C∗t (along with their obtained ψc∗t, Pc∗t, λc∗t values); NIC status to either active or not, ζ∗t∈0,1; and the optimal transmission drivers, M∗t. The entire process is repeated every time slot t when the controller can adjust the behavior given the new state information.Stateqt and φt are, respectively, measured and applied at the beginning of the time slot t, whereas the offered load Lt and the harvested energy Ht are accumulated during the time slot and their value becomes known only by the end of it. This means that, being at the beginning of the time slot t, the system state at the next time slot t+1 can only be estimated, which is formally written as:
(14)q^t+1=ϕqt,φt.Remark 1 (container provisioning and load distribution).
For a fair provisioning of the computing resources,Ct, and the expected workload allocation, ξ^t+1, a remark is presented. Firstly, each container can only compute an amount of up to λmax, and to meet the latency requirements, Ct is obtained as Ct=ξ^t+1/λmax, where ∙ returns the nearest upper integer. Secondly, to distribute the workload among the Ct containers, a heuristic process splits the computational workload λct=λmax to the first Ct−1 containers, and the remaining workload λct=ξ^t+1−Ct−1λmax to the last one.


## 4.2.3. Edge System Management Framework

In order to perform traffic load balancing using the green energy as a performance metric, a framework is defined that will identify the BS to be dynamically switched off and then steer the traffic load towards those BS with sufficient green energy. To do this, the available operating interval is defined as the ratio of the next time slot available green energy and the expected total power consumption (recall that the BS load is forecasted), per BS site, as(15)Int=bnt+1θBS,nt+1≥1.IfInt<1, the BS site will not have sufficient energy to handle the expected traffic and it becomes a potential BS to be switched off. In the case where Int≥1, the site energy will be sufficient to handle the expected traffic. The potential BS to be switched off, denoted by BS n, will offload its traffic load to a neighboring BS, denoted by BS nn′. For BS nn′ to be able to handle the offloaded traffic, the energy must be sufficient; thus, the green-based operating interval is defined as
(16)xnn′t=bnn′t+1θBS,nn′t+1,where θBS,nn′t+1 is the total energy consumption of the BS site when the traffic load from the neighboring BS is combined with the expected load of the BS, and bnn′t+1 is the next time slot energy. Next, the BS wake-up procedure is discussed.(1) BS Wake-Up Procedure. To support BS reactivation commands, the UE location fingerprints that are obtained from the LS API are considered. The UE trajectory is assumed to be sequential, i.e., from BS-to-BS along the same direction (while still associated with the MEC server), and this is represented as i1t→i2t→⋯→int, where int refers to the serving/target BS node n in association with user i, at time slot t. When a BS node is switched off, it goes into a discontinuous reception cycle and configures a timer to awake and listen. Here, the MEC server, as BSs manager, sends wake-up control information as wake-up signaling (the information is a single bit). The wake-up information is only sent during the listening period. A BS n can be woken up only if it meets the following conditions: (i) bnt+1>blow and (ii) a group of UEs that are associated with the MEC server is expected to receive the computed results via BS n (their trajectory is towards BS n as reported by the LS in the MEC).


## 4.2.4. Green-Based Edge Network Management (GENM) Algorithm

In order for the algorithm to manage the BS system, deciding upon the allocation of their transmission resources and also handling the computing and communication process, the best control action,φt=δnt,Ct,ψct,Pct,λct,ζt,Mt, that will yield the expected system behavior shall be obtained.The edge network management algorithm pseudocode is outlined in Algorithm1, and it is based on the LLC principles from [14, 28]. Starting from the initial state, the controller constructs, in a breadth-first fashion, a tree comprising all possible future states up to the prediction depth T. The algorithm proceeds as follows: a search set S consisting of the current system state is initialized (line 01), and it is accumulated as the algorithm traverse through the tree (line 03), accounting for predictions, accumulated workloads at the output buffer, mobile devices trajectory int, past outputs and controls, and operating intervals. The set of states reached at every prediction depth t+k is referred to as St+k (line 02). Given qt, the workload L^int+k and harvested energy H^t+k are estimated first, then obtain the operating intervals Int+k,xnn′t+k (line 02), and generate the next set of reachable control actions by applying the accepted workload ξt+k, energy harvested, and green-based operating interval (line 03). The energy cost function corresponding to each generated state q^t+k is then computed (line 04), where q^t+k take into account of ηn as observed from Loutt. Once the prediction horizon is explored, a sequence of reachable states yielding minimum energy consumption is obtained (line 05). The control action φ∗t corresponding to q^t+k (the first state in this sequence) is provided as input to the system while the rest are discarded (line 06). The process is repeated at the beginning of each time slot t.Algorithm 1. GENM algorithm pseudocode.
Input:qt (current state)Output:φ∗t (control input vector)01: Parameter initializationSt=qt.02:for (k within the prediction horizon of depth T) do-L^int+k:= forecast the workload-H^nt+k:= forecast the energy-Int+k:= operating interval of each BS-xnn′t+k:= green-based operating interval-St+k=∅03:for (each qt in St+k) do- generate all reachable statesq^t+k-St+k=St+k∪q^t+k04:for (each q^t+k in St+k) do- calculate the correspondingθEDGEq^t+ktaking into account ofκn from Louttend forend forend for05: - obtain a sequence of reachable states yieldingminimum energy cost06:φ∗t≔ control leading from qt to q^min07:Returnφ∗t(1) Algorithm Complexity. The algorithm is executed at each time instance, and the corresponding time complexity is obtained as follows. The time complexity associated with the computation of the Int and xnn′ is linear with the size of the BS group ∣N∣ interconnected to the MEC server. Next, the complexity associated with updating the load allocation for the active BSs is ∣N∣−1, which leads to ON2. In the worst-case scenario (no BS has been switched off), the total complexity is ∣N∣qtφtT, which is linear in all variables, namely, number of BSs interconnected to the MEC server, number of system states, number of control actions, and time horizon T.


## 5. Performance Evaluation

In this section, some selected numerical results for the scenario of Section3 are shown. The parameters that were used in the simulations are listed in Table 2.Table 2
System parameters.
ParameterValueMax. number of containers,C20Min. number of containers,β1Time slot duration,τ30Idle state energy for containerc,θidlect4Max. energy for containerc,θmax,mt10Per-container reconfiguration cost,ze0.005 J/(MHz)2TOE in idle state,θidleNICt13.1 JMax. allowed processing time,Δ0.8Processing rate set,fct{0,50,70,90,105}Bandwidth,Wc1 MHzMax. number of drivers,M6Noise spectral density,N0c-174 dBm/HzMax. containerc load, λmax10 MBNIC best performance throughput,η1.4 Gbit/JDriver energy,dmt1 J/sTarget transmission rate,r01 MbpsControllable factor of delay,u0.96Reconfiguration overhead,σ20 msLeakage energy,at2μJEnergy storage capacity,bmax490Lower energy threshold,blow30% ofbmaxUpper energy threshold,bup70% ofbmax


### 5.1. Simulation Setup

A virtualized MEC server in proximity to a group of BS is considered. The BS coverage areas overlap to enable load balancing. Our time slot durationτ is set to 30, and the time horizon is set to T=3 time slots. For simulation, Python is used as the programming language.


### 5.2. Numerical Results

Data preparation: The information from the used mobile and energy traces is aggregated to the set time slot duration. The mobile traces are aggregated from 10 observation time to τ. As for the wind and solar traces, they were aggregated from 15 observation time to τ. The used datasets are readily available in a public repository (see [39]).In Figure4, the real and predicted values for BS traffic load and harvested energy is shown. Here, the forecasting routing tracks each value and predicts it over one-step. The shown selected prediction results are for Cluster 3, Solar 3, and Wind 3. Then, Table 3 shows the average RMSE of the normalized harvested energy and traffic load processes, for different time horizon values, T∈1,2,3. In the table, the term Hwindt represents the forecasted values for energy harvested from wind turbines, and Hsolart is for the harvested energy from solar panels. From the obtained results, the prediction variations are observed between Ht and Lt when comparing the average RMSE. The measured accuracy is deemed good enough for the proposed optimization.Figure 4
One-step ahead predictive values forLt and Ht.Table 3
Average prediction error (RMSE) for harvested energy and traffic load processes, both normalized in [0,1].
T=1T=2T=3Lt0.0100.0130.018Hwindt0.0110.0130.016Hsolart0.0100.0110.014The GENM algorithm is benchmarked with another one, named Iterative-based Resource Manager with network impact Capability (IRMC), which is inspired by the iterative approach for computing platforms from [20] and the use of the network impact towards load balancing from [5]. Both algorithms make use of the learned information. Figures 5 and 6 show the average energy savings obtained by GENM in the MEC server. In Figure 5(a), the average results for GENM (ze=0.005,∣N∣=24,Γ=0.5,λmax=10MB) show energy savings of 59%, while IRMC achieves 34% on average. As expected, the highest energy savings gain is observed in the early hours of the day (1-8) as the aggregated computational workload was at its lowest. In Figure 5(b), the average energy savings obtained by GENM is 68% (ze=0.005,∣N∣=12,Γ=0.5,λmax=10MB) and for IRMC is 49%. Again, here, the highest peaks for energy savings are obtained from 1-8. The results are obtained with respect to the case where no energy management procedures are applied, i.e., the MEC server provisions the computing resources for maximum expected computation workload (maximum value of θMECt,C=20,∀t). Comparing the results of Figures 5(a) and 5(b), we observed that when the BSs being manage by the MEC server are reduced (i.e., 12<24), the aggregated delay-sensitive workload is also reduced and this translates to reduced computation process demands, which in turn results into high energy savings.Figure 5
Mean energy savings within the MEC server.
(a)
Mean energy savings forΓ=0.5,∣N∣=24,λmax=10MB(b)
Mean energy savings forΓ=0.5,∣N∣=12,λmax=10MBFigure 6
Energy savings versus BS group size.Figure6 shows the average energy savings obtained when green energy is used as a performance metric towards load balancing within a group of BSs. Here, the group size is increased from N=5to50, using an incremental step size of 5. The obtained energy savings are with respect to the case where all BSs are dimensioned for maximum expected capacity (maximum value of θCOMMt). From the results, it is observed that the energy savings increase as the BS cluster grows, thanks to the load balancing among active BSs.Then, Figure7 shows the average energy savings for the edge system. Here, the BS group size is set to ∣N∣=40, and the obtained energy saving results are with respect to the case where no energy management procedures are applied, i.e., the BSs are dimensioned for maximum expected capacity (maximum value of θCOMMt, ∀t) and the MEC server provisions the computing resources for the maximum expected computation workload (maximum value of θMECt, with C=20containers,∀t). The average results of GENM (ze=0.05,λmax=10MB, Γ=0.5) show energy savings of 51%, while IRMC achieves 44% on average. The effectiveness of the joint dynamic BS management, autoscaling and reconfiguration of the computing resources, and on/off switching of the fast tunable laser drivers, coupled with foresighted optimization, is observed in the obtained numerical results.Figure 7
Mean energy savings for the edge system.


## 5.1. Simulation Setup

A virtualized MEC server in proximity to a group of BS is considered. The BS coverage areas overlap to enable load balancing. Our time slot durationτ is set to 30, and the time horizon is set to T=3 time slots. For simulation, Python is used as the programming language.


## 5.2. Numerical Results

Data preparation: The information from the used mobile and energy traces is aggregated to the set time slot duration. The mobile traces are aggregated from 10 observation time to τ. As for the wind and solar traces, they were aggregated from 15 observation time to τ. The used datasets are readily available in a public repository (see [39]).In Figure4, the real and predicted values for BS traffic load and harvested energy is shown. Here, the forecasting routing tracks each value and predicts it over one-step. The shown selected prediction results are for Cluster 3, Solar 3, and Wind 3. Then, Table 3 shows the average RMSE of the normalized harvested energy and traffic load processes, for different time horizon values, T∈1,2,3. In the table, the term Hwindt represents the forecasted values for energy harvested from wind turbines, and Hsolart is for the harvested energy from solar panels. From the obtained results, the prediction variations are observed between Ht and Lt when comparing the average RMSE. The measured accuracy is deemed good enough for the proposed optimization.Figure 4
One-step ahead predictive values forLt and Ht.Table 3
Average prediction error (RMSE) for harvested energy and traffic load processes, both normalized in [0,1].
T=1T=2T=3Lt0.0100.0130.018Hwindt0.0110.0130.016Hsolart0.0100.0110.014The GENM algorithm is benchmarked with another one, named Iterative-based Resource Manager with network impact Capability (IRMC), which is inspired by the iterative approach for computing platforms from [20] and the use of the network impact towards load balancing from [5]. Both algorithms make use of the learned information. Figures 5 and 6 show the average energy savings obtained by GENM in the MEC server. In Figure 5(a), the average results for GENM (ze=0.005,∣N∣=24,Γ=0.5,λmax=10MB) show energy savings of 59%, while IRMC achieves 34% on average. As expected, the highest energy savings gain is observed in the early hours of the day (1-8) as the aggregated computational workload was at its lowest. In Figure 5(b), the average energy savings obtained by GENM is 68% (ze=0.005,∣N∣=12,Γ=0.5,λmax=10MB) and for IRMC is 49%. Again, here, the highest peaks for energy savings are obtained from 1-8. The results are obtained with respect to the case where no energy management procedures are applied, i.e., the MEC server provisions the computing resources for maximum expected computation workload (maximum value of θMECt,C=20,∀t). Comparing the results of Figures 5(a) and 5(b), we observed that when the BSs being manage by the MEC server are reduced (i.e., 12<24), the aggregated delay-sensitive workload is also reduced and this translates to reduced computation process demands, which in turn results into high energy savings.Figure 5
Mean energy savings within the MEC server.
(a)
Mean energy savings forΓ=0.5,∣N∣=24,λmax=10MB(b)
Mean energy savings forΓ=0.5,∣N∣=12,λmax=10MBFigure 6
Energy savings versus BS group size.Figure6 shows the average energy savings obtained when green energy is used as a performance metric towards load balancing within a group of BSs. Here, the group size is increased from N=5to50, using an incremental step size of 5. The obtained energy savings are with respect to the case where all BSs are dimensioned for maximum expected capacity (maximum value of θCOMMt). From the results, it is observed that the energy savings increase as the BS cluster grows, thanks to the load balancing among active BSs.Then, Figure7 shows the average energy savings for the edge system. Here, the BS group size is set to ∣N∣=40, and the obtained energy saving results are with respect to the case where no energy management procedures are applied, i.e., the BSs are dimensioned for maximum expected capacity (maximum value of θCOMMt, ∀t) and the MEC server provisions the computing resources for the maximum expected computation workload (maximum value of θMECt, with C=20containers,∀t). The average results of GENM (ze=0.05,λmax=10MB, Γ=0.5) show energy savings of 51%, while IRMC achieves 44% on average. The effectiveness of the joint dynamic BS management, autoscaling and reconfiguration of the computing resources, and on/off switching of the fast tunable laser drivers, coupled with foresighted optimization, is observed in the obtained numerical results.Figure 7
Mean energy savings for the edge system.


## 6. Conclusions

This paper envisioned an edge system where a group of BS is placed in proximity to a MEC server for ease of handling the offloaded computational workload and BS management, and also, the edge apparatuses are powered by hybrid supplies, i.e., green energy is used in order to promote energy self-sustainability within the network and as a performance metric for traffic load balancing. The extra energy can only be purchased from the grid supply to supplement the renewable energy supplies. The considered energy cost model takes into account the computing, caching, and communication processes within the MEC server, and the transmission-related energy consumption in BSs. To intelligently manage the edge system, an online algorithm based on forecasting, control theory, and heuristics is proposed with the goal of minimizing the overall energy consumption and to guarantee the quality of service within the network. The algorithm jointly performs (i) dynamic BS management using green energy as a performance metric; (ii) autoscaling and reconfiguration of the computing resources, workload, and processing rate allocation; and lastly, (iii) switching on/off of fast tunable drivers. Numerical results, obtained with real-world energy and traffic load traces, demonstrate that the proposed algorithm achieves energy savings of above 50% with respect to the allocated maximum per-container loads of 10 MB. The computing platform is able to achieve energy savings from 59% to 68%, depending on the size of the BS cluster. The energy-saving results are obtained with respect to the case where no energy management techniques are applied in the BS system and the MEC server.



---

*Source: 8825396-2020-12-15.xml*