# A Practical Framework for Energy-Efficient Node Activation in Heterogeneous LTE Networks

Authors: Antonio Virdis; Giovanni Stea; Dario Sabella; Marco Caretti
Journal: Mobile Information Systems
 (2017)

---

## Abstract

This paper presents a framework to activate and deactivate micronodes in a heterogeneous multicell LTE network, based on load and energy efficiency consideration. The framework exploits historical data (i.e., per-macro-cell load curves) to select a set of candidate switch-on/switch-off instants of microcells, assuming alimitednumber of state changes is allowed in a day. The switching instants are instead determinedonline, by taking into account the actual traffic as well as the load curves. Moreover, intercell interference is fully accounted for. Our simulations show that this framework allows a multicell network to sustain peak-hour load when necessary and to reconfigure to a minimum coverage baseline whenever feasible, thus saving power (up to 25% in our scenarios). Moreover, the framework isrobust, meaning that deviations of the actual traffic with respect to the prediction offered by the load curves can easily be handled.


---

## Body

## 1. Introduction

Computer networks have become a major contributor to electricity consumption all over the world ‎[1], which has fostered a large amount of research on how to make them more energy-efficient. This problem can be targeted from several, concurrent points of view: on the one hand, working at the hardware level to make network equipment more energy efficient; on the other hand, designing algorithms that exploit the inherentredundancyof computer networks and temporarily switch off unused or unnecessary portions of the networks during off-peak hours. This allows networks to be designed for peak, while operating at lower-than-peak power most of the time. As far as cellular networks are concerned, energy efficiency in LTE networks has been the subject of several studies. Leaving aside those that investigate battery saving at theuserside (e.g., ‎[2–4]), many works concentrate on theinfrastructureside, which is where energy consumption meets OPEX (see, e.g., ‎[5–9] and the references therein and again ‎[10–18]).LTE power saving schemes exploit the fact that cellular coverage isoverlapping, due to both reliability and (mostly) performance design issues and that traffic load ishighly variable. Coverage is designed to carrypeak-hourloads, which normally occur during working hours in business areas, while nodes use very few resources during (long) off-peak periods, for example, at night. Therefore, some nodes can switch off during off-peak hours, and nearby nodes will increase their transmission power accordingly to guarantee coverage. When the load is high, the first node will be reactivated and the second one will revert to normal radius. A similar concept can be applied toheterogeneous cellular network deployments, where amacronode presides over a relatively large area andmicronodes provide additional localized capacity to cover hotspots within the former. Micronodes can be switched on to offload the macro node (which is supposed to remain always on, lest coverage holes appear) when necessary or profitable from an energy efficiency point of view.Switch-off schemes—for both macro nodes, in homogeneous settings, and micronodes in heterogeneous settings—can be eitherofflineoronline‎[5]. Offline schemes rely on knowing the evolutions of load in time (the so-calledload curves,e.g., ‎[19]) and decide the optimal switch-on/switch-off intervals based on the shape of these curves. They are often used for resource provisioning purposes and, being based onload predictions, which may always prove wrong, need anonlinecounterpart to react to unforeseen circumstances (flash crowds, accidents, etc.).Online algorithms look at thecurrentload and decide when and where it is high enough to warrant a switch-on (or low enough for a switch-off) of some nodes. Online schemes normally work at coarse timescales, for example, 15-minute slots, because switching operations are not instantaneous (in fact, they may take up to minutes ‎[10]). Moreover, the ensuing topology alterations may trigger avalanche effects, such as massive handovers, or cause interference ripples in neighboring cells, which are difficult to predict and manage. For instance, switching off a micro node at locationxmay indeed increase the load innearby areas, because the macro node at locationxtakes over its load and hence generates more interference around itself.Most works in the existing literature usually make some assumptions which render their scheme impractical: the first one is that the number of per-node switch operations isunlimited. While we can expect future base stations to be able to tolerate frequent power transitions, most of the ones that are currently operational have not been designed with this characteristic ‎[6]. This means that frequent switches will decrease their energy efficiency and possibly decrease their mean time between failures (MTBF), something that has a significant impact on OPEX. This calls for algorithms that can maximize the power saved by the network while limiting the number of power transitions for the node to a predefined maximum. Another frequent assumption is to rely on a particularshape of the load curves (e.g., one with a clearly distinguishable peak-hour and off-peak valley). While this allows one to build an optimal algorithm, the same algorithm would make suboptimal decisions when the shape of the load curve is different. Several algorithms assumeclairvoyance: for instance, they assume that they know the exact amount of resources that a micro, currently switched off, will use to serve a given input traffic. This cannot be known in advance in practice and depends on the user positions and their perceived interference. Finally, they often makelocaldecisions neglecting theirglobaleffects: as anticipated, switching off a micro (based on local knowledge, e.g., few users connected to it) will save the network owner the power to operate it; however, the extra resources required to serve the same users at the macro will foster a higher interinterference in the surrounding area, whose net effect may well be to increase the power consumption. Therefore, theglobal effects of switching decisions should be considered.In this paper, we design and evaluate anoffline-online framework for heterogeneous cellular networks, whose job is to satisfy a given load demand with the minimum possible energy cost. Our framework exploits offline information, that is, load curves, to select which configurations arefeasible (i.e., able to carry the nominal load) at a given point in time. An optimization algorithm is then run on the set of feasible configurations, to decide the optimal switch-on points for the micro nodes, constrained to amaximum number of switch-ons in a day. To the best of our knowledge, ours is the only work that takes the latter aspect into account. The only work we know of that considers the number of switch-ons, that is, ‎[20], seeking to minimize the latter rather than taking it as a constraint. Theonlinepart of the framework, instead, measures the current load andenforcesswitch-on/off of micro nodes so as to satisfy the demand in an energy-efficient way. More specifically, it may bring forward or postpone switching instants with respect to the optimal ones computed offline, when it is more energetically efficient to do so. The scheme described in this paper is designed so as to require few hypotheses: it can work with load curves of arbitrary shape, it does not assume clairvoyance, and it takes into account global effects of switching decisions through an accurate estimation of intercell interference. We evaluate our scheme using both synthetic and real-life load curves taken from operator data. Our results show that power savings up to 25% are achievable over a whole day and that neglecting intercell interference leads to overoptimistic results. Moreover, we show that our scheme isrobust against variation in the expected load.The rest of the paper is organized as follows: in Section2 we describe our assumptions and state our problem formally. The related work is discussed in detail in Section 3. Section 4 describes the proposed framework, which is evaluated in Section 5. Finally, Section 6 concludes the paper.


## 2. System Model and Problem Statement

In LTE, cell transmissions are arranged in time slots called Transmission Time Intervals, (TTIs), whose duration is 1 ms. In a TTI, the node allocatesframes,that is, vectors of(Virtual) Resource Blocks(RBs) to its associated user equipment (UE) ‎[21]. UE is associated withonenode at a time.Each RB carries a fixed number of symbols, which translate to different amounts of bits depending on themodulation and coding scheme (MCS) used by the antenna on that RB. In general, more information-dense modulations (e.g., 64QAM, yielding 6 bits per symbol) are favored when a better channel to the UE is perceived (i.e., one with less interference). The quality of the wireless channel varies over both time and frequency. For this reason, UE reports their perceived downlink channel state to the eNodeB as a Channel Quality Indicator (CQI), computed according to the measured Signal to Interference and Noise Ratio (SINR).We consider a large-scale multicell LTE network as shown in Figure1. The network is logically divided into N areas (calledhexagons henceforth for simplicity), each one under the coverage of amacro node(M in the figure), that is, an eNodeB with high radiation capabilities. Macro nodes are anisotropic, as shown in Figure 1. A hexagon h can also accommodate Lhmicro nodes(m), 1≤h≤N, Lh=2, in the figure. The latter, usually located in hotspots or at hexagon edge, can be used to provide additional localized capacity at a lower power cost. The term “micro” will henceforth be used in a broad sense, to denote nodes with smaller power and coverage, thus encompassing everything from a micro to possibly femto, with the following provisos: micro nodes alonecannot provide full coverage of a hexagon (not even jointly), and all nodes are owned by the same entity (i.e., a telecommunication operator), which pays the electrical bill. Either macro or micro nodes are used to serve UE within the network. UE associates with the node from which they perceive the highest SINR, possibly using a Cell Range Expansion (CRE) boost for the micros.Figure 1
System scenario. Macro and micro nodes are represented withM andm, respectively.All the nodes share the same spectrum and hence interfere with each other. We assumeno coordination, either among or within hexagons. A node that has to allocate RBs to its users will pick them at random from a vector of B RBs and schedule each UE on each RB with the same probability. Cell coordination and Coordinated Multipoint (CoMP) techniques are a promising avenue of research to cope with interference and have been shown to increase the power saving opportunities ‎[2]. However, they work at a significantly faster pace than algorithms for node switching and are thus orthogonal to them.We assume that the operator possesses the following information: a per hexagon historicalload curve, detailing (at least) the overallbitrate requested in a hexagon over time in a sample day. There may be several such curves, of course, for a single hexagon, depending on thecontext. For instance, working days and holidays have different load curves (see, e.g., ‎[19]), and other human activities do affect the context: the typical example is an hexagon covering a major sports venue, whose load curve on the game days (e.g., Sundays) depends on whether the local sports team plays home or away. Moreover, the operator possesses a similar curve (or set thereof) detailing the number of users per cell over time, so that the averageper-user bitrate can be inferred (a possible method to obtain such information is usingmonitoring systems that measure the values of specific KPIs (e.g., number of types of UE) with a fixed time resolution ‎[22]. These measurements are repeated in different days with the same operating conditions (e.g., working days) in order to produce a daily profile of the average values for the given KPI.). Note that these curves do not contain information about thepositionof single users within a hexagon. The latter may be known in aprobabilisticsense, for example, uniform within a hexagon. We make no assumption on theshapeof the load curves, and—in particular—we do not rely on their having a single peak during business hours. The time resolution at which these curves are plotted is that of a sufficiently large interval (e.g., 15 minutes), called asnapshothenceforth. We denote with K the number of such snapshots in a day. Furthermore, we assume that we can retrieve thecurrentamount of allocated RBs at each macro/micro node, averaged over a snapshot.We only consider the downlink direction, which is the most critical from the point of view of both carried load and infrastructure power consumption. Figure2 shows the power consumed by an active node (taken from ‎[23]). The power is an affine function of the number of transmitted RBs on each TTI, that is, p=Pbase+ρ·n, where Pbase is abaseline power and n≤B is the number of allocated RBs. The power/RB curve has the same shape for all the nodes, whether macro or micro, whereas the values Poff, Pbase, Pmax, and ρ do depend on the type of node. According to equipment manufacturers and cellular operators, the value of the above parameters has changed over the years thanks to the benefits introduced by new technologies and will keep changing, following the progressive shift towards the fifth generation of mobile networks (5G). However, the affine shape of the power/RB curve has remained unchanged, and such trend is expected to persist ‎[23].Figure 2
Node power model.Amacro node cannot be powered off and hence, thevariablepart of its consumed power is p=ρ·n. For amicronode, the variable part is instead p=Pbase-Poff+ρ·n.Since load curves represent what happens in a snapshot, whereas power is consumed by allocating RBs at each TTI, a method to infer the latter from a snapshot’s load value is required. RBs are computed as follows: first we infer a per-user requested data rate from the load curves, then we show how to compute the number of RBs needed to meet that data rate given the UE’s SINR, and then we show how to compute the SINR of a UE undergoing interference using a statistical interference model.Focus on a single snapshot and a single hexagon, and callL and U the load and number of users for that hexagon at that snapshot. Theaverage demand per UE at a snapshot is D=L/U. Thedata rate per RB achievable by UE u associated with node e is a function of itsaverageSINR, FSINRue, an example of which is represented in Figure 3 (curves like the one in Figure 3 are obtained through link-level simulations, see, e.g., ‎[24]). ηMAX is the maximum data rate that can be achieved for values of SINR equal or above SINRMAX. UE whose SINR is below SINRmin is considered out of range. Thus, the average number of RBs required to satisfy u’s demand can be computed as(1)RBu=DFSINRue.Figure 3
Data rate versus SINR.Note thatRBu may not be integer. This is not a problem, since RBu is anaverage value, and the time span of a snapshot is large enough as to allow a fluid approximation. For instance, an allocation of RBu=2.5 RBs per TTI can be enforced in practice by reserving two RBs on even TTIs and three on odd ones and so on.In order to computeSINRue, we need to quantify both the signal received from node e and the interference from nodes x≠e. Call ne and nx the number of RBs allocated by nodes e and x, respectively. Given that they do not coordinate, the probability that the same j RBs are allocatedsimultaneouslyby both is ‎[25](2)πe,x,j=Bj·B-jne-j·B-nenx-jBne·Bnx=nej·B-nenx-jBnx,with max⁡0,ne+nx-B≤j≤min⁡ne,nx. Therefore, the expected value of the number of overlapping RBs between e and x is(3)Δe,x=∑j=max⁡0,ne+nx-Bmin⁡ne,nxj·πe,x,j.Thus, the percentage of RBs wherex will interfere is Δe,x/ne, whereas the remaining 1-Δe,x/ne will be exempt from interference from x. Call Px,u the power received by u from node x (which depends on the distance and angle between them, the propagation model and the transmitting power of x). Then, theaverage SINR of UE u in the snapshot is(4)SINRue=Pe,uNG+∑x≠ePx,u·Δe,x/ne,where NG is the Gaussian noise.As far as switching is concerned, as already specified, macro nodes cannot be powered off, because this would create holes in the coverage. Micro nodes, instead,can be powered off, for example, during low-load periods. However, switch-on/off operations should befew, for several reasons: first, especially with older equipment, it takes a considerable amount of time before a micro node is fully operational (in the order of tens of seconds or minutes), due to the operating system boot procedure, control and management plane setup operations (e.g., path setup in the Evolved Packet Core network), and so on, and it takes some time to shut it down as well. Second, topology alterations often have unpredictable ripples in a multicell network: a switch-off may cause massive handovers (even whencell wilting is used, ‎[6]), possibly overloading the corresponding macroand affecting nearby macros through a change in the interference pattern. Third, switching operations increase the wear and tear of equipment, hence reducing its MTBF ‎[6]. For this reason, we assume that an upper bound of 2·Smax switching operations must be enforced on all micro nodes. In other words, micro nodes can be powered up inat mostSmaxdisjoint time intervals within a day.The aim of our framework is toplan andenforce switching operations at each snapshot for each micro node, so that(1)
user requests are satisfied, if it is possible to do so. In other words, nooverloadoccurs that prevents UE from obtaining the required rate;(2)
as little energy as possible is consumed by the network, provided that UE requirement is met;(3)
micro nodes are switched on up toSmax times per day.Our frameworkplansswitching operations in advance (i.e., offline), based on historical load curves, so as to keep the macro overload probability below a predetermined threshold, andenforcesswitching operationsonlineeither to avoid overload or when it is energetically efficient to do so, by comparing the current load conditions to the expected (historical) ones.


## 3. Related Work

As already outlined, several schemes have been proposed in the past few years to deal with macro and micro node deactivation. Surveys ‎[5–9] provide a general overview and taxonomy, besides the customary wealth of references.In the context of homogeneous networks, many works advocate cellbreathing(e.g., ‎[11]): the radius of some cells is shrunk during off-peak hours, and nearby cells increase theirs accordingly to guarantee coverage. If the scenario allows it, a node can be completely switched off, which increases the saving. Cellwiltingandblossomingare procedures for graceful shut-down and power-up of nodes, whose aim is to avoid massive handovers during topology changes ‎[12].It must be observed that operators often consider hazardous to vary the cell radius, because it makes it difficult, at best, to ensure the absence of coverage holes ‎[6]. Moreover, most of the current equipment can only be powered on or off, lacking the capability to switch among a discrete number of power levels that is instead required in such schemes. Paper ‎[13] models the energy saving sleep problem assuming a single macro and several small cells, with symmetric traffic, no interference, and “1 : 1 convertible” resource allocation between nodes. A larger-scale scenario (cluster of 7 macros, with no micro) is considered in ‎[10], where cell breathing is advocated. Authors of ‎[14] describe an offline algorithm that does not allow multiple switching of the same node in a day. In ‎[15] a planning tool is described, which also takes cell breathing into account. A similar problem is tackled in ‎[26], where authors focus on positioning of both macro and micro stations to achieve more capacity at a small cost in power consumption. Finally, ‎[16] determines the optimal set of cells to be switched at any timet, given the instantaneous load at that time.Among the above works, those advocating offline schemes often make assumptions on the shape of load curves, meaning that theyonlywork when the load curves match those envisaged. Our work does not make such an assumption, and can work withanyshape of load curves. Moreover, these works seldom consider intercell interference, which—as shown in the previous section—clearly influences the switching decisions. None of the above works considers (or may easily incorporate) constraining the number of switching decisions at a node, something which—as we will demonstrate—changes the picture completely.Finally, we observe that the problem of energy-efficient resource allocation in LTE base stations can be tackled at different timescales. Besides theresource planningtimescale (i.e., hours or days), which is the one of all the above works, we can also work at theframe timescale, that is, few tens of ms, for example, by scheduling MBSFN subframes in a frame ‎[27]. During off-peak periods, even short ones, the eNB pools the few RBs that it has to transmit into as few subframes as possible, leaving the other ones blank, thus trading a modest reduction of responsiveness for a remarkable power saving. These solutions are complementary, rather than alternative, to the ones dealt with in this paper.


## 4. Node-Switching Framework

We first explain some of the practical challenges for a switching algorithm, which also sets the rationale behind our framework. Then we describe the framework at a high level and each of its components in detail.We will use the shorthandm to indicate thatallthe micros in a group of hexagons are switched on and − to indicate that they are all off. We will often distinguish two groups of hexagons: ataggedone, whose performance we monitor, and the rest of the network. Accordingly, m∣- will indicate that micros are switchedon in the tagged hexagon andoff in the other ones.


### 4.1. Justification and Practical Challenges

As already explained, we consider two performance metrics: overload and energy consumption. We first assess the impact of a micro switch-on/switch-off decision on both metrics.The interference of amicro is, by and large, negligibleoutside its own hexagon. This means that, if switchingona micro succeeds in offloading the macro of some RBs, then the effects on overloadat large, that is, in the whole network, can only be beneficial, since the intercell interference which is mainly due to the macro will be reduced through (4). Unless micros are misplaced, it is usually true that switching on a micro offloads the macro of some RBs and that the overload probability in a hexagondecreaseswhen the micros are switched on. This means that switching microsonwhen a hexagon is in overload are the right thing to do. In order to demonstrate these two phenomena let us consider the following example: suppose we are monitoring the channel quality perceived by the UE in hexagon h0 placed at the center of a scenario composed of other nine hexagons, each of them with one macro and two micro nodes. We define four configurations: a first one (-∣-) with all the micros turned off, a second one (m∣m) with all the micros turned on, a third one (-∣m) with all the micros turned on except within h0, and a fourth one (m∣-) with the micro nodes turned ononly in h0. In Figure 4 we show the perceived channel quality in h0: the color of each point represents the best SINR computed among the active nodes of the considered hexagon. As we can see if we compare -∣- and -∣m, the SINR in h0 slightly increases even if we just turn on micros in the other hexagons, thus confirming that interference from the latter has decreased. On the other hand, turning on micros improves the SINR of h0 in both m∣m and m∣- scenarios, a benefit that comes along with the added capacity in terms of RBs that are made available by each active micro. An additional analysis of the large-scale effects of interference will be given in Section 5.Figure 4
Channel quality perceived by UE inh0when all micros are turned off (-∣-), all micros are turned on (m∣m), micros are turned on only in h0 (m∣-), and micros are all turned on except in h0 (-∣m).From an energy point of view, switching on a micro incurs a baseline penalty given to the differencePbasem-Poffm. However, RBs allocated at the micro are normally cheaper than those at the macro, since ρm<ρM, where subscripts and superscripts are self-explanatory. When a micro is switched on, and some UE switches from the macro to the micro, there is a global decrease of intercell interference through (4), which is also beneficial energy-wise. From the above, we obtain that keeping a micro on for very few allocated RBs is inefficient. However, it isimpossibleto estimate the minimum number Thm of RBs that the micro should allocate to pay for its switch-on cost fromonlineinformation only. In fact, that number can only be inferred by solving the following equation:(5)Pbasem-Poffm+ρm·nm=ρM·nM,where the left-hand side is the increase of power when a micro is switched on and allocates nm RBs, and the right-hand side is the reduction of power because the macro allocates nM fewer RBs. However, nm and nM depend on which kinds of UE will switch to the micro and what their SINR would be if they were associated with the micro. This is of course unknown, both because you do not know where this UE is locatedandbecause the micro is switched off, which creates a chicken-and-egg problem.Similarly, it is unrealistic to assume that we can infer when it is energetically efficient to switch a microoff based on online data only. In fact, this would require knowing the SINR of each UE to the macroafterthe micro has been switched off. The latter is not equal to the one measuredbeforethe switch-off, due to the mutated intra- and interhexagon interference.Nevertheless, we can figure out an approximate thresholdThm bysimulatingthe networkoffline using load curves and space distributions. Thus, the insight gathered using offline simulations can surrogate the lack of SINR-omniscience required in an online-only approached.Another challenge stems by the constraint on the number of switch-ons in a day. This implies that making the decision to switchoff a micro based on thecurrenttraffic conditions only (in a purely online approach) is the wrong thing to do. This can be explained through an example.Example 1.
In Figure5 we show an example of average per-snapshot RBs allocation over time in a scenario with one macro (M) and one micro node (m), constrained by a maximum number of switch-on operations Smax=2. Call nM and nm the number of RBs allocated by M and m, and Thsat the macro saturation threshold. We start with m off. The macro saturates at t1, triggering the activation of m. At t2 node M is no longer in saturation, and the number of allocated RBs starts to decrease on both nodes, reaching value Thm on the microside: judging by the allocated RBs only, we switch m off. The same occurs in t4,t6:m is activated whenM saturates (t4), so the second and last micro switch-ons are spent. This means that when the load decreases again later on, we cannot switch m off, even though nm drops below Thm, since doing so would prevent us from reacting to possible future saturation conditions. Micro m must therefore be kept active until t7, where the value of nm rises again above Thm. Thus, we are able to cope with the saturation that occurs between t8 and t9. Note that, in the absence of switch-on constraints, the micro would be switchedoff at t6 andon again at t8.Figure 5
Resource block utilization Example1.Using the solution described above, we can react to overload conditions at the macro node and to inefficient utilization of micro ones, by just looking at theinstantaneous allocation of RBs. Two considerations are in order.If more energy was consumed int6,t8 than is saved in t3,t4, it would be more efficient to switch the micro off in the former and keep it active in the latter.Int7,t8, nm grows above ThmbeforenM reaches Thsat. In this case, we can exploit m efficiently, energy-wise, even though the macro node is not in saturation yet.The above two observations can only be made with hindsight, hence areprecluded to an online algorithm. However, we can again exploithistoricalknowledge, harvested from the daily load curves, to endow an online algorithm with prediction capabilities and increase its efficiency.


### 4.2. Description of the Framework

Our framework obtains historical information from the network, processes itoffline, and makes switching decisionsonline, based on the current load and the historical knowledge. The framework and its information exchange with the network nodes are shown in Figure 6. The offline part of the framework is composed of atrainer and anOffline Analyzer. The trainer takes as input the historical load curves (detailing the requested bitrate and number of users per hexagon) andsimulateswhat happens at each snapshot when micros are switched on and off, respectively. The outputs of this simulation are, for each snapshot, anoverload probability, that is, the probability that a given hexagon is unable to carry the requested load when the micros are switched off, and aresource consumption, that is, the number of RBs allocated within a hexagon when the micros are turned on and off, respectively (note that the trainer could also be realized in different way, which may not involve simulation at all, e.g., as a machine learning system trained with a large set of measurements from the network in various configurations. As long as the trainer is able to compute overload probabilities and resource consumption, the rest of our framework will be able to work with it).Figure 6
High-level view of the switching framework.The Offline Analyzer takes as input the above information, plus the power models of the macros and micros and the constraint on the number of switch-ons, and finds theenergy-optimal activation pattern for each hexagon, that is, the intervals where switching micros off allows the highest power saving, given that the requested load is satisfied.The above information, plus the current load at the hexagons, is used by theOnline Deciderto enforce switching decisions. Hereafter, we present the three blocks separately.


#### 4.2.1. Trainer

The trainer considers one snapshot at a time. In a snapshot, itsimulatesthe network with the given number of users per hexagon, their spatial distribution, and requested bitrate, in ascenario. Ascenariois defined by the number of nodes that is switched on: for instance, amicro-offscenario is one where all micros are switched off, and amicro-onscenario is the one where all micros are switched on.The simulation is done as follows: users are dropped in the hexagon according to the spatial distribution. Figure7 shows a uniform (a) and hotspot-based (b) spatial distribution. In the latter, UE is dropped in the portion of hexagon that intersects a circle of centerc and radiusr. Note that spatial distributions are usedonly by the trainer, while theOnline Decider totally does without them. At a snapshot boundary, UE is created or deleted to match the required number, and the position of the remaining ones is randomly changed within a configured area. This allows us to simulate random mobility within the cell or parts thereof.Figure 7
Uniform (a) and hotspot (b) users distribution in an exemplary hexagon. In the latter case two hotspots, hs1 and hs2, are shown.
(a)
(b)UE is associatedsequentially, that is, one by one, to the active node with the highest SINR, with a CRE boost for the micros.The average SINR of each UE is computed through (4). Note that there is a circular dependence between the SINR and the RBs allocated to a UE. When a UE is allocated some RBs (say, in hexagone), in fact, Δe,x grows, thus increasing the interference suffered by UE attached tox and reducing their SINR. This in turn increases their RB demand and so on. This means that the average SINR must be computed iteratively, until convergence is reached. The algorithm that does this is shown in Algorithm 1.Algorithm 1:Pseudocode for the node association and RB allocation procedure.
(
1
)   iteration = 0
(
2
)   While(interfChanges or iteration >Itmax)
(
3
)
resetAllocation
()
(
4
)
for each hexagon i
(
5
)
for each UE u in i
(
6
)
if
(
itearation < It
)
// Assoc. Ph.
(
7
)
k = chooseEnb(u)
(
8
)
else
// Conv. Ph.
(
9
)
k = getServingEnb
(
u
)
(
10
)
allocateBlocks
(
u,k
)
(
11
)
end for
(
12
)
end for
(
13
)
interfChanges = updateInterference
()
(
14
)
iteration++;
(
15
)  end whileThe procedure is a cycle repeated for up toItmax iterations or until convergence is reached. Each iteration cycles through every UE. We distinguish two phases.(i) Association Phase. For the first It iterations (lines 6-7) UE is allowed toselectthe serving node, according to a best SINR policy (line 7). While doing so, the procedure also allocates RBs according to (1) (line 10).(ii) Convergence Phase. For all the remaining iterations, UE does not change their serving node, and the allocated RBs are updated (line 10).In both cases, at the end of an iteration the interference is updated (line 13) through (3). If any Δe,x changes significantly from one iteration to the next, theinterfChanges flag is set, to signal that the convergence has not been reached yet. Both Prxue and Px,u from (4) are computed using the ITU pathloss and angular attenuation models ‎[28].Note that, at the first iteration of theAssociation Phase, no RBs have yet been allocated at any node, hence the interference is null and the association is distance-based rather than SINR-based. After the first iteration the interference is updated, hence the nearest node may not be the one with the best SINR anymore. This is the reason why theAssociation Phase is repeated It times. However, It has to be limited; otherwise some UE will end up oscillating indefinitely between two or more nodes. Figure 8 shows how the SINR decreases with the iterations, quickly converging to a stable value.Figure 8
Evolution of the average user SINR during iterative interference evaluation of three hexagons in a snapshot.We say that a hexagon, during snapshotk, isoverloadedin a scenario if it cannot serve its requested traffic. This means that all the resources of the macro (and possibly of the micros, if they are switched on) are depleted. Our first concern is to ensure that the network is able to satisfy the requested load. Therefore, we need to take note of overloaded hexagons. Moreover, we need to assess whether a scenario is more energy-efficient than another, hence to compute the number of allocated RBs at all active nodes, which contribute to the variable part of the power.Obviously, both the above depend on the way UE has been dropped. We recall that UE dropping is random, according to a preestablished distribution. Therefore, the trainer repeats the simulation of the same scenario forr times, dropping UE independently at every replica. This way, we obtain anoverload probabilityDh for each hexagon h in the considered scenario, given by(6)Dh=1r·∑i=1r1hoverloadedinreplicai.Moreover, we obtain a vector ofaverage allocated RBs, whose elements are the RBs allocated at each node in the hexagon, averaged over thenreplicas: (7)Rh,e=1r·∑i=1rRh,ei,where e is the identifier of the node within hexagon h.The above two quantities can be used to compare scenarios, specifically to identify the minimum-power one having the overload probabilities below a given threshold. Unfortunately, an exhaustive comparison of scenarios is impossible. This would in fact require simulating as many scenarios as the powerset of the set of micros. In a network with few tens hexagons, micros are easily in the hundreds, which makes the task impossible. We therefore settle for a simple trade-off, which proves to be effective nonetheless. We simulate only the two limit scenarios with all micros switchedonand switchedoff. We denote these two scenarios using subscripts ↑ and ↓. The rationale beyond this choice is that the two configurations are expected to achieve, respectively,(i)
the minimum overload probability (which should be null, unless the network is underprovisioned), but a high power consumption;(ii)
a larger overload probability, with a considerably smaller power consumption.All the quantities defined above are computed for each snapshot, thus values at snapshotk will be represented as Dhk and Rh,ek.


#### 4.2.2. Offline Analyzer

ValuesDh, RB↑e,h, and RB↓e,h, and the node power models are then passed to a module calledOffline Analyzer(OA), whose purpose is to process this information and to generate guidelines for online decisions. The OA makes independent decisions for each hexagon. It compares Dhk to a predefined overload probability threshold (which may also depend on the snapshot and hexagon) and computes a binary value Ohk that marks whether the overload probability is above or below the threshold. First it computes the expected power consumption of each of hexagons E↑hk and E↓hk in the two configurations as (8)E↓hk=PbaseM,h+RB↓M,hk·PinM,h+∑m=0LhPoffm,h,E↑hk=PbaseM,h+RB↑M,hk·ρM,h+∑m=0LhPbasem,h+RB↑m,hk·ρm,h,where Lh is the number of micros in hexagon h.Note that each node may have its own power model parameters, thus accounting, for example, for different versions of the apparatus and different cost of energy. The same applies to the switching-off thresholds. However, for the sake of readability, we will assume that all the power models are the same and drop the superscript whenever this does not generate ambiguity. For instance, we will useThm to denote the switch-off threshold.We can then obtain an activation pattern for micro nodes in a hexagon during a day, as a vector ofK binary values ahk, where ahk=0 indicates that all micros in the hexagon are switched off and ahk=1 if they are all on. We compute the energy-optimal pattern separately for each hexagon as the optimum of the following problem, where hexagon superscript h is omitted for the sake of readability:(9)min∑kE↓k·1-ak+E↑k·aks.t.iak≥Okiisk≥ak-ak-1iiis0≥a0-aK-1iv∑k=0⋯Ksk≤Smaxvsk,ak∈0,1.sk is a binary variable that counts switch-on operations. The objective function to be minimized is the expected overall power consumed by a cell throughout the whole day. Constraint (i) states that micros should be on when saturation may occur, (ii) forces sk to one when the micro is switched on at k, (iii) takes care of the wrap-around at the end of the day, and finally (iv) constrains the number of switch-ons to Smax. The above is an Integer-Linear Problem (ILP) with Θ(K) constraints and variables and can be easily solved using commercial solvers such as CPLEX ‎[29].In Figure9, we show an example of computation of the energy-optimal activation pattern, based on the same scenario of Figure 5. Each vector represents the values of Ohk, E↓hk, and E↑hk, respectively, throughout a day of K=12 snapshots within hexagon h. Marked elements in the second and third vectors identify the lowest-power configuration (all off orall on) at that snapshot. Note that the same time instants ti as the ones in Figure 5 are reported. The dashed line indicates the pattern that will be chosen by our optimization problem if Smax=2. As we can see we have three snapshots where Ohk=1, where keeping micros on is thus mandatory. The optimal solution is to keep the micros on in t3,t4 and switch them off in t5,t7 as the power saved in snapshot three is E↑h3-E↓h3=10, whereas in snapshots 5 to 8 is ∑k=58E↑hk-E↓hk=35. Moreover, note that, as observed in Example 1, the optimization problem brings forward the second switch-on event to t7.Figure 9
Example of energy-optimal pattern computation.


#### 4.2.3. Online Decider

The output of the OA is fed to theOnline Decider that monitors the system utilization and takes decisions on a per-snapshot timescale. More specifically this module decides which micro to switch on/off, based onboth current and historical information. It includes anactivation anddeactivation procedure, performed sequentially. Figure 10 reports the flow diagrams for such procedures. Each box represents a processing step, whereas each diamond represents a decision. Deactivation takes as an input a list of micro nodes, ordered by increasing load. Each of those micro nodesm is switched off ifall the following conditions hold simultaneously at snapshotk:(i)
no node in the hexagon is overloaded,(ii)
ahk=0 for the given hexagon,(iii)
m is underloaded, that is, RBm≤Thm.Figure 10
Micro deactivation (a) and activation (b) procedures.
(a)
(b)Conversely, we activate all the micros of a hexagon ifeither of or both the following conditions hold:(i)
at least one node in the hexagon is overloaded;(ii)
ahk=1.Example 2.
Consider as an example the RB utilization shown in Figure11 where a hexagon with one macro and one micro is considered and the energy-optimal pattern of Figure 9 is applied. At t1 theactivation procedure detects saturation and switches the micro on. At t3, it is kept on since ahk=1. The first switch-off opportunity occurs at t5: in this case the micro is kept on until RBm<Thm, that is, at t6. The last switch on is at t7, as in the energy-optimal pattern.Figure 11
Resource block utilization Example2.


### 4.3. Implementation Issues

The software framework described in this paper is meant to be run at a centralized element, which performs resource planning. The location and nature of this resource planner depend on the operator choices for RAN deployment. In a Distributed RAN, it should be a monitoring server in the operator’s network. In a Centralized RAN (C-RAN), it would naturally be a process running in the same data center that hosts the virtualized Baseband Units (BBUs) of the nodes in an area. A software system that supports a decision framework such as the one described in this paper is being designed and deployed in a testbed as part of the Flex5Gware EU 5G-PPP project ‎[30].Regarding the processing cost, our measurements show that a single-day run (i.e., 96 snapshots) of the trainer occupies approximately 1.5 minutes of a single core on a desktop PC. Solving the optimization problem at optimality takes less than one second and hence is negligible. The number of runs should be calibrated based on the required overload probability threshold: more runs allow one to estimate small probabilities with greater accuracy. Now, the planning normally occurs on a daily basis, and a single core can run the trainer up to ~950 times per day. This means that our framework can be made to work with arbitrary accuracy on off-the-shelf hardware in a day. More to the point, this very fact can also be exploited to compensate for unforeseen context mismatches. Suppose that anunforeseen event occurs, such as a large traffic jam, whichpredictablychanges the context for the rest of the day (e.g., by postponing by 1-2 snapshots the onset of the daily peak in all the cells of a business district). One can then run the offline trainer again, starting from the current time up to the end of the day, discounting the switching operations that have already occurred from Smax, and come up with an optimized planning for the rest of the day in a reasonable time, still executing tens of training runs on a multicore machine within a snapshot’s worth of computation time.The communication overhead is also small to negligible. A vector of 96 binaries must be communicated daily (or on demand, in the above cases) to the macros, and the macros themselves can issue the switch-on/off commands to their micros according to the daily plan. Communications among macro and micro nodes belonging to the same hexagon can be realized exploiting the X2 communication and in particular the X2 Application Protocol (X2AP) ‎[31]. The latter defines a set of standard messages that eNBs can exchange. For example, an overload condition can be signaled using the Load Indication Information Element, and an eNB can be switched on using the cell activation procedure.


## 4.1. Justification and Practical Challenges

As already explained, we consider two performance metrics: overload and energy consumption. We first assess the impact of a micro switch-on/switch-off decision on both metrics.The interference of amicro is, by and large, negligibleoutside its own hexagon. This means that, if switchingona micro succeeds in offloading the macro of some RBs, then the effects on overloadat large, that is, in the whole network, can only be beneficial, since the intercell interference which is mainly due to the macro will be reduced through (4). Unless micros are misplaced, it is usually true that switching on a micro offloads the macro of some RBs and that the overload probability in a hexagondecreaseswhen the micros are switched on. This means that switching microsonwhen a hexagon is in overload are the right thing to do. In order to demonstrate these two phenomena let us consider the following example: suppose we are monitoring the channel quality perceived by the UE in hexagon h0 placed at the center of a scenario composed of other nine hexagons, each of them with one macro and two micro nodes. We define four configurations: a first one (-∣-) with all the micros turned off, a second one (m∣m) with all the micros turned on, a third one (-∣m) with all the micros turned on except within h0, and a fourth one (m∣-) with the micro nodes turned ononly in h0. In Figure 4 we show the perceived channel quality in h0: the color of each point represents the best SINR computed among the active nodes of the considered hexagon. As we can see if we compare -∣- and -∣m, the SINR in h0 slightly increases even if we just turn on micros in the other hexagons, thus confirming that interference from the latter has decreased. On the other hand, turning on micros improves the SINR of h0 in both m∣m and m∣- scenarios, a benefit that comes along with the added capacity in terms of RBs that are made available by each active micro. An additional analysis of the large-scale effects of interference will be given in Section 5.Figure 4
Channel quality perceived by UE inh0when all micros are turned off (-∣-), all micros are turned on (m∣m), micros are turned on only in h0 (m∣-), and micros are all turned on except in h0 (-∣m).From an energy point of view, switching on a micro incurs a baseline penalty given to the differencePbasem-Poffm. However, RBs allocated at the micro are normally cheaper than those at the macro, since ρm<ρM, where subscripts and superscripts are self-explanatory. When a micro is switched on, and some UE switches from the macro to the micro, there is a global decrease of intercell interference through (4), which is also beneficial energy-wise. From the above, we obtain that keeping a micro on for very few allocated RBs is inefficient. However, it isimpossibleto estimate the minimum number Thm of RBs that the micro should allocate to pay for its switch-on cost fromonlineinformation only. In fact, that number can only be inferred by solving the following equation:(5)Pbasem-Poffm+ρm·nm=ρM·nM,where the left-hand side is the increase of power when a micro is switched on and allocates nm RBs, and the right-hand side is the reduction of power because the macro allocates nM fewer RBs. However, nm and nM depend on which kinds of UE will switch to the micro and what their SINR would be if they were associated with the micro. This is of course unknown, both because you do not know where this UE is locatedandbecause the micro is switched off, which creates a chicken-and-egg problem.Similarly, it is unrealistic to assume that we can infer when it is energetically efficient to switch a microoff based on online data only. In fact, this would require knowing the SINR of each UE to the macroafterthe micro has been switched off. The latter is not equal to the one measuredbeforethe switch-off, due to the mutated intra- and interhexagon interference.Nevertheless, we can figure out an approximate thresholdThm bysimulatingthe networkoffline using load curves and space distributions. Thus, the insight gathered using offline simulations can surrogate the lack of SINR-omniscience required in an online-only approached.Another challenge stems by the constraint on the number of switch-ons in a day. This implies that making the decision to switchoff a micro based on thecurrenttraffic conditions only (in a purely online approach) is the wrong thing to do. This can be explained through an example.Example 1.
In Figure5 we show an example of average per-snapshot RBs allocation over time in a scenario with one macro (M) and one micro node (m), constrained by a maximum number of switch-on operations Smax=2. Call nM and nm the number of RBs allocated by M and m, and Thsat the macro saturation threshold. We start with m off. The macro saturates at t1, triggering the activation of m. At t2 node M is no longer in saturation, and the number of allocated RBs starts to decrease on both nodes, reaching value Thm on the microside: judging by the allocated RBs only, we switch m off. The same occurs in t4,t6:m is activated whenM saturates (t4), so the second and last micro switch-ons are spent. This means that when the load decreases again later on, we cannot switch m off, even though nm drops below Thm, since doing so would prevent us from reacting to possible future saturation conditions. Micro m must therefore be kept active until t7, where the value of nm rises again above Thm. Thus, we are able to cope with the saturation that occurs between t8 and t9. Note that, in the absence of switch-on constraints, the micro would be switchedoff at t6 andon again at t8.Figure 5
Resource block utilization Example1.Using the solution described above, we can react to overload conditions at the macro node and to inefficient utilization of micro ones, by just looking at theinstantaneous allocation of RBs. Two considerations are in order.If more energy was consumed int6,t8 than is saved in t3,t4, it would be more efficient to switch the micro off in the former and keep it active in the latter.Int7,t8, nm grows above ThmbeforenM reaches Thsat. In this case, we can exploit m efficiently, energy-wise, even though the macro node is not in saturation yet.The above two observations can only be made with hindsight, hence areprecluded to an online algorithm. However, we can again exploithistoricalknowledge, harvested from the daily load curves, to endow an online algorithm with prediction capabilities and increase its efficiency.


## 4.2. Description of the Framework

Our framework obtains historical information from the network, processes itoffline, and makes switching decisionsonline, based on the current load and the historical knowledge. The framework and its information exchange with the network nodes are shown in Figure 6. The offline part of the framework is composed of atrainer and anOffline Analyzer. The trainer takes as input the historical load curves (detailing the requested bitrate and number of users per hexagon) andsimulateswhat happens at each snapshot when micros are switched on and off, respectively. The outputs of this simulation are, for each snapshot, anoverload probability, that is, the probability that a given hexagon is unable to carry the requested load when the micros are switched off, and aresource consumption, that is, the number of RBs allocated within a hexagon when the micros are turned on and off, respectively (note that the trainer could also be realized in different way, which may not involve simulation at all, e.g., as a machine learning system trained with a large set of measurements from the network in various configurations. As long as the trainer is able to compute overload probabilities and resource consumption, the rest of our framework will be able to work with it).Figure 6
High-level view of the switching framework.The Offline Analyzer takes as input the above information, plus the power models of the macros and micros and the constraint on the number of switch-ons, and finds theenergy-optimal activation pattern for each hexagon, that is, the intervals where switching micros off allows the highest power saving, given that the requested load is satisfied.The above information, plus the current load at the hexagons, is used by theOnline Deciderto enforce switching decisions. Hereafter, we present the three blocks separately.


### 4.2.1. Trainer

The trainer considers one snapshot at a time. In a snapshot, itsimulatesthe network with the given number of users per hexagon, their spatial distribution, and requested bitrate, in ascenario. Ascenariois defined by the number of nodes that is switched on: for instance, amicro-offscenario is one where all micros are switched off, and amicro-onscenario is the one where all micros are switched on.The simulation is done as follows: users are dropped in the hexagon according to the spatial distribution. Figure7 shows a uniform (a) and hotspot-based (b) spatial distribution. In the latter, UE is dropped in the portion of hexagon that intersects a circle of centerc and radiusr. Note that spatial distributions are usedonly by the trainer, while theOnline Decider totally does without them. At a snapshot boundary, UE is created or deleted to match the required number, and the position of the remaining ones is randomly changed within a configured area. This allows us to simulate random mobility within the cell or parts thereof.Figure 7
Uniform (a) and hotspot (b) users distribution in an exemplary hexagon. In the latter case two hotspots, hs1 and hs2, are shown.
(a)
(b)UE is associatedsequentially, that is, one by one, to the active node with the highest SINR, with a CRE boost for the micros.The average SINR of each UE is computed through (4). Note that there is a circular dependence between the SINR and the RBs allocated to a UE. When a UE is allocated some RBs (say, in hexagone), in fact, Δe,x grows, thus increasing the interference suffered by UE attached tox and reducing their SINR. This in turn increases their RB demand and so on. This means that the average SINR must be computed iteratively, until convergence is reached. The algorithm that does this is shown in Algorithm 1.Algorithm 1:Pseudocode for the node association and RB allocation procedure.
(
1
)   iteration = 0
(
2
)   While(interfChanges or iteration >Itmax)
(
3
)
resetAllocation
()
(
4
)
for each hexagon i
(
5
)
for each UE u in i
(
6
)
if
(
itearation < It
)
// Assoc. Ph.
(
7
)
k = chooseEnb(u)
(
8
)
else
// Conv. Ph.
(
9
)
k = getServingEnb
(
u
)
(
10
)
allocateBlocks
(
u,k
)
(
11
)
end for
(
12
)
end for
(
13
)
interfChanges = updateInterference
()
(
14
)
iteration++;
(
15
)  end whileThe procedure is a cycle repeated for up toItmax iterations or until convergence is reached. Each iteration cycles through every UE. We distinguish two phases.(i) Association Phase. For the first It iterations (lines 6-7) UE is allowed toselectthe serving node, according to a best SINR policy (line 7). While doing so, the procedure also allocates RBs according to (1) (line 10).(ii) Convergence Phase. For all the remaining iterations, UE does not change their serving node, and the allocated RBs are updated (line 10).In both cases, at the end of an iteration the interference is updated (line 13) through (3). If any Δe,x changes significantly from one iteration to the next, theinterfChanges flag is set, to signal that the convergence has not been reached yet. Both Prxue and Px,u from (4) are computed using the ITU pathloss and angular attenuation models ‎[28].Note that, at the first iteration of theAssociation Phase, no RBs have yet been allocated at any node, hence the interference is null and the association is distance-based rather than SINR-based. After the first iteration the interference is updated, hence the nearest node may not be the one with the best SINR anymore. This is the reason why theAssociation Phase is repeated It times. However, It has to be limited; otherwise some UE will end up oscillating indefinitely between two or more nodes. Figure 8 shows how the SINR decreases with the iterations, quickly converging to a stable value.Figure 8
Evolution of the average user SINR during iterative interference evaluation of three hexagons in a snapshot.We say that a hexagon, during snapshotk, isoverloadedin a scenario if it cannot serve its requested traffic. This means that all the resources of the macro (and possibly of the micros, if they are switched on) are depleted. Our first concern is to ensure that the network is able to satisfy the requested load. Therefore, we need to take note of overloaded hexagons. Moreover, we need to assess whether a scenario is more energy-efficient than another, hence to compute the number of allocated RBs at all active nodes, which contribute to the variable part of the power.Obviously, both the above depend on the way UE has been dropped. We recall that UE dropping is random, according to a preestablished distribution. Therefore, the trainer repeats the simulation of the same scenario forr times, dropping UE independently at every replica. This way, we obtain anoverload probabilityDh for each hexagon h in the considered scenario, given by(6)Dh=1r·∑i=1r1hoverloadedinreplicai.Moreover, we obtain a vector ofaverage allocated RBs, whose elements are the RBs allocated at each node in the hexagon, averaged over thenreplicas: (7)Rh,e=1r·∑i=1rRh,ei,where e is the identifier of the node within hexagon h.The above two quantities can be used to compare scenarios, specifically to identify the minimum-power one having the overload probabilities below a given threshold. Unfortunately, an exhaustive comparison of scenarios is impossible. This would in fact require simulating as many scenarios as the powerset of the set of micros. In a network with few tens hexagons, micros are easily in the hundreds, which makes the task impossible. We therefore settle for a simple trade-off, which proves to be effective nonetheless. We simulate only the two limit scenarios with all micros switchedonand switchedoff. We denote these two scenarios using subscripts ↑ and ↓. The rationale beyond this choice is that the two configurations are expected to achieve, respectively,(i)
the minimum overload probability (which should be null, unless the network is underprovisioned), but a high power consumption;(ii)
a larger overload probability, with a considerably smaller power consumption.All the quantities defined above are computed for each snapshot, thus values at snapshotk will be represented as Dhk and Rh,ek.


### 4.2.2. Offline Analyzer

ValuesDh, RB↑e,h, and RB↓e,h, and the node power models are then passed to a module calledOffline Analyzer(OA), whose purpose is to process this information and to generate guidelines for online decisions. The OA makes independent decisions for each hexagon. It compares Dhk to a predefined overload probability threshold (which may also depend on the snapshot and hexagon) and computes a binary value Ohk that marks whether the overload probability is above or below the threshold. First it computes the expected power consumption of each of hexagons E↑hk and E↓hk in the two configurations as (8)E↓hk=PbaseM,h+RB↓M,hk·PinM,h+∑m=0LhPoffm,h,E↑hk=PbaseM,h+RB↑M,hk·ρM,h+∑m=0LhPbasem,h+RB↑m,hk·ρm,h,where Lh is the number of micros in hexagon h.Note that each node may have its own power model parameters, thus accounting, for example, for different versions of the apparatus and different cost of energy. The same applies to the switching-off thresholds. However, for the sake of readability, we will assume that all the power models are the same and drop the superscript whenever this does not generate ambiguity. For instance, we will useThm to denote the switch-off threshold.We can then obtain an activation pattern for micro nodes in a hexagon during a day, as a vector ofK binary values ahk, where ahk=0 indicates that all micros in the hexagon are switched off and ahk=1 if they are all on. We compute the energy-optimal pattern separately for each hexagon as the optimum of the following problem, where hexagon superscript h is omitted for the sake of readability:(9)min∑kE↓k·1-ak+E↑k·aks.t.iak≥Okiisk≥ak-ak-1iiis0≥a0-aK-1iv∑k=0⋯Ksk≤Smaxvsk,ak∈0,1.sk is a binary variable that counts switch-on operations. The objective function to be minimized is the expected overall power consumed by a cell throughout the whole day. Constraint (i) states that micros should be on when saturation may occur, (ii) forces sk to one when the micro is switched on at k, (iii) takes care of the wrap-around at the end of the day, and finally (iv) constrains the number of switch-ons to Smax. The above is an Integer-Linear Problem (ILP) with Θ(K) constraints and variables and can be easily solved using commercial solvers such as CPLEX ‎[29].In Figure9, we show an example of computation of the energy-optimal activation pattern, based on the same scenario of Figure 5. Each vector represents the values of Ohk, E↓hk, and E↑hk, respectively, throughout a day of K=12 snapshots within hexagon h. Marked elements in the second and third vectors identify the lowest-power configuration (all off orall on) at that snapshot. Note that the same time instants ti as the ones in Figure 5 are reported. The dashed line indicates the pattern that will be chosen by our optimization problem if Smax=2. As we can see we have three snapshots where Ohk=1, where keeping micros on is thus mandatory. The optimal solution is to keep the micros on in t3,t4 and switch them off in t5,t7 as the power saved in snapshot three is E↑h3-E↓h3=10, whereas in snapshots 5 to 8 is ∑k=58E↑hk-E↓hk=35. Moreover, note that, as observed in Example 1, the optimization problem brings forward the second switch-on event to t7.Figure 9
Example of energy-optimal pattern computation.


### 4.2.3. Online Decider

The output of the OA is fed to theOnline Decider that monitors the system utilization and takes decisions on a per-snapshot timescale. More specifically this module decides which micro to switch on/off, based onboth current and historical information. It includes anactivation anddeactivation procedure, performed sequentially. Figure 10 reports the flow diagrams for such procedures. Each box represents a processing step, whereas each diamond represents a decision. Deactivation takes as an input a list of micro nodes, ordered by increasing load. Each of those micro nodesm is switched off ifall the following conditions hold simultaneously at snapshotk:(i)
no node in the hexagon is overloaded,(ii)
ahk=0 for the given hexagon,(iii)
m is underloaded, that is, RBm≤Thm.Figure 10
Micro deactivation (a) and activation (b) procedures.
(a)
(b)Conversely, we activate all the micros of a hexagon ifeither of or both the following conditions hold:(i)
at least one node in the hexagon is overloaded;(ii)
ahk=1.Example 2.
Consider as an example the RB utilization shown in Figure11 where a hexagon with one macro and one micro is considered and the energy-optimal pattern of Figure 9 is applied. At t1 theactivation procedure detects saturation and switches the micro on. At t3, it is kept on since ahk=1. The first switch-off opportunity occurs at t5: in this case the micro is kept on until RBm<Thm, that is, at t6. The last switch on is at t7, as in the energy-optimal pattern.Figure 11
Resource block utilization Example2.


## 4.2.1. Trainer

The trainer considers one snapshot at a time. In a snapshot, itsimulatesthe network with the given number of users per hexagon, their spatial distribution, and requested bitrate, in ascenario. Ascenariois defined by the number of nodes that is switched on: for instance, amicro-offscenario is one where all micros are switched off, and amicro-onscenario is the one where all micros are switched on.The simulation is done as follows: users are dropped in the hexagon according to the spatial distribution. Figure7 shows a uniform (a) and hotspot-based (b) spatial distribution. In the latter, UE is dropped in the portion of hexagon that intersects a circle of centerc and radiusr. Note that spatial distributions are usedonly by the trainer, while theOnline Decider totally does without them. At a snapshot boundary, UE is created or deleted to match the required number, and the position of the remaining ones is randomly changed within a configured area. This allows us to simulate random mobility within the cell or parts thereof.Figure 7
Uniform (a) and hotspot (b) users distribution in an exemplary hexagon. In the latter case two hotspots, hs1 and hs2, are shown.
(a)
(b)UE is associatedsequentially, that is, one by one, to the active node with the highest SINR, with a CRE boost for the micros.The average SINR of each UE is computed through (4). Note that there is a circular dependence between the SINR and the RBs allocated to a UE. When a UE is allocated some RBs (say, in hexagone), in fact, Δe,x grows, thus increasing the interference suffered by UE attached tox and reducing their SINR. This in turn increases their RB demand and so on. This means that the average SINR must be computed iteratively, until convergence is reached. The algorithm that does this is shown in Algorithm 1.Algorithm 1:Pseudocode for the node association and RB allocation procedure.
(
1
)   iteration = 0
(
2
)   While(interfChanges or iteration >Itmax)
(
3
)
resetAllocation
()
(
4
)
for each hexagon i
(
5
)
for each UE u in i
(
6
)
if
(
itearation < It
)
// Assoc. Ph.
(
7
)
k = chooseEnb(u)
(
8
)
else
// Conv. Ph.
(
9
)
k = getServingEnb
(
u
)
(
10
)
allocateBlocks
(
u,k
)
(
11
)
end for
(
12
)
end for
(
13
)
interfChanges = updateInterference
()
(
14
)
iteration++;
(
15
)  end whileThe procedure is a cycle repeated for up toItmax iterations or until convergence is reached. Each iteration cycles through every UE. We distinguish two phases.(i) Association Phase. For the first It iterations (lines 6-7) UE is allowed toselectthe serving node, according to a best SINR policy (line 7). While doing so, the procedure also allocates RBs according to (1) (line 10).(ii) Convergence Phase. For all the remaining iterations, UE does not change their serving node, and the allocated RBs are updated (line 10).In both cases, at the end of an iteration the interference is updated (line 13) through (3). If any Δe,x changes significantly from one iteration to the next, theinterfChanges flag is set, to signal that the convergence has not been reached yet. Both Prxue and Px,u from (4) are computed using the ITU pathloss and angular attenuation models ‎[28].Note that, at the first iteration of theAssociation Phase, no RBs have yet been allocated at any node, hence the interference is null and the association is distance-based rather than SINR-based. After the first iteration the interference is updated, hence the nearest node may not be the one with the best SINR anymore. This is the reason why theAssociation Phase is repeated It times. However, It has to be limited; otherwise some UE will end up oscillating indefinitely between two or more nodes. Figure 8 shows how the SINR decreases with the iterations, quickly converging to a stable value.Figure 8
Evolution of the average user SINR during iterative interference evaluation of three hexagons in a snapshot.We say that a hexagon, during snapshotk, isoverloadedin a scenario if it cannot serve its requested traffic. This means that all the resources of the macro (and possibly of the micros, if they are switched on) are depleted. Our first concern is to ensure that the network is able to satisfy the requested load. Therefore, we need to take note of overloaded hexagons. Moreover, we need to assess whether a scenario is more energy-efficient than another, hence to compute the number of allocated RBs at all active nodes, which contribute to the variable part of the power.Obviously, both the above depend on the way UE has been dropped. We recall that UE dropping is random, according to a preestablished distribution. Therefore, the trainer repeats the simulation of the same scenario forr times, dropping UE independently at every replica. This way, we obtain anoverload probabilityDh for each hexagon h in the considered scenario, given by(6)Dh=1r·∑i=1r1hoverloadedinreplicai.Moreover, we obtain a vector ofaverage allocated RBs, whose elements are the RBs allocated at each node in the hexagon, averaged over thenreplicas: (7)Rh,e=1r·∑i=1rRh,ei,where e is the identifier of the node within hexagon h.The above two quantities can be used to compare scenarios, specifically to identify the minimum-power one having the overload probabilities below a given threshold. Unfortunately, an exhaustive comparison of scenarios is impossible. This would in fact require simulating as many scenarios as the powerset of the set of micros. In a network with few tens hexagons, micros are easily in the hundreds, which makes the task impossible. We therefore settle for a simple trade-off, which proves to be effective nonetheless. We simulate only the two limit scenarios with all micros switchedonand switchedoff. We denote these two scenarios using subscripts ↑ and ↓. The rationale beyond this choice is that the two configurations are expected to achieve, respectively,(i)
the minimum overload probability (which should be null, unless the network is underprovisioned), but a high power consumption;(ii)
a larger overload probability, with a considerably smaller power consumption.All the quantities defined above are computed for each snapshot, thus values at snapshotk will be represented as Dhk and Rh,ek.


## 4.2.2. Offline Analyzer

ValuesDh, RB↑e,h, and RB↓e,h, and the node power models are then passed to a module calledOffline Analyzer(OA), whose purpose is to process this information and to generate guidelines for online decisions. The OA makes independent decisions for each hexagon. It compares Dhk to a predefined overload probability threshold (which may also depend on the snapshot and hexagon) and computes a binary value Ohk that marks whether the overload probability is above or below the threshold. First it computes the expected power consumption of each of hexagons E↑hk and E↓hk in the two configurations as (8)E↓hk=PbaseM,h+RB↓M,hk·PinM,h+∑m=0LhPoffm,h,E↑hk=PbaseM,h+RB↑M,hk·ρM,h+∑m=0LhPbasem,h+RB↑m,hk·ρm,h,where Lh is the number of micros in hexagon h.Note that each node may have its own power model parameters, thus accounting, for example, for different versions of the apparatus and different cost of energy. The same applies to the switching-off thresholds. However, for the sake of readability, we will assume that all the power models are the same and drop the superscript whenever this does not generate ambiguity. For instance, we will useThm to denote the switch-off threshold.We can then obtain an activation pattern for micro nodes in a hexagon during a day, as a vector ofK binary values ahk, where ahk=0 indicates that all micros in the hexagon are switched off and ahk=1 if they are all on. We compute the energy-optimal pattern separately for each hexagon as the optimum of the following problem, where hexagon superscript h is omitted for the sake of readability:(9)min∑kE↓k·1-ak+E↑k·aks.t.iak≥Okiisk≥ak-ak-1iiis0≥a0-aK-1iv∑k=0⋯Ksk≤Smaxvsk,ak∈0,1.sk is a binary variable that counts switch-on operations. The objective function to be minimized is the expected overall power consumed by a cell throughout the whole day. Constraint (i) states that micros should be on when saturation may occur, (ii) forces sk to one when the micro is switched on at k, (iii) takes care of the wrap-around at the end of the day, and finally (iv) constrains the number of switch-ons to Smax. The above is an Integer-Linear Problem (ILP) with Θ(K) constraints and variables and can be easily solved using commercial solvers such as CPLEX ‎[29].In Figure9, we show an example of computation of the energy-optimal activation pattern, based on the same scenario of Figure 5. Each vector represents the values of Ohk, E↓hk, and E↑hk, respectively, throughout a day of K=12 snapshots within hexagon h. Marked elements in the second and third vectors identify the lowest-power configuration (all off orall on) at that snapshot. Note that the same time instants ti as the ones in Figure 5 are reported. The dashed line indicates the pattern that will be chosen by our optimization problem if Smax=2. As we can see we have three snapshots where Ohk=1, where keeping micros on is thus mandatory. The optimal solution is to keep the micros on in t3,t4 and switch them off in t5,t7 as the power saved in snapshot three is E↑h3-E↓h3=10, whereas in snapshots 5 to 8 is ∑k=58E↑hk-E↓hk=35. Moreover, note that, as observed in Example 1, the optimization problem brings forward the second switch-on event to t7.Figure 9
Example of energy-optimal pattern computation.


## 4.2.3. Online Decider

The output of the OA is fed to theOnline Decider that monitors the system utilization and takes decisions on a per-snapshot timescale. More specifically this module decides which micro to switch on/off, based onboth current and historical information. It includes anactivation anddeactivation procedure, performed sequentially. Figure 10 reports the flow diagrams for such procedures. Each box represents a processing step, whereas each diamond represents a decision. Deactivation takes as an input a list of micro nodes, ordered by increasing load. Each of those micro nodesm is switched off ifall the following conditions hold simultaneously at snapshotk:(i)
no node in the hexagon is overloaded,(ii)
ahk=0 for the given hexagon,(iii)
m is underloaded, that is, RBm≤Thm.Figure 10
Micro deactivation (a) and activation (b) procedures.
(a)
(b)Conversely, we activate all the micros of a hexagon ifeither of or both the following conditions hold:(i)
at least one node in the hexagon is overloaded;(ii)
ahk=1.Example 2.
Consider as an example the RB utilization shown in Figure11 where a hexagon with one macro and one micro is considered and the energy-optimal pattern of Figure 9 is applied. At t1 theactivation procedure detects saturation and switches the micro on. At t3, it is kept on since ahk=1. The first switch-off opportunity occurs at t5: in this case the micro is kept on until RBm<Thm, that is, at t6. The last switch on is at t7, as in the energy-optimal pattern.Figure 11
Resource block utilization Example2.


## 4.3. Implementation Issues

The software framework described in this paper is meant to be run at a centralized element, which performs resource planning. The location and nature of this resource planner depend on the operator choices for RAN deployment. In a Distributed RAN, it should be a monitoring server in the operator’s network. In a Centralized RAN (C-RAN), it would naturally be a process running in the same data center that hosts the virtualized Baseband Units (BBUs) of the nodes in an area. A software system that supports a decision framework such as the one described in this paper is being designed and deployed in a testbed as part of the Flex5Gware EU 5G-PPP project ‎[30].Regarding the processing cost, our measurements show that a single-day run (i.e., 96 snapshots) of the trainer occupies approximately 1.5 minutes of a single core on a desktop PC. Solving the optimization problem at optimality takes less than one second and hence is negligible. The number of runs should be calibrated based on the required overload probability threshold: more runs allow one to estimate small probabilities with greater accuracy. Now, the planning normally occurs on a daily basis, and a single core can run the trainer up to ~950 times per day. This means that our framework can be made to work with arbitrary accuracy on off-the-shelf hardware in a day. More to the point, this very fact can also be exploited to compensate for unforeseen context mismatches. Suppose that anunforeseen event occurs, such as a large traffic jam, whichpredictablychanges the context for the rest of the day (e.g., by postponing by 1-2 snapshots the onset of the daily peak in all the cells of a business district). One can then run the offline trainer again, starting from the current time up to the end of the day, discounting the switching operations that have already occurred from Smax, and come up with an optimized planning for the rest of the day in a reasonable time, still executing tens of training runs on a multicore machine within a snapshot’s worth of computation time.The communication overhead is also small to negligible. A vector of 96 binaries must be communicated daily (or on demand, in the above cases) to the macros, and the macros themselves can issue the switch-on/off commands to their micros according to the daily plan. Communications among macro and micro nodes belonging to the same hexagon can be realized exploiting the X2 communication and in particular the X2 Application Protocol (X2AP) ‎[31]. The latter defines a set of standard messages that eNBs can exchange. For example, an overload condition can be signaled using the Load Indication Information Element, and an eNB can be switched on using the cell activation procedure.


## 5. Performance Evaluation

In this section we analyze the performance of the system described in the previous section. We feed our simulator with both synthetic and realistic load curves, specifying for each snapshot and for each hexagon, the number of active types of UE together with their traffic request in Mbps. Realistic curves are taken from data collected by an operator management system from real deployments. Data are from three hexagons of two different sites, namely,Site-A andSite-B, and include the average number of active UE and the bandwidth due to data communications during a snapshot. We also consider each UE as performing a VoIP call 1 minute long at 20 kbps. To obtain the total data volume we sum the bandwidth due to data and VoIP communication, and we scale the result by a factor of 1.25 and 1.5 to obtain low, medium, and high system loads. As an example, the resulting average data rates forSite-A in the three cases are, respectively, equal to 35, 44, and 53 Mbps. The overall traffic request for the medium load configuration for the whole set of snapshot is represented in Figure 12 by means of curves. Note that their shape is irregular and clearly different from the one assumed in most of the related work.Figure 12
Load curves for medium load,Site-A andSite-B, and hexagons 0 to 2.x-axis reports the snapshot number.We first analyze the effects of interference on the system, using synthetic load curves with artificially unbalanced loadamong the hexagons. Then we compare the performance of our algorithm in various configurations against two baselines: a first one -∣- with all the micros always off and a second one m∣m with all the micros always on. The above analysis is performed both in case of uniform and nonuniform spatial user distributionwithin each hexagon. Finally, we will test our framework for robustness, verifying how it reacts to changes in the expected load.System performance will be evaluated in terms of the power variation overm∣m. We will consider only thevariable part of the power consumption and assess the saving of configuration X over baseline B as S=PvarX/PvarB-1.The main simulation parameters are summarized in Table1.Table 1
Simulation parameters.
Macro
Micro
Tx power
40 dBm
36 dBm
P
b
a
s
e
320 W
16 W
P
o
f
f
174 W
7 W
ρ
2.51 W
0.450 W
Antenna gain
18 dBi
11 dBi
Intercell distance
500 m
Path loss model
Urban macro (ITU-R)
Bandwidth
10 MHz (50 RBs)
#runs
5 (simulation) 20 (training)
#snapshots
96


### 5.1. Synthetic Load

In this section, we demonstrate the importance of keeping large-scale effects (such as intercell interference) into account. For this purpose we define a set ofsyntheticload curves, as depicted in Figure 13: hexagon 0 (h0)—the one being observed— has a constant requested load, whileall the others (1 to 56) have a variable one. Figure 14 shows the RB occupancy of h0 in the configurations previously defined (-∣- and m∣m) plus a third one (-∣m) withall the micros on, except the ones in hexagon 0. Two considerations are in order: first, the number of allocated RBs in h0, thus its power consumption, varies proportionally with the load of neighboring hexagons, despite the requested load in the former being constant (see curve m∣m). The interference from the other hexagons increases in fact with their load.Figure 13
Load curves for thesynthetic scenario.Figure 14
Power consumed inh0 for thesynthetic scenario. -∣-:all micros off; m∣m: all micros on; and -∣m:all micros onexcept in hexagon 0.Second, using micros offloads the macro node, thus avoiding saturation and using resources more efficiently at high loads, as shown by curvem∣m. Note that the interference suffered by h0, thus its power consumption, drops also when activating micro nodes in neighboring cells only (-∣m).


### 5.2. Realistic Load

We now analyze the behavior of the system with realistic traffic curves with 57 hexagons, each one composed of one macro and two micro nodes. Each set of three load curves is applied to the central triplet of hexagons and is then replied in the rest of the system using the same pattern, as represented in Figure15. With reference to the latter, all the hexagons of the same color will request the same data rate. However, UE is dropped randomly within each hexagon, thus hexagons will differ—possibly in a significant manner—in terms of requestedresources(which also depend on the UE SINR). This approach allows us to have diverse levels of request resources throughout the system, while still using realistic values of requested data rates.Figure 15
Multihexagon network for the realistic load scenario.As a first step we analyze the performance of our system with various values of the thresholdThm, then we will investigate the effects of Smax. Figure 16 shows the power variation in the central triple of hexagons. As we can see the latter is not affected by the value of Thm, regardless of the number of switches Smax. This happens because our algorithm tries to follow the suggested energy-optimal pattern, thus mitigating the effects of erroneous settings of Thm.Figure 16
Power variation with respect tom∣m, obtained inSite-A with uniform user distribution, for various Thm and Smax=∞.Figure17 instead shows the power variation inSite-A for various configurations of Smax. As we can see the highest savings are achieved at low loads, when micros are switched off most of the time. As the load increases, the impact of Smax increases as well, but not in a significant manner. Similar things occur withSite-B, as shown in Figure 18, with an impact of Smax that is even lower. This effect is due to the “smoothness” of the load curves and to the uniform distribution of UE among the system. When comparingSite-A andSite-B loads, higher saving is obtained with the former, as it has lower load requests throughout the day, thus allowing for longer periods with the micros turned off.Figure 17
Power variation with respect tom∣m, obtained inSite-A with uniform user distribution.Figure 18
Power variation with respect tom∣m, obtained inSite-B with uniform user distribution.To better understand the effects ofSmax, in Figure 19 we report the cell load of h0 inSite-A with high load. Moreover in the lower part of the figure we show three binary curves, each one representing the optimal energy-activation pattern for the micros, for three values of Smax, in the shape of three binary curves indicating the values of akh. There are two saturation zones (circled peaks at the top of the figure). In these cases every micromust be turned on so as to provide additional capacity. There are two more zones (boxed areas) where the macro is not saturated, but the amount of requested resources is such that having the micros on is energetically profitable. In the remaining snapshots the best solution is having the micros off. The first two conditions (saturation) are satisfied with every value of Smax, while the last one is fully achieved only for Smax, when micros are switched only when necessary, regardless of the number of switches.Figure 19
Switches forh0,Site-A with level = high, Smax=1,3,∞, and uniform user distribution.In Figures20 and 21 we show instead the total number of overloaded RBs during the whole day, that is, the number of RBs, beyond those available, which would be required to cope with the traffic request of a day. This allows us to measure the ability of the system to satisfy the traffic request itself. As we can see, configuration m∣m performs way better than -∣-; thus its performance has to be considered as the target of our algorithm. The latter in fact obtains the same number of overloaded RBs as m∣m, regardless of the choice of Smax, thus enforcing energy saving withno impact on the carried load.Figure 20
Per hexagon overloaded RBs over a day, reported as a percentage of the number of available RBs, obtained inSite-A with uniform user distribution.Figure 21
Total number of per hexagon overloaded RBs over a day, reported as a percentage of the number of available RBs, obtained inSite-B with uniform user distribution.


### 5.3. Nonuniform User Distribution

In this section, we will analyze the performance of our algorithm in case of nonuniform and time-varying distribution of UE within the system. For each hexagon, users are randomly dropped insideone hotspot of centerc and radiusr, as presented in Figure 7(a). Valuecis changed every five snapshots, alternating between two positions, closer to the macro (hs1) and to the micro (hs2), respectively. The value ofr is instead picked randomly in the interval [20m,30m], every snapshot.In Figures22 and 23 we show the power variation obtained with respect to m∣m, for bothSite-A andSite-B. The impact of Smax is in this case greater than with a uniform user distribution. Moreover, this effect increases with load, as the energy-wise efficiency of micros is higher with larger traffic demands. As in the previous scenario, the performance of our algorithm from the carried load point of view, is equal to those obtained with m∣m and is thus not shown.Figure 22
Power variation with respect tom∣m, obtained inSite-A with hotspot user distribution.Figure 23
Power variation with respect tom∣m, obtained inSite-B with hotspot user distribution.The analysis in Figures17–23 suggests that a larger Smax (which, as we remark, is an input datum and not a decision variable) yields diminishing returns in terms of power savings. Most of the power, in fact, is saved with the first few switch-offs—sometimes even a single one—beyond which topology alterations get progressively less useful. However, while theremustbe a point where increasing Smax stops yielding tangible benefits (unless load curves are collections of IID points, which is not what appears from measurements), its location depends on several factors: the shape of the load curves, the distribution of the position of UE, the power profiles of the nodes, and so on. In fact, Figures 22 and 23 show that increasing Smax from one to three in a hotspot scenario yields considerable benefits, unlike with a uniform distribution (see Figures 17 and 18).


### 5.4. Resilience Analysis

In this section, we will test how our solution reacts to variation in the expected load. More in detail, we want to check if theOnline Decideris able to recover from wrong suggestions coming from theOffline Analyzer or to react to unexpected changes in the actual load of the system.As a first step, given a load curve of an hexagon for a certain site, for every snapshot we take theoriginal bitrate request,calling it br, and we compute themodifiedbitrate request as b~r=br×γ, where γ is a random value extracted from a uniform distribution in the interval γL,γR. Note that we are not varying the number of UE within a hexagon, but only their load requests. Figure 24 shows a portion of theoriginal load curve in h0 forSite-A load, compared against threemodified load curves obtained with a γ extracted in 0.6,1.4 with three different seeds.Figure 24
Comparison oforiginal load against threemodified loads, computed with γ extracted from the interval 0.6,1.4 with three different seeds.Although the shape of the curve is roughly the same, pointwise variations can be clearly observed. We repeated the above procedure in order to obtain various modified load curves. For each of them we first fed thetrainer with the original load curve, and then we perform simulations using the modified ones. We used this method for bothSite-A andSite-B and for each load level. As we can see in Figures 25 and 26, the effects of the variations on the power saving are negligible in all cases. This means that, as far as the shape of the load curve is maintained, theOnline Decider is able to react to variation in the requested load.Figure 25
Power variation obtained usingSmax=1 with respect to m∣m, withoriginal Site-A load curves andmodified ones with γ randomly extracted from two intervals.Figure 26
Power variation obtained usingSmax=1 with respect to m∣m, withoriginal Site-B load curves andmodified ones with γ randomly extracted from two intervals.


## 5.1. Synthetic Load

In this section, we demonstrate the importance of keeping large-scale effects (such as intercell interference) into account. For this purpose we define a set ofsyntheticload curves, as depicted in Figure 13: hexagon 0 (h0)—the one being observed— has a constant requested load, whileall the others (1 to 56) have a variable one. Figure 14 shows the RB occupancy of h0 in the configurations previously defined (-∣- and m∣m) plus a third one (-∣m) withall the micros on, except the ones in hexagon 0. Two considerations are in order: first, the number of allocated RBs in h0, thus its power consumption, varies proportionally with the load of neighboring hexagons, despite the requested load in the former being constant (see curve m∣m). The interference from the other hexagons increases in fact with their load.Figure 13
Load curves for thesynthetic scenario.Figure 14
Power consumed inh0 for thesynthetic scenario. -∣-:all micros off; m∣m: all micros on; and -∣m:all micros onexcept in hexagon 0.Second, using micros offloads the macro node, thus avoiding saturation and using resources more efficiently at high loads, as shown by curvem∣m. Note that the interference suffered by h0, thus its power consumption, drops also when activating micro nodes in neighboring cells only (-∣m).


## 5.2. Realistic Load

We now analyze the behavior of the system with realistic traffic curves with 57 hexagons, each one composed of one macro and two micro nodes. Each set of three load curves is applied to the central triplet of hexagons and is then replied in the rest of the system using the same pattern, as represented in Figure15. With reference to the latter, all the hexagons of the same color will request the same data rate. However, UE is dropped randomly within each hexagon, thus hexagons will differ—possibly in a significant manner—in terms of requestedresources(which also depend on the UE SINR). This approach allows us to have diverse levels of request resources throughout the system, while still using realistic values of requested data rates.Figure 15
Multihexagon network for the realistic load scenario.As a first step we analyze the performance of our system with various values of the thresholdThm, then we will investigate the effects of Smax. Figure 16 shows the power variation in the central triple of hexagons. As we can see the latter is not affected by the value of Thm, regardless of the number of switches Smax. This happens because our algorithm tries to follow the suggested energy-optimal pattern, thus mitigating the effects of erroneous settings of Thm.Figure 16
Power variation with respect tom∣m, obtained inSite-A with uniform user distribution, for various Thm and Smax=∞.Figure17 instead shows the power variation inSite-A for various configurations of Smax. As we can see the highest savings are achieved at low loads, when micros are switched off most of the time. As the load increases, the impact of Smax increases as well, but not in a significant manner. Similar things occur withSite-B, as shown in Figure 18, with an impact of Smax that is even lower. This effect is due to the “smoothness” of the load curves and to the uniform distribution of UE among the system. When comparingSite-A andSite-B loads, higher saving is obtained with the former, as it has lower load requests throughout the day, thus allowing for longer periods with the micros turned off.Figure 17
Power variation with respect tom∣m, obtained inSite-A with uniform user distribution.Figure 18
Power variation with respect tom∣m, obtained inSite-B with uniform user distribution.To better understand the effects ofSmax, in Figure 19 we report the cell load of h0 inSite-A with high load. Moreover in the lower part of the figure we show three binary curves, each one representing the optimal energy-activation pattern for the micros, for three values of Smax, in the shape of three binary curves indicating the values of akh. There are two saturation zones (circled peaks at the top of the figure). In these cases every micromust be turned on so as to provide additional capacity. There are two more zones (boxed areas) where the macro is not saturated, but the amount of requested resources is such that having the micros on is energetically profitable. In the remaining snapshots the best solution is having the micros off. The first two conditions (saturation) are satisfied with every value of Smax, while the last one is fully achieved only for Smax, when micros are switched only when necessary, regardless of the number of switches.Figure 19
Switches forh0,Site-A with level = high, Smax=1,3,∞, and uniform user distribution.In Figures20 and 21 we show instead the total number of overloaded RBs during the whole day, that is, the number of RBs, beyond those available, which would be required to cope with the traffic request of a day. This allows us to measure the ability of the system to satisfy the traffic request itself. As we can see, configuration m∣m performs way better than -∣-; thus its performance has to be considered as the target of our algorithm. The latter in fact obtains the same number of overloaded RBs as m∣m, regardless of the choice of Smax, thus enforcing energy saving withno impact on the carried load.Figure 20
Per hexagon overloaded RBs over a day, reported as a percentage of the number of available RBs, obtained inSite-A with uniform user distribution.Figure 21
Total number of per hexagon overloaded RBs over a day, reported as a percentage of the number of available RBs, obtained inSite-B with uniform user distribution.


## 5.3. Nonuniform User Distribution

In this section, we will analyze the performance of our algorithm in case of nonuniform and time-varying distribution of UE within the system. For each hexagon, users are randomly dropped insideone hotspot of centerc and radiusr, as presented in Figure 7(a). Valuecis changed every five snapshots, alternating between two positions, closer to the macro (hs1) and to the micro (hs2), respectively. The value ofr is instead picked randomly in the interval [20m,30m], every snapshot.In Figures22 and 23 we show the power variation obtained with respect to m∣m, for bothSite-A andSite-B. The impact of Smax is in this case greater than with a uniform user distribution. Moreover, this effect increases with load, as the energy-wise efficiency of micros is higher with larger traffic demands. As in the previous scenario, the performance of our algorithm from the carried load point of view, is equal to those obtained with m∣m and is thus not shown.Figure 22
Power variation with respect tom∣m, obtained inSite-A with hotspot user distribution.Figure 23
Power variation with respect tom∣m, obtained inSite-B with hotspot user distribution.The analysis in Figures17–23 suggests that a larger Smax (which, as we remark, is an input datum and not a decision variable) yields diminishing returns in terms of power savings. Most of the power, in fact, is saved with the first few switch-offs—sometimes even a single one—beyond which topology alterations get progressively less useful. However, while theremustbe a point where increasing Smax stops yielding tangible benefits (unless load curves are collections of IID points, which is not what appears from measurements), its location depends on several factors: the shape of the load curves, the distribution of the position of UE, the power profiles of the nodes, and so on. In fact, Figures 22 and 23 show that increasing Smax from one to three in a hotspot scenario yields considerable benefits, unlike with a uniform distribution (see Figures 17 and 18).


## 5.4. Resilience Analysis

In this section, we will test how our solution reacts to variation in the expected load. More in detail, we want to check if theOnline Decideris able to recover from wrong suggestions coming from theOffline Analyzer or to react to unexpected changes in the actual load of the system.As a first step, given a load curve of an hexagon for a certain site, for every snapshot we take theoriginal bitrate request,calling it br, and we compute themodifiedbitrate request as b~r=br×γ, where γ is a random value extracted from a uniform distribution in the interval γL,γR. Note that we are not varying the number of UE within a hexagon, but only their load requests. Figure 24 shows a portion of theoriginal load curve in h0 forSite-A load, compared against threemodified load curves obtained with a γ extracted in 0.6,1.4 with three different seeds.Figure 24
Comparison oforiginal load against threemodified loads, computed with γ extracted from the interval 0.6,1.4 with three different seeds.Although the shape of the curve is roughly the same, pointwise variations can be clearly observed. We repeated the above procedure in order to obtain various modified load curves. For each of them we first fed thetrainer with the original load curve, and then we perform simulations using the modified ones. We used this method for bothSite-A andSite-B and for each load level. As we can see in Figures 25 and 26, the effects of the variations on the power saving are negligible in all cases. This means that, as far as the shape of the load curve is maintained, theOnline Decider is able to react to variation in the requested load.Figure 25
Power variation obtained usingSmax=1 with respect to m∣m, withoriginal Site-A load curves andmodified ones with γ randomly extracted from two intervals.Figure 26
Power variation obtained usingSmax=1 with respect to m∣m, withoriginal Site-B load curves andmodified ones with γ randomly extracted from two intervals.


## 6. Conclusions

This paper presented a framework for the activation/deactivation of nodes in a heterogeneous network, where the number of switch-ons in a day is constrained. We first analyzed the limits of both online- and offline-only algorithms, demonstrating how optimality requires unrealistic hypotheses. We then proposed and evaluated an online-offline method consisting of three phases: training, offline analysis, and online decision. The training phase is done offline and allows one to evaluate the energy consumption and saturation probabilities of alternative configurations, taking into account intercell interferenceat large. The offline analysis provides the optimal (constrained) switching pattern, and the online decision algorithm evaluates the current network load and makes switching decisions, erring on the safe side as far as saturation probability is concerned. Our framework works with arbitrary load curves and power figures, and it takes a predefined maximum number of switch-offs as an input. Simulation results show performance savings between 10 and 25% with both synthetic and real traffic loads and with both uniform and nonuniform user distributions. Future work includes developing and analyzing switching algorithms that takes into account more detailed information on the spatial distribution of UE, as well as the possible benefits given byenhanced Intercell Interference Coordination(eICIC) techniques at the eNBs. Both frequency-based (e.g., coordinated scheduling or partial frequency reuse) and time-based (e.g., provisioning of Absolute Blank Subframes) can easily be taken into account into our framework. All it takes is to modify expression (4), by correctly accounting the number of overlapping RBs between couples of eNBs depending on the eICIC technique.Finally, minimizing the number of topology alterations (i.e., switch-ons and switch-offs) required to achieve a target power saving in a network is a related problem that we plan to investigate in the future.



---

*Source: 2495934-2017-06-07.xml*