# Energy Cooperation in Ultradense Network Powered by Renewable Energy Based on Cluster and Learning Strategy

Authors: Chunhong Duo; Baogang Li; Yongqian Li; Yabo Lv
Journal: Wireless Communications and Mobile Computing
 (2017)

---

## Abstract

A new method about renewable energy cooperation among small base stations (SBSs) is proposed, which is for maximizing the energy efficiency in ultradense network (UDN). In UDN each SBS is equipped with energy harvesting (EH) unit, and the energy arrival times are modeled as a Poisson counting process. Firstly, SBSs of large traffic demands are selected as the clustering centers, and then all SBSs are clustered using dynamick-means algorithm. Secondly, SBSs coordinate their renewable energy within each formed cluster. The process of energy cooperation among SBSs is considered as Markov decision process. Q-learning algorithm is utilized to optimize energy cooperation. In the algorithm there are four different actions and their corresponding reward functions. Q-learning explores the action as much as possible and predicts better action by calculating reward. In addition, ε greedy policy is used to ensure the algorithm convergence. Finally, simulation results show that the new method reduces data dimension and improves calculation speed, which furthermore improves the utilization of renewable energy and promotes the performance of UDN. Through online optimization, the proposed method can significantly improve the energy utilization rate and data transmission rate.


---

## Body

## 1. Introduction

With the rapid development of mobile network, wireless data traffic has increased exponentially. More and more small base stations (SBSs) are needed to satisfy the traffic demand, which results in ultradense network (UDN). Meanwhile energy harvesting (EH) has become a very promising technology because of its flexible deployment and renewable energy [1]. The combination between SBS and EH can make it possible to harvest energy to extend the network running time. However, there are some difficulties for EH technology [2, 3], that is, the randomness and instability of renewable energy, the limited energy storage of EH devices, and so on.The energy cooperation in EH wireless networks has recently been studied extensively. There are lots of researches for maximizing network throughout, energy cooperation, traffic management, and so on. Energy cooperation schemes among different BSs are considered [4–6], and the energy efficiency (EE) is improved by adopting the evolutionary algorithms. The jointly optimal policy that maximizes sum-throughput is proposed [7–9]; both energy cooperation and traffic management are considered. The capacity region coincides with a traditionalK-user Gaussian MAC, and users can perform energy cooperation [10]. The cooperation between primary and secondary users at information and energy levels is considered [11]. These studies mainly focus on energy cooperation between the cells or users in cellular network or the hybrid power supply system, which aims at improving the EE and traffic management. With the increase of SBSs in the future communication network, it is necessary to divide SBSs into small groups for optimal energy cooperation.UDN has a much higher deployment density than current mobile network, which greatly improves the network throughput. Many studies on spectrum sensing, spectrum efficiency, and intercell interference have been published [12–15]. EE and resource management are studied in UDN [16–18], and a cluster-based EE resource allocation scheme is proposed [19]. Taking advantage of the density feature of SBS, enhancing the energy cooperation will promote the network performance. With the increasing density of SBSs, it is more efficient and practical to utilize cluster and online learning method for greater network throughput.There are also researches about energy supply especially EH technology. A system consisting of two cooperative microgrids is considered [20], and the two microgrids exchange renewable energy through the transmission line. Two energy sources for supplying the energy required for system operation are proposed, that is, an energy harvester and a constant energy source driven by a nonrenewable resource [21–23]. An energy harvesting circuit is equipped at the energy harvesting receiver and introduces various nonlinearities into the wireless power transfer [24]. A dense small cell network consisting of a set of small cells and a set of users is considered. In the network every user selects an SBS by itself, and multiple users can be served by a single SBS [25]. For simplicity, we only consider the renewable energy cooperation among clustered SBSs in UDN.Considering that clustering technique obtains optimal energy cooperation and reinforcement learning gains greater EE, we combine EH technology with clustered energy cooperation in UDN to solve EE problems. A renewable energy cooperation management algorithm is proposed based on clustered SBSs in UDN. Firstly, according to the geographical location and traffic load, sampling technique is used to determine the centers of data division, and then all SBSs are clustered using dynamick-means algorithm. Secondly, within each formed cluster, SBSs coordinate their renewable energy. The process of energy cooperation among SBSs is considered as MDP. Q-learning algorithm is adopted to optimize energy cooperation, in which ε greedy policy can ensure algorithm convergence. There are four different actions and their corresponding immediate reward functions for each SBS. The reward function represents the energy value after taking the corresponding action. Q-learning explores the action as much as possible, learns how to predict the relationship between the action and the reward, and furthermore predicts better actions by calculating rewards. At last, simulation results show that the new method reduces data dimension and improves calculation speed, which furthermore improves the utilization of renewable energy and promotes the performance of UDN. Through online optimization, the proposed method can significantly improve the energy utilization rate and data transmission rate.This paper is organized as follows: Section2 presents the system model according to MDP. In Section 3, we optimize the problem of energy cooperation among SBSs in UDN using clustering technology and reinforcement learning. Simulation results are given in Section 4. Section 5 concludes the paper.


## 2. System Model

In practice, the EH model depends on its specific implementation. There are solar panel and wind turbine-generator which can generate renewable energy, but the EH characteristics are different in both cases. The energy arrival times in the energy harvester can be modeled as a Poisson counting process [21]. Sinusoidal curve can also be selected [22]. In order to provide a general model for EH communication systems, we model it as a stochastic process in order to isolate the considered problem from specific implementation assumptions. In this paper, the location of SBSs in UDN is randomly deployed [26, 27], and each SBS is equipped with EH unit and a limited capacity battery. Assume that a limited time-slot (TS) system, the renewable energy, and the required data are arrived at the beginning of each TS. The channel state information is H(t), which is kept constant in the same TS. The processing of energy/data packet can be considered as one-order discrete Markov model [28]. In each TS, the amount of data SBS processed is Dn, and the minimum energy required for data transmission is EnT(t). The energy arrival times in the EH unit are modeled as a Poisson counting process with λE. The harvested energy EnH(t) in each TS is EnHt=λEt/t!∗e-λE,t=0,1,2,….In UDN, for a certain SBSn, the system parameters are given in System Parameters in UDN section.Energy charged to the battery iscn(t), cn(t)≥0; and energy discharged from the battery is dn(t), dn(t)≥0. At any time, SBS will charge/discharge energy to/from battery. There is at most one of cn(t) and dn(t) that is strictly positive, that is, cnt∗dnt=0.The SBS operation in TSt satisfies the following constraints (α represents battery charging efficiency):C1EnHt≥0EnTt≥0cnt≥0dnt≥0cnt∗dnt=00≤Bnt≤Bnmax0≤Bnt+αcnt-dnt≤Bnmax.We consider one hot region in UDN, which is shown in Figure1. The colored SBSs are the cluster centers, which form groups with the uncolored SBSs in their circles. Other SBSs join to the nearest group. The SBSs in one cluster achieve energy cooperation. For simplicity, we only consider the scenario of one user. For the request data rate of user, we ignore the fluctuation of data services and suppose that there is the full traffic data case. The user always has data to be received, and SBSs are in full load operation.Figure 1
System model.To simplify the system model, we give one SBS energy harvesting model in UDN, which is shown in Figure2.Figure 2
SBS energy harvesting model in UDN.To find the required energy to reliably transmit a data packet over the channel we consider Shannon’s capacity formula for Gaussian channels.D
n
t can be approximately calculated by(1)Dn=WΔTxlog2⁡1+HnPWN0,where P is the transmit power and N0 is the noise power density.E
n
T
(
t
) is an integer multiple of the energy unit [22] and can be calculated by (2)EnT=feDn,Hn=Dnlog2⁡N0Hn.In this paper, we define that EE in terms of bits/Hz/Joule is(3)EE=∑k=1Klog2⁡1+pk∗gkPf+Pc+φ∑k=1Kpk,where ∑k=1Kpk is the sum power consumption of K SBSs in one cluster, gk denotes the channel-gain-to-noise ratio, Pf denotes the addition power dissipation due to SBS’s circuitry, Pc is the static power dissipation, and φ is the power amplifier efficiency.A multiuser scenario can be derived similarly. At the beginning of eachTS, each SBS schedules its users, such that a single user is allocated to each subcarrier. Let μu,m=1 denote that subcarrier m is assigned to user u, and μu,m=0, otherwise. Accordingly, EE can be expressed as(4)EE=∑u=1U∑m=1Muu,m/Mlog2⁡1+pu,m∗gu,mPf+Pc+φ∑u=1U∑m=1Mpu,m,where U is the user number assigned for one SBS, M is the subcarrier number, and pu,m denotes power allocation.(5)gu,m=hm,u2N0W/M,where hm,u is the channel gain, N0 is the noise power spectral density of additive white Gaussian noise, and W is the bandwidth.


## 3. Energy Cooperation Optimization Algorithm Based on SBS Clustering and Learning Strategy

This section optimizes the problem of energy cooperation among SBSs in UDN using clustering technology and reinforcement learning.


### 3.1. Dynamick-Means Clustering Method for SBSs in UDN

In this paper, a cluster-based approach for maximizing EE in UDN is proposed. According to the different realization process, the common clustering methods are divided into hierarchical clustering and partition clustering. There are also clustering methods for large scale data sets. One of the most classic clustering methods isk-means, which uses cluster centers to represent the whole group. The cluster centers need to be updated repeatedly in the process of determining the final result [29, 30]. All data should be redivided before updating cluster centers, which makesk-means lose the ability to deal with very large scale data, and the execution time of these algorithms grows with the data number increases.Since there are more SBSs than the traditional network in UDN, it is very time-consuming to run the clustering algorithm on the whole data set. To remedy this problem, sampling strategy is selected, which can greatly save storage space and reduce computation amount. Sampling technology is used to select some samples from the original data. According to the cluster result, the distribution of the original data set is estimated. In this paper, the sampling technique is used to determine the centers of data division. If the distance between SBSs is too long, energy cooperation may cause power loss and transmission delay. We cluster SBSs in UDN according to distance and traffic. The sampled SBS is selected according to SBS traffic from large one to small, and the SBS traffic refers to the average data packets sent in the past week or month. Each SBS is viewed as a data point in a two-dimensional space, and the distances between sampling SBSs are saved [31, 32].Assume that there arem SBSs in UDN. The distances between all SBSs are saved in the matrix Mm∗m. The dynamic k-means algorithm is described as follows:(1)
Selectr sampling SBSs (according to SBS traffic from large to small) and save their distances in the matrix Dr∗r; each column represents the distances between one SBS and other sampling SBSs, Dr∗r=d(xi,xj)(i,j=1,2,…,r), where d(xi,xj) represents the distances between sampling SBSs.(2)
Find the minimum distance between sampling SBSs,min⁡d(xi,xj)(i≠j).(3)
Calculate the average value of each columnRi=∑j=1rdxi,xj/T-1i=1,2,…,r, where r-1 represents each column vector in the distance matrix Dr∗r which has a “0” element, which is the distance between each point and its own.(4)
Calculate the average value of allRi, R¯=∑i=1rRi/r.(5)
Calculate high density radiusR=R¯+min⁡dxi,xj; min⁡dxi,xj is added to make R big enough so as to ensure that most high density points are correctly labeled.(6)
Calculate cluster radiusr according to r=R+min⁡dxi,xj and select the two furthest points from sampling SBSs as the initial cluster centers and mark k=2 to ensure that the centers come from different clusters.(7)
Divide the data near the centers into two clusters according tor, find the next farthest point according to the centers, mark k=k+1, and divide the data again until all the data are clustered completely.(8)
Cluster the remaining SBSs into the nearest center point.(9)
Calculatem=r+r, and h=min⁡dxi,xj+min⁡dxi,xj. If the distance between the cluster centers is smaller than m and the distance between the boundary points is smaller than h, then combine the two clusters and label k=k-1.(10)
Give the final clustering results and thek value.How to determinek value is a very difficult problem. Once the k value is not reasonable, it is likely to lead to great errors in clustering. For this case, according to the data distribution properties and their distances, after a series of transformations, the final clustering number is obtained. Obtaining k value is a changing, dynamic process, and there is no need to know initial experience value. The dynamical division is closer to actual demand. It solves the problem that the algorithm needs manual input and improves the automatic clustering ability.EE is considered to be one of the main benefits from clustering architecture. When the network is dense, the benefit of clustering is the improved stability of cooperative relationships. Clusters are formed in order to maximize the time availability of clusters, hence in this paper maximizing the availability of energy cooperative. The clustering centers are all SBSs of large traffics, which is benefit for energy cooperation among SBSs of small traffics. It also ensures the supply of renewable energy and improves the data transmission rate of the network.


### 3.2.Q-Learning Approach for Energy Cooperation

Due to the instability of the renewable energy and the arbitrary distribution of SBSs, it is necessary to improve EE through energy cooperation. In this paper, we propose a renewable energy cooperation scheme among different SBSs, in which one SBS can collect/share energy from/to another SBS. The energy cooperation efficiency will be improved when the number of the cooperation SBSs is large.Consider energy cooperation among SBSs in UDN as the finite states and discrete time MDP.S,A,Patst,st+1,Ratst,st+1, where S is the finite environment state space; A is the finite system action space; Patst,st+1∈0,1 and Ratst,st+1, respectively, represent the state transition probability and the immediate reward of transferring the state from st to st+1 by taking action at. The probability and the immediate reward depend only on the current state st and the selected action at and are irrelevant to the past states and actions.In the proposed model, the system state ofnth SBS in TS t is Sn=EnHt,EnTt,Hnt,Bnt, and action set is A=a1,a2,a3,a4. At the beginning of each TS, SBS has four actions for each data packet, which is shown in Table 1.Table 1
Actions for SBS processing each data packet.
Actions
Transmitted/dropped and achieve/no energy cooperation
a1
Transmit the data packet and achieve energy cooperation
a2
Transmit the data packet and have no energy cooperation
a3
Drop the data packet and achieve energy cooperation
a4
Drop the data packet and have no energy cooperationAssume that the optimal state value function and the optimal action value function areV∗ and Q∗, which satisfy the Behrman optimal equation:(6)V∗s=maxπ⁡Vπs=maxa∈A∑s′∈SPas,s′Ras,s′+γV∗s′.Q∗s,a=maxπ⁡Qπs,a=∑s′∈SPas,s′Ras,s′+γmaxa′∈A⁡Q∗s′,a′As a result, the optimal policy can be obtained:(7)π∗s=arg⁡maxa∈A⁡∑s′∈SPas,s′Ras,s′+γV∗s′π∗s=arg⁡maxa∈A⁡Q∗s,a.The goal of MDP is to find the system’s optimal policyπ∗, which can be obtained by the optimal value function [33]. Q-learning algorithm is adopted, and its iterative formula is as follows:(8)Qst,at⟵Qst,at+δrt+γmaxa⁡Qst+1,a-Qst,at,where (st,at) is a state-action pair in TS t, δ(0<δ<1) is the learning factor, and γ(0<γ<1) is the discount factor.Q iterative learning uses the reward of state-action pair as the evaluation function. First initialize Q value, then determine the action at in the state st according to ε greedy policy, get the knowledge and experience of training samples (st,at,st+1,Rt), and then modify Q value. When agent gets the target state, the algorithm terminates one iteration loop. The algorithm starts from the initial state again until the end of learning.Q-learning algorithm is applied in renewable energy cooperation within one SBS cluster based on UDN, and its process is as follows:(1)
Initialize:Q(s,a)← any value, δ and γ: given values.(2)
Repeat.Given initial stateS: EnH0,EnT0,Hn0,Bn0.Repeat.①  Chooseat according to ε greedy policy, then obtain Rt and st+1.②Qst,at←Qst,at+δ[rt + γmaxa⁡Qst+1,a  −  Qst,at].③st←st+1.Untilst is termination state.Until allQ(s,a) are converged.(3)
Output the final policy:πs=arg⁡maxa∈A⁡Qs,a.In ①,at has four choices and the corresponding rt shown in Table 2 (α represents storage efficiency of the battery, and β represents resistance loss).Table 2
Four choices ofat and the corresponding rt.
at
Correspondingrt
a1
Transmit the data packet and achieve energy cooperationrt=EnTt+βEnHt-EnTt-Bnt-Bnt-1α
a2
Transmit the data packet and have no energy cooperationrt=EnTt
a3
Drop the data packet and achieve energy cooperationrt=βEnHt-Bnt-Bnt-1α
a4
Drop the data packet and have no energy cooperationrt=0Actually, four actives can be combined together:(9)rt=A×EnTt+B×βEnHt-EnTt-Bnt-Bnt-1α.Their values are listed in Table3.Table 3
Values ofA and B according to different actives.
Action
A
B
rt
a1
1
1
r
t
=
E
n
T
(
t
)
+
β
E
n
H
t
-
E
n
T
t
-
B
n
t
-
B
n
t
-
1
α
a2
1
0
r
t
=
E
n
T
t
a3
0
1
r
t
=
β
E
n
H
t
-
B
n
t
-
B
n
t
-
1
α
a4
0
0
r
t
=
0The reward function is actually the sum of the energy for sending data package and the energy for cooperation. In eachTS, it encourages sending data and carrying out energy cooperation.InQ-learning algorithm, actions with the highest Q values at a particular state should be taken at each step. The agent who rigidly follows this rule might underperform since the same decision will be investigated over and over again. In order to be exploited, the state-action pair needs to be explored firstly [34]. In this paper, we utilize ε greedy policy (0<ε<1), which is commonly used during the process of state-action space exploration. It enforces sporadic jumps to suboptimal states for the exploration purposes, but also to detect changes of the environmental conditions. Whenever a decision is to be made, the one will be picked at random with the (1-ε) probability, which is given to the action with the highest Q value.Q-learning converges to the optimal Q function. While t→∞, Q(s,a) converges to Q∗(s,a) with probability 1 [35], and the convergence rate is related to many factors. The convergence rate increases with the value of δ and the number of learning iterations NL and decreases with the number of a, s, and γ [36]. Action selection follows the ε greedy policy with probability (1-ε) at each TS. The exploration probability is ε, and the exploitation probability is (1-ε).


## 3.1. Dynamick-Means Clustering Method for SBSs in UDN

In this paper, a cluster-based approach for maximizing EE in UDN is proposed. According to the different realization process, the common clustering methods are divided into hierarchical clustering and partition clustering. There are also clustering methods for large scale data sets. One of the most classic clustering methods isk-means, which uses cluster centers to represent the whole group. The cluster centers need to be updated repeatedly in the process of determining the final result [29, 30]. All data should be redivided before updating cluster centers, which makesk-means lose the ability to deal with very large scale data, and the execution time of these algorithms grows with the data number increases.Since there are more SBSs than the traditional network in UDN, it is very time-consuming to run the clustering algorithm on the whole data set. To remedy this problem, sampling strategy is selected, which can greatly save storage space and reduce computation amount. Sampling technology is used to select some samples from the original data. According to the cluster result, the distribution of the original data set is estimated. In this paper, the sampling technique is used to determine the centers of data division. If the distance between SBSs is too long, energy cooperation may cause power loss and transmission delay. We cluster SBSs in UDN according to distance and traffic. The sampled SBS is selected according to SBS traffic from large one to small, and the SBS traffic refers to the average data packets sent in the past week or month. Each SBS is viewed as a data point in a two-dimensional space, and the distances between sampling SBSs are saved [31, 32].Assume that there arem SBSs in UDN. The distances between all SBSs are saved in the matrix Mm∗m. The dynamic k-means algorithm is described as follows:(1)
Selectr sampling SBSs (according to SBS traffic from large to small) and save their distances in the matrix Dr∗r; each column represents the distances between one SBS and other sampling SBSs, Dr∗r=d(xi,xj)(i,j=1,2,…,r), where d(xi,xj) represents the distances between sampling SBSs.(2)
Find the minimum distance between sampling SBSs,min⁡d(xi,xj)(i≠j).(3)
Calculate the average value of each columnRi=∑j=1rdxi,xj/T-1i=1,2,…,r, where r-1 represents each column vector in the distance matrix Dr∗r which has a “0” element, which is the distance between each point and its own.(4)
Calculate the average value of allRi, R¯=∑i=1rRi/r.(5)
Calculate high density radiusR=R¯+min⁡dxi,xj; min⁡dxi,xj is added to make R big enough so as to ensure that most high density points are correctly labeled.(6)
Calculate cluster radiusr according to r=R+min⁡dxi,xj and select the two furthest points from sampling SBSs as the initial cluster centers and mark k=2 to ensure that the centers come from different clusters.(7)
Divide the data near the centers into two clusters according tor, find the next farthest point according to the centers, mark k=k+1, and divide the data again until all the data are clustered completely.(8)
Cluster the remaining SBSs into the nearest center point.(9)
Calculatem=r+r, and h=min⁡dxi,xj+min⁡dxi,xj. If the distance between the cluster centers is smaller than m and the distance between the boundary points is smaller than h, then combine the two clusters and label k=k-1.(10)
Give the final clustering results and thek value.How to determinek value is a very difficult problem. Once the k value is not reasonable, it is likely to lead to great errors in clustering. For this case, according to the data distribution properties and their distances, after a series of transformations, the final clustering number is obtained. Obtaining k value is a changing, dynamic process, and there is no need to know initial experience value. The dynamical division is closer to actual demand. It solves the problem that the algorithm needs manual input and improves the automatic clustering ability.EE is considered to be one of the main benefits from clustering architecture. When the network is dense, the benefit of clustering is the improved stability of cooperative relationships. Clusters are formed in order to maximize the time availability of clusters, hence in this paper maximizing the availability of energy cooperative. The clustering centers are all SBSs of large traffics, which is benefit for energy cooperation among SBSs of small traffics. It also ensures the supply of renewable energy and improves the data transmission rate of the network.


## 3.2.Q-Learning Approach for Energy Cooperation

Due to the instability of the renewable energy and the arbitrary distribution of SBSs, it is necessary to improve EE through energy cooperation. In this paper, we propose a renewable energy cooperation scheme among different SBSs, in which one SBS can collect/share energy from/to another SBS. The energy cooperation efficiency will be improved when the number of the cooperation SBSs is large.Consider energy cooperation among SBSs in UDN as the finite states and discrete time MDP.S,A,Patst,st+1,Ratst,st+1, where S is the finite environment state space; A is the finite system action space; Patst,st+1∈0,1 and Ratst,st+1, respectively, represent the state transition probability and the immediate reward of transferring the state from st to st+1 by taking action at. The probability and the immediate reward depend only on the current state st and the selected action at and are irrelevant to the past states and actions.In the proposed model, the system state ofnth SBS in TS t is Sn=EnHt,EnTt,Hnt,Bnt, and action set is A=a1,a2,a3,a4. At the beginning of each TS, SBS has four actions for each data packet, which is shown in Table 1.Table 1
Actions for SBS processing each data packet.
Actions
Transmitted/dropped and achieve/no energy cooperation
a1
Transmit the data packet and achieve energy cooperation
a2
Transmit the data packet and have no energy cooperation
a3
Drop the data packet and achieve energy cooperation
a4
Drop the data packet and have no energy cooperationAssume that the optimal state value function and the optimal action value function areV∗ and Q∗, which satisfy the Behrman optimal equation:(6)V∗s=maxπ⁡Vπs=maxa∈A∑s′∈SPas,s′Ras,s′+γV∗s′.Q∗s,a=maxπ⁡Qπs,a=∑s′∈SPas,s′Ras,s′+γmaxa′∈A⁡Q∗s′,a′As a result, the optimal policy can be obtained:(7)π∗s=arg⁡maxa∈A⁡∑s′∈SPas,s′Ras,s′+γV∗s′π∗s=arg⁡maxa∈A⁡Q∗s,a.The goal of MDP is to find the system’s optimal policyπ∗, which can be obtained by the optimal value function [33]. Q-learning algorithm is adopted, and its iterative formula is as follows:(8)Qst,at⟵Qst,at+δrt+γmaxa⁡Qst+1,a-Qst,at,where (st,at) is a state-action pair in TS t, δ(0<δ<1) is the learning factor, and γ(0<γ<1) is the discount factor.Q iterative learning uses the reward of state-action pair as the evaluation function. First initialize Q value, then determine the action at in the state st according to ε greedy policy, get the knowledge and experience of training samples (st,at,st+1,Rt), and then modify Q value. When agent gets the target state, the algorithm terminates one iteration loop. The algorithm starts from the initial state again until the end of learning.Q-learning algorithm is applied in renewable energy cooperation within one SBS cluster based on UDN, and its process is as follows:(1)
Initialize:Q(s,a)← any value, δ and γ: given values.(2)
Repeat.Given initial stateS: EnH0,EnT0,Hn0,Bn0.Repeat.①  Chooseat according to ε greedy policy, then obtain Rt and st+1.②Qst,at←Qst,at+δ[rt + γmaxa⁡Qst+1,a  −  Qst,at].③st←st+1.Untilst is termination state.Until allQ(s,a) are converged.(3)
Output the final policy:πs=arg⁡maxa∈A⁡Qs,a.In ①,at has four choices and the corresponding rt shown in Table 2 (α represents storage efficiency of the battery, and β represents resistance loss).Table 2
Four choices ofat and the corresponding rt.
at
Correspondingrt
a1
Transmit the data packet and achieve energy cooperationrt=EnTt+βEnHt-EnTt-Bnt-Bnt-1α
a2
Transmit the data packet and have no energy cooperationrt=EnTt
a3
Drop the data packet and achieve energy cooperationrt=βEnHt-Bnt-Bnt-1α
a4
Drop the data packet and have no energy cooperationrt=0Actually, four actives can be combined together:(9)rt=A×EnTt+B×βEnHt-EnTt-Bnt-Bnt-1α.Their values are listed in Table3.Table 3
Values ofA and B according to different actives.
Action
A
B
rt
a1
1
1
r
t
=
E
n
T
(
t
)
+
β
E
n
H
t
-
E
n
T
t
-
B
n
t
-
B
n
t
-
1
α
a2
1
0
r
t
=
E
n
T
t
a3
0
1
r
t
=
β
E
n
H
t
-
B
n
t
-
B
n
t
-
1
α
a4
0
0
r
t
=
0The reward function is actually the sum of the energy for sending data package and the energy for cooperation. In eachTS, it encourages sending data and carrying out energy cooperation.InQ-learning algorithm, actions with the highest Q values at a particular state should be taken at each step. The agent who rigidly follows this rule might underperform since the same decision will be investigated over and over again. In order to be exploited, the state-action pair needs to be explored firstly [34]. In this paper, we utilize ε greedy policy (0<ε<1), which is commonly used during the process of state-action space exploration. It enforces sporadic jumps to suboptimal states for the exploration purposes, but also to detect changes of the environmental conditions. Whenever a decision is to be made, the one will be picked at random with the (1-ε) probability, which is given to the action with the highest Q value.Q-learning converges to the optimal Q function. While t→∞, Q(s,a) converges to Q∗(s,a) with probability 1 [35], and the convergence rate is related to many factors. The convergence rate increases with the value of δ and the number of learning iterations NL and decreases with the number of a, s, and γ [36]. Action selection follows the ε greedy policy with probability (1-ε) at each TS. The exploration probability is ε, and the exploitation probability is (1-ε).


## 4. Numerical Simulations

According to the traffic amount of SBS from large to small, their relative positions in UDN are marked as Table4. All SBSs satisfy the constraints C1.Table 4
The positions of SBS.
xi
58
42
52
33
43
22
58
76
53
64
21
38
79
68
46
57
80
5
60
5
yi
69
21
84
63
13
20
61
63
37
58
45
4
2
31
1
38
68
9
3
6120 SBSs are, respectively,xi,yi, i=1,2,…,20, which are denoted by ith data point. 20 SBSs are clustered according to dynamick-means clustering method mentioned in Section 3.1. The sampling rate is 50%; that is, the (1–10) data points are selected. As shown in Figure 3, the cluster radius is r=50.7556, the number of cluster is k=3, and the cluster centers are [x3,y3], [x5,y5], and [x6,y6], which are denoted as ⊕. The three final clusters are 3⊕,1,4,7,8,10,17, 5⊕,2,9,12,13,14,15,16,19, and 6⊕,11,18,20.Figure 3
The dynamick-means clustering result for SBSs in UDN.Through dynamick-means clustering, without knowing the k value before, SBSs that have large business amount can be selected as the cluster centers. There are relatively larger energy demands for cluster centers, which can effectively improve the utilization rate of energy. In addition, Q-learning in this paper only considers the energy cooperation between the same clusters, which can reduce the dimension disaster problem caused by too many states.In the numerical analysis, we take one class{6⊕,11,18,20} as an example. All parameters are based on an IEEE802.15.4e [36] communication system. The system parameters are listed in Table 5. Each time-slot ΔTS is 10 ms, in which 5 ms is used to send data [30] and 5 ms to zero signal level. The channel state at TS n is Hn, Hn∈{h1,h2}. The channel state transition probability function is characterized by PHn(h1,h1)=PHn(h2,h2)=0.9. The transmit power is P. We consider Shannon’s capacity formula for Gaussian channels. The battery capacity is 5, that is, Bn=0,1,2,3,4,5. Each basic energy unit is 2.5μj [37, 38]. The possible data packet sizes are 300 bits or 600 bits.Table 5
System parameters.
Parameters
Values
W
2 MHz
ΔTS
10 ms
N0
10−20.4 w/Hz
h1
1.655∗10-13
h2
3.311∗10-13
PHn
0.9
Bmax
5
γ
0.9As shown in Figure4, Q-learning approach for energy cooperation in one class is convergent. The horizontal axis is the iteration times (the time of one iteration is 10 ms, and it is the same as other figures), and the vertical axis is the difference between the adjacent two sampling value functions. The line uses 5-degree polynomial fitting, and the sampling interval is 100μs (it is the same as in Figures 4–10). The change of the value function proves the algorithm’s convergence. When the iteration number (NL) reaches about 6×104, the function value is basically unchanged.Figure 4
Algorithm convergence process.Figure 5
Package transmission rate forδ=0.5,0.6,0.7.Figure 6
Package transmission rate forε=0.1,0.2,0.3.Figure 7
Data transmission rate forBmax={5,6,7,8,9} and PHn=0.9.Figure 8
Data transmission rate forPHn=0.5,0.6,0.7,0.8,0.9 and Bmax=5.Figure 9
Data transmission rate with and without energy cooperation.Figure 10
Average energy efficiency with and without energy cooperation.In Figure5 it shows that the learning factor δ in Q-learning algorithm can influence the iteration times when the packet transmission rate reaches stable. When δ=0.5 and NL≈3.0∗104, the black diamond line is basically unchanged; when δ=0.6 and NL≈2.5∗104, the triangle blue line reaches stable; when δ=0.7 and NL≈1.5∗104, the red star line keeps stable. We can conclude that Q-learning algorithm can keep stable with smaller iteration times as δ increases.As shown in Figure6, ε greedy policy can ensure the convergence of Q-learning algorithm. When ε uses different values, it can influence the final package transmission rate. When ε=0.1, the package transmission rate reaches about 60% as NL increases; when ε=0.2, it reaches about 55%; when ε=0.3, it reaches about 50%. We can conclude that Q-learning algorithm is able to learn the optimal policy with increasing accuracy as ε decreases.In Figure7 we show the effect of the max battery size Bmax on the expected data transmission rate. We can conclude that the expected data transmission rate increases with Bmax for the proposed algorithm. In our model, the system state of nth SBS in TS t is Sn=EnHt,EnTt,Hnt,Bnt, in which all the parameter values are finite and discrete. If Bmax has a relatively larger value, then Bnt∈0,1,…,Bmax has more choices, which leads to increased computation and dimension disaster in Q learning. In this paper we make Bmax=5 to simplify the algorithm, but we have executed exhaustive numerical simulations with different parameter settings and observed similar results.Figure8 displays the data transmission rate for different PHn values. We see that the expected data transmission rate increases with PHn. It means that the more stable the channel state is, the higher the data transmission rate is. In class 6⊕,11,18,20, when PHn=0.5, the data transmission rate is about 28 kbit/s; when PHn=0.9, the data transmission rate is about 40 kbit/s. As PHn increases, EH process becomes less random, and the proposed algorithm can better estimate its future states and adapt to it.In Figure9 it shows the relationship between the data transmission rate and iteration times. Energy cooperation can improve the energy utilization; that is, the data transmission rate can be higher through energy cooperation. The black diamond line which represents the rate of energy cooperation reaches about 28 kbit/s, and the green triangular line which represents the rate of no energy cooperation is below 24 kbit/s. The transmission rate through energy cooperation is about 15% higher than no energy cooperation. It is the same with other clusters in UDN. The proposed method in this paper can significantly improve the data transmission rate.The ratio of the network throughput to the power consumption per unit area is defined as EE (the energy efficiency). The energy efficiency metric is a performance indicator that measures the benefit-cost ratio by comparing the achievable rate to the energy costs. In Figure10 we illustrate, together with the performance of the other approaches, the expected average energy efficiency by the proposed approach against the number of learning iterations times. It can be observed that the average EE of energy cooperation is higher than that of no energy cooperation. The black curve is more closer to the Offline-LP algorithm. The proposed method can significantly improve energy utilization rate.


## 5. Conclusion

This paper presents a renewable energy cooperation management algorithm based on cluster and learning strategy in UDN. Firstly, according to the geographical location and traffic load, SBSs are clustered using dynamick-means algorithm, in which sampling technology is utilized to improve computation speed and clustering effect. Secondly, within each formed cluster, SBSs coordinate their renewable energy. The process of energy cooperation is considered as MDP. Q-learning algorithm is adopted to optimize energy cooperation, which considers four immediate reward functions, and the convergence of the algorithm is realized by ε greedy policy. Thirdly, simulation results show that the new method can improve the utilization of renewable energy and promote the data transmission rate. At last, conclusion and future research directions are presented, which include energy cooperation between clusters, combination of renewable energy, smart grid, and so on.



---

*Source: 2794324-2017-10-16.xml*