# Reinforcement Learning Optimization for Energy-Efficient Cellular Networks with Coordinated Multipoint Communications

Authors: Huibin Lu; Baozhu Hu; Zhiyuan Ma; Shuhuan Wen
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

Recently, there is an emerging trend of addressing “energy efficiency” aspect of wireless communications. And coordinated multipoint (CoMP) communication is a promising method to improve energy efficiency. However, since the downlink performance is also important for users, we should improve the energy efficiency as well as keeping a perfect downlink performance. This paper presents a control theoretical approach to study the energy efficiency and downlink performance issues in cooperative wireless cellular networks with CoMP communications. Specifically, to make the decisions for optimal base station grouping in energy-efficient transmissions in CoMP, we develop a Reinforcement Learning (RL) Algorithm. We apply theQ-learning of the RL Algorithm to get the optimal policy for base station grouping with introduction of variations at the beginning of the Q-learning to prevent Q from falling into local maximum points. Simulation results are provided to show the process and effectiveness of the proposed scheme.


---

## Body

## 1. Introduction

The continuously growing demand for ubiquitous wireless access leads to the rapid development of wireless cellular networks during the last decade [1–3]. Such tremendous growth in wireless industry has made it become one of the leading sources of world energy consumption and is expected to grow dramatically in the future. Rapidly rising energy costs and increasingly rigid environmental standards have led to an emerging trend of addressing “energy efficiency” aspect of wireless communication technologies.On account of the deterioration of air pollution and the worsening of greenhouse effect, optimization of energy utilization and the sustainable development have become one of the hot topics in academia. Base station costs much energy consumption in wireless cellular communication networks, which causes huge waste in the case of the number of users reducing to little. It goes against the energy optimization and the implementation of environmental protection. Therefore, it is necessary to reduce the energy consumption of base stations in order to optimize the energy efficiency of wireless cellular network.In a typical wireless cellular network, base stations account for about 70% of the total energy consumption [4]. In addition, a base station consumes even more than 90% of its peak energy when there is little or no traffic [5]. In allusion to the energy efficiency optimization, base station sleeping strategy is proposed, in which the base station should be turned off when the traffic is low and the surrounding base stations cooperate to serve the users. It combines the coordinated multipoint communication and base station sleeping strategy in order to implement an optimal CoMP grouping to sever the sleeping cell. Coordinated multipoint communication is a new method that helps with the dynamic base station cooperation, where signals transmitted or received by spatially separated antenna sites are jointly processed [6].The Reinforcement Learning Algorithm is one of the important methods in the Machine Learning. In this paper, the Reinforcement Learning Algorithm is used to optimize the cooperation of base station in CoMP communication. Reinforcement Learning (RL) is learning through the direct experimentation. It does not assume the existence of a teacher that provides examples upon which learning of a task takes place. Instead, in RL the experience is the only teacher. With historical roots on the study of conditioned reflexes, RL gets its advantages in fields as diverse as Operational Research and Robotics because of its theoretical relevance and potential applications. In this paper, the RL Algorithm operates for the Operational Research in CoMP communication.The rest of this paper is organized as follows. Section2 presents the system models. Section 3 presents the RL Algorithm in in CoMP communication. Simulation results are presented and discussed in Section 4. Finally, we conclude this study in Section 5.


## 2. System Models

### 2.1. Coordinated Multipoint Communications

Coordinated Multipoint Communication is a key feature in LTE-Advanced technologies. For the sake of better application of CoMP in practice, we build small scale model units to research the CoMP scheme, which supplies higher energy efficiency for base stations and better communication performance for users. In this way can we “green” the field of communications.In the wireless cellular networks, one small scale model unit consists of 19 cellular cells, in which the most lightly loaded cell is chosen as the center one. The base stations in the second channel are too far to serve the users in the center cell, so they are considered as the noise source. Then the first channel hexagonal cellular cells are numbered from 0 to 6 with base stations located in the center of the cells, and it is 500 m between two base stations as shown in Figure1. The center base station is switched off and the users are served by the dynamically CoMP cooperating of base stations 1~6. The number and locations of the users in the 0 cell are both optional and alterable, so the CoMP scheme must be adjusted along with that. As the definition of CoMP is explained initially in 3GPP 36.814 [7] as dynamic coordination among multiple geographically separated points referred to as CoMP cooperating set for downlink transmission and uplink reception, the scheme is to find the number and locations of the base stations participating in the CoMP to provide the best communications as the environment changes. The CoMP scheme is considered in the situation where the downlink payload and data are available at each point in the CoMP cooperating set and downlink payload is transmitted on Physical Downlink Shared Channel (PDSCH) from multiple points in the CoMP set [8]. However in this paper, we mainly focus on the downlink transmission in the perfect channel state.Figure 1
CoMP cells model.


### 2.2. Large Scale Propagation and Pathloss Models

Pathloss in wireless communication is defined as the difference in dB between the transmitted and the received signal powers due to the attenuation during the propagation [9]. The traditional log-normal shadowing for large scale pathloss modeling is formulated as
(1)
P
RX
(
dBm
)
=
P
TX
(
dBm
)
-
P
L
(
dBm
)
,
where P
RX represents the received signal power at the user equipment (UE), P
TX represents the transmitted signal power at the severing base station, and the pathloss is denoted by P
L formulated as
(2)
P
L
(
dBm
)
=
P
L
(
d
o
)
+
10
n
lg
(
d
d
o
)
+
x
σ
,
where P
L
(
d
o
) denotes the pathloss at the reference distance, d represents the propagation distance, n is the path loss exponent, and x
σ is the Gaussian random variable with zero mean and standard deviation σ modeling the shadowing effect of the media [8].Urban Macro (UMa) pathloss model is used according to the below equations and parameters specified in Table1 with respect to ITU for line of sight (LoS) and none line of sight (NLoS) scenarios. LoS probability is model as a Bernoulli random variable [10]:
(3)
P
L
LoS
(
dB
)
=
22
lg
d
+
28
+
20
lg
f
c
,
5000
m
<
d
<
d
BP
;
P
L
LoS
(
dB
)
=
40
lg
d
+
7.8
+
2
lg
f
c
-
18
lg
h
BS
-
lg
h
UT
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
d
BP
<
d
<
5000
m
;
P
L
NLoS
=
161.04
-
7.11
lg
h
B
-
[
24.37
-
3.7
×
(
h
h
BS
)
2
]
×
lg
h
BS
+
(
433.42
-
3.1
lg
h
BS
)
×
(
lg
d
-
3
)
+
20
lg
f
c
-
[
3.2
lg
(
11.75
h
UT
)
2
-
4.97
]
,
P
rob
(
LoS
)
=
min
⁡
⁡
(
18
d
,
1
)
×
(
1
-
e
-
d
/
63
)
+
e
-
d
/
63
.Table 1
Parameters for UMA pathloss model.
Definition
Parameter
Value
Carrier frequency
f
c
2110 MHz
Base station antenna height
h
BS
24 m
User terminal antenna height
h
UT
0.5 m
Average building height
h
B
20 m
Average street width
L
20 m
LoS shadowing
σ
LoS
4 dB
NLoS shadowing
σ
NLoS
6 dB
Break point distance
d
BP
337.6 m
Transmission power
P
TX
20 W


### 2.3. Downlink Performance

When making CoMP scheme decision, energy efficiency is not the only index that we take into consideration. We should also pay attention to the downlink performance to guarantee the quality of communications.The initial motivation for CoMP was to increase the cell edge user throughput and spectral efficiency by making use of intercell orthogonal resource assignments [11]. And the received SINR is calculated as
(4)
SINR
=
P
RX
(
n
=
j
)
∑
n
≠
j
P
RX
(
n
)
+
P
noise
,
where P
RX
(
n
=
j
) is the received signal power from the serving base station using the UMa large scale pathloss model while the remaining P
RX act as the interferers and the receiver noise power P
noise formulated as
(5)
P
noise
(
dBm
)
=
N
0
+
10
lg
W
(
i
)
,
where N
0
=
-
174 dB/Hz is the noise spectral density and W
(
i
) is the frequency bandwidth assigned to the user i.However in the CoMP cellular system, members of the CoMP scheme perform joint scheduling on PDSCH to transfer the user plane data using TM-9. So the received downlink SINR in joint transmission systems is formulized according to [8] as
(6)
SIN
R
CoMP
=
∑
n
∈
N
JT
P
RX
(
n
)
∑
n
∈
N
∖
N
JT
P
RX
(
n
)
+
P
noise
,
where the N
JT represents the CoMP set while the N
∖
N
JT represents the received power outside the CoMP set. In this way, both the SINR and the performance of the cell edge users’ communications are increased.Downlink capacity received by each useri is formulated as
(7)
C
(
i
)
=
W
(
i
)
log
⁡
2
[
1
+
SIN
R
CoMP
(
i
)
]
.Depending on the user location and mobility, each useri has a distinct CoMP transmission set N
JT
(
i
). Since the downlink capacity is also an important index in CoMP communication, it is also necessary to coordinate all the N
JT
(
i
) to find the best CoMP scheme.


### 2.4. Bits/Joule Energy Efficiency

One common method to measure energy efficiency is to use bits per Joule. For a CoMP schemeN
JT, energy efficiency can be defined as the ratio of transmission capacity to transmission power as follows:
(8)
C
(
N
JT
)
=
∑
C
N
JT
(
i
)
,
E
(
N
JT
)
=
C
(
N
JT
)
P
CoMP
,
where C
(
N
JT
) is the total downlink capacity and P
CoMP is the total transmission power in CoMP scheme N
JT, which is expressed as
(9)
P
CoMP
=
∑
n
∈
N
JT
P
(
n
)
,
where P
(
n
) is the total power consumption for base station n in CoMP scheme N
JT, which is calculated using the assumptions from [12] and [13] as
(10)
P
(
n
)
=
N
S
×
N
PA
/
sector
×
(
p
TX
p
Aeff
+
P
SP
)
×
(
1
+
C
C
)
(
1
+
C
PSBB
)
+
P
BH
,
where P
(
n
) is the radiated power per base station, P
SP is the signal processing power per base station, and P
BH is the power due to backhauling. p
Aeff is the power amplifier efficiency and C
C and C
PSBB are the cooling and battery backup losses in the system.The signal processing powerP
SP per sector as a function of different cooperation sizes scales as
(11)
P
SP
=
58
×
(
0.87
+
0.1
N
C
+
0.03
N
C
2
)
,
where N
C is the number of base stations participating in the CoMP scheme N
JT.Backhauling power consumptionP
BH for base stations using CoMP is modeled in [12] as
(12)
P
BH
=
C
BH
100
Mbits
/
sec
×
50
W
,
where C
BH is a given average backhaul requirement per base station. The expression of C
BH is expressed as
(13)
C
BH
=
N
C
×
(
2
N
C
)
×
p
×
q
T
S
,
where p is the additional pilot density, q is the CSI signaling under CoMP network, and T
s
=
66.7 μs is the sample period which is the reciprocal of the assumed OFDM subcarrier spacing at 15 kHz [8].As the models above, we try to get the best CoMP schemeN
JT which both ensures the quality of the communication and maximizes the energy efficiency.


## 2.1. Coordinated Multipoint Communications

Coordinated Multipoint Communication is a key feature in LTE-Advanced technologies. For the sake of better application of CoMP in practice, we build small scale model units to research the CoMP scheme, which supplies higher energy efficiency for base stations and better communication performance for users. In this way can we “green” the field of communications.In the wireless cellular networks, one small scale model unit consists of 19 cellular cells, in which the most lightly loaded cell is chosen as the center one. The base stations in the second channel are too far to serve the users in the center cell, so they are considered as the noise source. Then the first channel hexagonal cellular cells are numbered from 0 to 6 with base stations located in the center of the cells, and it is 500 m between two base stations as shown in Figure1. The center base station is switched off and the users are served by the dynamically CoMP cooperating of base stations 1~6. The number and locations of the users in the 0 cell are both optional and alterable, so the CoMP scheme must be adjusted along with that. As the definition of CoMP is explained initially in 3GPP 36.814 [7] as dynamic coordination among multiple geographically separated points referred to as CoMP cooperating set for downlink transmission and uplink reception, the scheme is to find the number and locations of the base stations participating in the CoMP to provide the best communications as the environment changes. The CoMP scheme is considered in the situation where the downlink payload and data are available at each point in the CoMP cooperating set and downlink payload is transmitted on Physical Downlink Shared Channel (PDSCH) from multiple points in the CoMP set [8]. However in this paper, we mainly focus on the downlink transmission in the perfect channel state.Figure 1
CoMP cells model.


## 2.2. Large Scale Propagation and Pathloss Models

Pathloss in wireless communication is defined as the difference in dB between the transmitted and the received signal powers due to the attenuation during the propagation [9]. The traditional log-normal shadowing for large scale pathloss modeling is formulated as
(1)
P
RX
(
dBm
)
=
P
TX
(
dBm
)
-
P
L
(
dBm
)
,
where P
RX represents the received signal power at the user equipment (UE), P
TX represents the transmitted signal power at the severing base station, and the pathloss is denoted by P
L formulated as
(2)
P
L
(
dBm
)
=
P
L
(
d
o
)
+
10
n
lg
(
d
d
o
)
+
x
σ
,
where P
L
(
d
o
) denotes the pathloss at the reference distance, d represents the propagation distance, n is the path loss exponent, and x
σ is the Gaussian random variable with zero mean and standard deviation σ modeling the shadowing effect of the media [8].Urban Macro (UMa) pathloss model is used according to the below equations and parameters specified in Table1 with respect to ITU for line of sight (LoS) and none line of sight (NLoS) scenarios. LoS probability is model as a Bernoulli random variable [10]:
(3)
P
L
LoS
(
dB
)
=
22
lg
d
+
28
+
20
lg
f
c
,
5000
m
<
d
<
d
BP
;
P
L
LoS
(
dB
)
=
40
lg
d
+
7.8
+
2
lg
f
c
-
18
lg
h
BS
-
lg
h
UT
,
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
d
BP
<
d
<
5000
m
;
P
L
NLoS
=
161.04
-
7.11
lg
h
B
-
[
24.37
-
3.7
×
(
h
h
BS
)
2
]
×
lg
h
BS
+
(
433.42
-
3.1
lg
h
BS
)
×
(
lg
d
-
3
)
+
20
lg
f
c
-
[
3.2
lg
(
11.75
h
UT
)
2
-
4.97
]
,
P
rob
(
LoS
)
=
min
⁡
⁡
(
18
d
,
1
)
×
(
1
-
e
-
d
/
63
)
+
e
-
d
/
63
.Table 1
Parameters for UMA pathloss model.
Definition
Parameter
Value
Carrier frequency
f
c
2110 MHz
Base station antenna height
h
BS
24 m
User terminal antenna height
h
UT
0.5 m
Average building height
h
B
20 m
Average street width
L
20 m
LoS shadowing
σ
LoS
4 dB
NLoS shadowing
σ
NLoS
6 dB
Break point distance
d
BP
337.6 m
Transmission power
P
TX
20 W


## 2.3. Downlink Performance

When making CoMP scheme decision, energy efficiency is not the only index that we take into consideration. We should also pay attention to the downlink performance to guarantee the quality of communications.The initial motivation for CoMP was to increase the cell edge user throughput and spectral efficiency by making use of intercell orthogonal resource assignments [11]. And the received SINR is calculated as
(4)
SINR
=
P
RX
(
n
=
j
)
∑
n
≠
j
P
RX
(
n
)
+
P
noise
,
where P
RX
(
n
=
j
) is the received signal power from the serving base station using the UMa large scale pathloss model while the remaining P
RX act as the interferers and the receiver noise power P
noise formulated as
(5)
P
noise
(
dBm
)
=
N
0
+
10
lg
W
(
i
)
,
where N
0
=
-
174 dB/Hz is the noise spectral density and W
(
i
) is the frequency bandwidth assigned to the user i.However in the CoMP cellular system, members of the CoMP scheme perform joint scheduling on PDSCH to transfer the user plane data using TM-9. So the received downlink SINR in joint transmission systems is formulized according to [8] as
(6)
SIN
R
CoMP
=
∑
n
∈
N
JT
P
RX
(
n
)
∑
n
∈
N
∖
N
JT
P
RX
(
n
)
+
P
noise
,
where the N
JT represents the CoMP set while the N
∖
N
JT represents the received power outside the CoMP set. In this way, both the SINR and the performance of the cell edge users’ communications are increased.Downlink capacity received by each useri is formulated as
(7)
C
(
i
)
=
W
(
i
)
log
⁡
2
[
1
+
SIN
R
CoMP
(
i
)
]
.Depending on the user location and mobility, each useri has a distinct CoMP transmission set N
JT
(
i
). Since the downlink capacity is also an important index in CoMP communication, it is also necessary to coordinate all the N
JT
(
i
) to find the best CoMP scheme.


## 2.4. Bits/Joule Energy Efficiency

One common method to measure energy efficiency is to use bits per Joule. For a CoMP schemeN
JT, energy efficiency can be defined as the ratio of transmission capacity to transmission power as follows:
(8)
C
(
N
JT
)
=
∑
C
N
JT
(
i
)
,
E
(
N
JT
)
=
C
(
N
JT
)
P
CoMP
,
where C
(
N
JT
) is the total downlink capacity and P
CoMP is the total transmission power in CoMP scheme N
JT, which is expressed as
(9)
P
CoMP
=
∑
n
∈
N
JT
P
(
n
)
,
where P
(
n
) is the total power consumption for base station n in CoMP scheme N
JT, which is calculated using the assumptions from [12] and [13] as
(10)
P
(
n
)
=
N
S
×
N
PA
/
sector
×
(
p
TX
p
Aeff
+
P
SP
)
×
(
1
+
C
C
)
(
1
+
C
PSBB
)
+
P
BH
,
where P
(
n
) is the radiated power per base station, P
SP is the signal processing power per base station, and P
BH is the power due to backhauling. p
Aeff is the power amplifier efficiency and C
C and C
PSBB are the cooling and battery backup losses in the system.The signal processing powerP
SP per sector as a function of different cooperation sizes scales as
(11)
P
SP
=
58
×
(
0.87
+
0.1
N
C
+
0.03
N
C
2
)
,
where N
C is the number of base stations participating in the CoMP scheme N
JT.Backhauling power consumptionP
BH for base stations using CoMP is modeled in [12] as
(12)
P
BH
=
C
BH
100
Mbits
/
sec
×
50
W
,
where C
BH is a given average backhaul requirement per base station. The expression of C
BH is expressed as
(13)
C
BH
=
N
C
×
(
2
N
C
)
×
p
×
q
T
S
,
where p is the additional pilot density, q is the CSI signaling under CoMP network, and T
s
=
66.7 μs is the sample period which is the reciprocal of the assumed OFDM subcarrier spacing at 15 kHz [8].As the models above, we try to get the best CoMP schemeN
JT which both ensures the quality of the communication and maximizes the energy efficiency.


## 3. Reinforcement Learning

In this section, we present the application of reinforcement learning (RL) algorithm to get the best CoMP schemeN
JT. Firstly, we start with the introduction of the RL Algorithm and its branch Q-learning. Then we build an appropriate value function applying RL to get the optimal CoMP scheme N
JT.


### 3.1. RL Algorithm

Generally speaking, the RL Algorithm is to try to find an optimal action policy to solve a given task in an unknown environment. During the RL, a learner observes the state of the environment and according to which it chooses its action. Firstly, a policy is taken arbitrarily, so it is a very typical learning process that the learner makes wrong actions. Then the learner receives a reward feedback signal from the environment and based on the reward it improves its policy.The learner chooses its actionA depending on the environment state S and the reward R, and the policy U is adjusted along with action A as it is shown in Figure 2.Figure 2
Basic algorithm of the RL.During the RL a learner looks for an optimal policyU
* for which it will always receive the best rewards from an environment. For such a policy, the value function V
U
*
(
s
) is always largest or equal to the value function V
U
(
s
) to any policy U.At present, RL Algorithm mainly includes two categories: the value function estimation and the strategy space search method. Concretely, there are Monte Carlo algorithm, Temporal Difference (TD) algorithm, Dynamic Programming algorithm,Q-learning algorithm, and Sarsa learning algorithm in RL Algorithm.


### 3.2.Q-Learning Algorithm

Q-learning algorithm is formed by Watkins and Dayan in 1989 [14], which is a milestone in the development of reinforcement learning. Currently, Q-learning algorithm is also one of the most widely applied reinforcement learning algorithms.Compared to other RL Algorithms,Q-learning algorithm gets value for eachState-Action, and the value function is expressed as Q
(
s
t
,
a
t
). Value of Q
(
s
t
,
a
t
) is the accumulation rewards on the basis of the recycle implementation that action a
t is chosen according to related policy in state s
t. The optimal policy U
* of Q-learning is to maximize Q
(
s
t
,
a
t
); thus the optimal policy U
* can be expressed as
(14)
U
*
(
s
t
)
=
arg
max
⁡
a
t
Q
(
s
t
,
a
t
)
,As a result, the learner only needs to consider the current state and available actions and then choose the action which maximizes theQ
(
s
t
,
a
t
) according to the policy. When choosing the action, we only have to select the maximum value from the Q list, which greatly simplifies the process of decision-making. Values in Q list are the results of the iterative learning step by step. Learner needs constant interaction with the environment to enrich Q list, so that it can cover all possible situations. The Q-learning iterative calculation formula of value function is
(15)
Q
(
s
t
,
a
t
)
=
Q
(
s
t
,
a
t
)
+
α
[
+
γ
max
⁡
a
t
+
1
[
Q
(
s
t
+
1
,
a
t
+
1
)
-
Q
(
s
t
,
a
t
)
]
r
(
s
t
,
a
t
)
+
γ
max
⁡
a
t
+
1
[
Q
(
s
t
+
1
,
a
t
+
1
)
-
Q
(
s
t
,
a
t
)
]
]
,
where α is the learning coefficient and γ is the discount factor.Steps ofQ-learning algorithm are as follows.(1)
Initialize value function,α, γ factors, and Q
(
s
t
,
a
t
)
.
(2)
Observe the current states
t.
(3)
According to the current state, select the action along with the policy, and observe the next state.
(4)
According to the value function of newState-Action estimate Q
(
s
t
,
a
t
).
(5)
Check if the learning ends; if not, sett
=
t
+
1 and go back to (1).


### 3.3. RL Algorithm In CoMP Communication

In CoMP communication, our purpose is to find the best CoMP schemeN
JT, which is called optimal policy U
* in RL Algorithm. As the models built in Section 2, there are 6 base stations participating in the cooperation. The base stations are numbered from 1 to 6 as Figure 1 shows. Then the cooperation policy U can be a 1 × 6 matrix. For example, if the optimal policy U
* is the cooperation of base stations 1, 3, and 6, then the policy U
* is N
JT
=
[
1,0
,
1,0
,
0,1
].The environment state is the number and the locations of the users. The value functionQ
(
s
t
,
a
t
) is the energy efficiency. The rewards are related to the downlink capacity and received SINR. We set the downlink capacity standard as 1 Mbits/s and the received SINR standard as 21 dB. The rates of the users who reach the standards are expressed as P
cap and P
SINR. After different trials, Table 2 shows the relations between the P
cap, P
SINR, and the rewards.Table 2
Rewards for downlink capacity and SINR.
P
cap
r
1 (capacity)
P
SINR
r
2 (SINR)
0.90~1.00
3.75
0.90~1.00
3.75
0.75~0.90
0
0.75~0.90
0
0.60~0.75
−5.00
0.60~0.75
−5.00
0~0.60
−7.50
0~0.60
−7.50Because the policyU is a 1
×
6 matrix, the actions interacting with the policy are complex as follows:
(16)
U
(
t
+
1
)
=
U
(
t
)
∣
action
(
i
)
,
0
<
i
≤
6
,
U
(
t
+
1
)
=
U
(
t
)
&
action
(
i
)
,
6
<
i
≤
12
,
[
action
(
1
)
action
(
2
)
action
(
3
)
action
(
4
)
action
(
5
)
action
(
6
)
]
=
[
1,0
,
0,0
,
0,0
0,1
,
0,0
,
0,0
0,0
,
1,0
,
0,0
0,0
,
0,1
,
0,0
0,0
,
0,0
,
1,0
0,0
,
0,0
,
0,1
]
,
[
action
(
7
)
action
(
8
)
action
(
9
)
action
(
10
)
action
(
11
)
action
(
12
)
]
=
[
0,1
,
1,1
,
1,1
1,0
,
1,1
,
1,1
1,1
,
0
,
1,1
,
1
1,1
,
1,0
,
1,1
1,1
,
1,1
,
0,1
1,1
,
1,1
,
1,0
]
.When choosing the actions 1~6, it means adding a base station in CoMP, while when choosing the actions 7~12, it means removing a base station from CoMP. But not all the actions are available at each step.ThenQ-learning iterative calculation formula for CoMP is
(17)
Q
(
s
t
,
a
t
)
=
Q
(
s
t
,
a
t
)
+
α
[
+
γ
max
⁡
a
t
+
1
[
Q
(
s
t
+
1
,
a
t
+
1
)
-
Q
(
s
t
,
a
t
)
]
r
1
(
s
t
,
a
t
)
+
r
2
(
s
t
,
a
t
)
+
γ
max
⁡
a
t
+
1
[
Q
(
s
t
+
1
,
a
t
+
1
)
-
Q
(
s
t
,
a
t
)
]
]
.In practice, the number and the locations change much more slowly than the process of finding the optimal policyU
*, so for a period the optimal policy is the same. In order to prevent Q from falling into a local optimal point, we introduce variations to U every six steps. After sixty steps, we remove the variations; then the system operates in the optimal policy U
* with the maximal Q while the state changes very faintly. When the state changes, the Q-learning repeats the process above automatically. In this way, it leads to a green cellular network in both energy efficiency and downlink performance way.


## 3.1. RL Algorithm

Generally speaking, the RL Algorithm is to try to find an optimal action policy to solve a given task in an unknown environment. During the RL, a learner observes the state of the environment and according to which it chooses its action. Firstly, a policy is taken arbitrarily, so it is a very typical learning process that the learner makes wrong actions. Then the learner receives a reward feedback signal from the environment and based on the reward it improves its policy.The learner chooses its actionA depending on the environment state S and the reward R, and the policy U is adjusted along with action A as it is shown in Figure 2.Figure 2
Basic algorithm of the RL.During the RL a learner looks for an optimal policyU
* for which it will always receive the best rewards from an environment. For such a policy, the value function V
U
*
(
s
) is always largest or equal to the value function V
U
(
s
) to any policy U.At present, RL Algorithm mainly includes two categories: the value function estimation and the strategy space search method. Concretely, there are Monte Carlo algorithm, Temporal Difference (TD) algorithm, Dynamic Programming algorithm,Q-learning algorithm, and Sarsa learning algorithm in RL Algorithm.


## 3.2.Q-Learning Algorithm

Q-learning algorithm is formed by Watkins and Dayan in 1989 [14], which is a milestone in the development of reinforcement learning. Currently, Q-learning algorithm is also one of the most widely applied reinforcement learning algorithms.Compared to other RL Algorithms,Q-learning algorithm gets value for eachState-Action, and the value function is expressed as Q
(
s
t
,
a
t
). Value of Q
(
s
t
,
a
t
) is the accumulation rewards on the basis of the recycle implementation that action a
t is chosen according to related policy in state s
t. The optimal policy U
* of Q-learning is to maximize Q
(
s
t
,
a
t
); thus the optimal policy U
* can be expressed as
(14)
U
*
(
s
t
)
=
arg
max
⁡
a
t
Q
(
s
t
,
a
t
)
,As a result, the learner only needs to consider the current state and available actions and then choose the action which maximizes theQ
(
s
t
,
a
t
) according to the policy. When choosing the action, we only have to select the maximum value from the Q list, which greatly simplifies the process of decision-making. Values in Q list are the results of the iterative learning step by step. Learner needs constant interaction with the environment to enrich Q list, so that it can cover all possible situations. The Q-learning iterative calculation formula of value function is
(15)
Q
(
s
t
,
a
t
)
=
Q
(
s
t
,
a
t
)
+
α
[
+
γ
max
⁡
a
t
+
1
[
Q
(
s
t
+
1
,
a
t
+
1
)
-
Q
(
s
t
,
a
t
)
]
r
(
s
t
,
a
t
)
+
γ
max
⁡
a
t
+
1
[
Q
(
s
t
+
1
,
a
t
+
1
)
-
Q
(
s
t
,
a
t
)
]
]
,
where α is the learning coefficient and γ is the discount factor.Steps ofQ-learning algorithm are as follows.(1)
Initialize value function,α, γ factors, and Q
(
s
t
,
a
t
)
.
(2)
Observe the current states
t.
(3)
According to the current state, select the action along with the policy, and observe the next state.
(4)
According to the value function of newState-Action estimate Q
(
s
t
,
a
t
).
(5)
Check if the learning ends; if not, sett
=
t
+
1 and go back to (1).


## 3.3. RL Algorithm In CoMP Communication

In CoMP communication, our purpose is to find the best CoMP schemeN
JT, which is called optimal policy U
* in RL Algorithm. As the models built in Section 2, there are 6 base stations participating in the cooperation. The base stations are numbered from 1 to 6 as Figure 1 shows. Then the cooperation policy U can be a 1 × 6 matrix. For example, if the optimal policy U
* is the cooperation of base stations 1, 3, and 6, then the policy U
* is N
JT
=
[
1,0
,
1,0
,
0,1
].The environment state is the number and the locations of the users. The value functionQ
(
s
t
,
a
t
) is the energy efficiency. The rewards are related to the downlink capacity and received SINR. We set the downlink capacity standard as 1 Mbits/s and the received SINR standard as 21 dB. The rates of the users who reach the standards are expressed as P
cap and P
SINR. After different trials, Table 2 shows the relations between the P
cap, P
SINR, and the rewards.Table 2
Rewards for downlink capacity and SINR.
P
cap
r
1 (capacity)
P
SINR
r
2 (SINR)
0.90~1.00
3.75
0.90~1.00
3.75
0.75~0.90
0
0.75~0.90
0
0.60~0.75
−5.00
0.60~0.75
−5.00
0~0.60
−7.50
0~0.60
−7.50Because the policyU is a 1
×
6 matrix, the actions interacting with the policy are complex as follows:
(16)
U
(
t
+
1
)
=
U
(
t
)
∣
action
(
i
)
,
0
<
i
≤
6
,
U
(
t
+
1
)
=
U
(
t
)
&
action
(
i
)
,
6
<
i
≤
12
,
[
action
(
1
)
action
(
2
)
action
(
3
)
action
(
4
)
action
(
5
)
action
(
6
)
]
=
[
1,0
,
0,0
,
0,0
0,1
,
0,0
,
0,0
0,0
,
1,0
,
0,0
0,0
,
0,1
,
0,0
0,0
,
0,0
,
1,0
0,0
,
0,0
,
0,1
]
,
[
action
(
7
)
action
(
8
)
action
(
9
)
action
(
10
)
action
(
11
)
action
(
12
)
]
=
[
0,1
,
1,1
,
1,1
1,0
,
1,1
,
1,1
1,1
,
0
,
1,1
,
1
1,1
,
1,0
,
1,1
1,1
,
1,1
,
0,1
1,1
,
1,1
,
1,0
]
.When choosing the actions 1~6, it means adding a base station in CoMP, while when choosing the actions 7~12, it means removing a base station from CoMP. But not all the actions are available at each step.ThenQ-learning iterative calculation formula for CoMP is
(17)
Q
(
s
t
,
a
t
)
=
Q
(
s
t
,
a
t
)
+
α
[
+
γ
max
⁡
a
t
+
1
[
Q
(
s
t
+
1
,
a
t
+
1
)
-
Q
(
s
t
,
a
t
)
]
r
1
(
s
t
,
a
t
)
+
r
2
(
s
t
,
a
t
)
+
γ
max
⁡
a
t
+
1
[
Q
(
s
t
+
1
,
a
t
+
1
)
-
Q
(
s
t
,
a
t
)
]
]
.In practice, the number and the locations change much more slowly than the process of finding the optimal policyU
*, so for a period the optimal policy is the same. In order to prevent Q from falling into a local optimal point, we introduce variations to U every six steps. After sixty steps, we remove the variations; then the system operates in the optimal policy U
* with the maximal Q while the state changes very faintly. When the state changes, the Q-learning repeats the process above automatically. In this way, it leads to a green cellular network in both energy efficiency and downlink performance way.


## 4. Simulation and Discussions

In this section, computer simulations are carried out to evaluate the performance of the proposed CoMP schemeN
JT in the RL Algorithm. The parameters in the simulations are shown in Table 1. We assume that the channel state information is perfect in this paper. The CoMP scheme N
JT changes along with the communication standards. In different standard, the number of the base stations participating in the CoMP communication is different.Figure3(a) shows the value of Q changes with the SINR standard. When the SINR standard is low, most of the users in the center cell can reach it, so the SINR reward of each set is positive and the energy efficiency mainly affect the value of Q. As the SINR standard increases, less users in the center cell can reach the standard, and the reward turns negative, so the Q value descends. Figure 3(b) shows the best CoMP set in different SINR standards. As the SINR standard increases, the best CoMP set increases to meet the standard. While it increases to 30 dB, even CoMP set 6 cannot meet the standard, so it turns down to CoMP set 1 by the impact of energy efficiency.(a)Q value changes with SINR standard. (b) Best CoMP set in different Capacity standard.
(a)
(b)Figure4(a) shows the value of Q changes with the Capacity standard. When the Capacity standard is low, most of the users in the center cell can reach it, so the Capacity reward of each set is positive and the energy efficiency mainly affect the value of Q. As the Capacity standard increases, less users in the center cell can reach the standard, and the reward turns negative, so the Q value descends. Figure 4(b) shows the best CoMP set in different Capacity standards. As the Capacity standard increases, the best CoMP set increases to meet the standard. While it increases to 11 Mbits/s, even CoMP set 6 cannot meet the standard, so it turns down to CoMP set 2 by the impact of energy efficiency.(a)Q value changes with Capacity standard. (b) Best CoMP set in different Capacity standard.
(a)
(b)Figures5–9 are simulations under the standard in which Capacity standard is 1 Mbit/s and SINR Standard is 21 dB. Figure 5 shows that energy efficiency descends as the CoMP set increases. Figure 6 shows that both Downlink Capacity and SINR increase as the CoMP set increases. Figure 7 shows that, in this situation, CoMP set 4 usually gets the largest Q value.Figure 5
Energy Efficiency changes in different User Count.(a) Capacity change in different User Count. (b) SINR change in different User Count.
(a)
(b)Figure 7
Q value changes in different User Count.(a)Q-learning Result without variations. (b) Q-learning Process without variations.
(a)
(b)(a)Q-learning Result with variations. (b) Q-learning Process with variations.
(a)
(b)In Figures8(a) and 9(a), the red circles represent the base stations in the CoMP result from the Q-learning, while the yellow ones represent the best base station grouping by the enumeration search. Figure 8 shows the result and process of Q-learning without variations. It traps into the local optimum easily, which leads to the wrong base station grouping checked by enumeration search. To come over it, variation is introduced to the Q-learning every six steps as Figure 9 shows. In this way, the local optimum is eliminated and the best base station grouping can be got according to the Q-learning method.Enumeration search is the auxiliary examination method to check theQ-learning result. Compared to the enumeration search, Q-learning method can optimize the base station grouping automatically in a short time as Figure 10 shows.Figure 10
Comparison between Enumeration search andQ-learning.Figure11 shows the process of Q-learning with a variation every six steps. For each state, we set sixty steps to adjust the strategy. So within the 60 steps, there is a variation added every six steps, which makes the Q value oscillate. After sixty steps, the variations are removed. Figure 12 shows that we find the maximal Q in the first sixty steps and act the optimal policy U
* with the Q. As Figures 11 and 12 show, the state changes little. At the 121 and 272 steps, s
t
+
1
≠
s
t, so the Q-learning goes back to the variation to find the new optimal policy U
*. And the relevant energy efficiency and the rewards are shown in Figures 8 and 12. The relevant states and the optimal policy U
* are shown in Table 3.Table 3
Optimal policyU
* in different states.
State S1
s
t
+
1
=
s
t(steps  1~120)
State S2
s
t
+
1
=
s
t(steps  122~271)
State S3
s
t
+
1
=
s
t(steps  273~394)
U
1
*
N
JT
=
[
0,1
,
1,1
,
0,1
]
U
2
*
N
JT
=
[
1,0
,
1,0
,
1,0
]
U
3
*
N
JT
=
[
1,0
,
1,1
,
0,1
]Figure 11
Q-learning Process and the variations.Figure 12
Q-learning Process and optimal policy U.


## 5. Conclusion

In wireless cellular network, it is very important to increase the energy efficiency of radio access networks to meet the challenges. It is also important to increase the downlink performance of the wireless network to meet the requirement of the users. In this paper, we proposed the RL Algorithm to derive the optimal base station grouping decisions for efficient transmissions in CoMP. In addition, we introduce variations preventingQ from trapping into local maximum. Simulation results have been presented to show the process of the Q-learning and the optimal policy U
* it finds.More research is in progress to make the communication model more realistic by the consideration of imperfect channel state information. In addition, two or more base stations in the small scale model unit can be dynamically switched off to further “green” the wireless communications.



---

*Source: 698797-2014-07-15.xml*