# Decentralized and Dynamic Band Selection in Uplink Enhanced Licensed-Assisted Access: Deep Reinforcement Learning Approach

Authors: Fitsum Debebe Tilahun; Chung G. Kang
Journal: Wireless Communications and Mobile Computing
 (2020)

---

## Abstract

Enhanced licensed-assisted access (eLAA) is an operational mode that allows the use of unlicensed band to support long-term evolution (LTE) service via carrier aggregation technology. The extension of additional bandwidth is beneficial to meet the demands of the growing mobile traffic. In the uplink eLAA, which is prone to unexpected interference from WiFi access points, resource scheduling by the base station, and then performing a listen before talk (LBT) mechanism by the users can seriously affect the resource utilization. In this paper, we present a decentralized deep reinforcement learning (DRL)-based approach in which each user independently learns dynamic band selection strategy that maximizes its own rate. Through extensive simulations, we show that the proposed DRL-based band selection scheme improves resource utilization while supporting certain minimum quality of service (QoS).


---

## Body

## 1. Introduction

The rapid mobile traffic demand has resulted in the scarcity of the available radio spectrum. To meet this ever-increasing demand, extending systems like long-term evolution (LTE) to unlicensed spectrum is one of the promising approaches to boost users’ quality of service by providing higher data rates [1]. In this regard, initiatives such as the licensed-assisted access (LAA) [2], LTE-unlicensed (LTE-U) [3], and MulteFire (MF) systems [4] can be mentioned. The focus of this article is, however, on the LAA system, which 3GPP has initially introduced and standardized in Rel.-13 for downlink operations only [2]. By using the carrier aggregation (CA) technology, carriers on licensed band are primarily used to carry control signals and critical data, while the additional secondary carriers from unlicensed band are used to opportunistically boost the data rates of the users [5]. To obey regional spectrum regulations such as restrictions on the maximum transmitting power and channel occupancy time [6] while fairly coexisting with the existing systems such as WiFi, it is mandatory for an LAA base station (BS) to perform listen before talk mechanism before transmitting over unlicensed band [7–9]. The enhanced version of LAA, named as enhanced licensed-assisted access (eLAA), that supports both uplink and downlink operations was later approved in Rel.14 [10]. The uplink eLAA mode over unlicensed band is designed to meet the channel access mechanisms of the two bands, meaning the BS performs LBT and allocates uplink resources for the scheduled users, and then the scheduled users perform the second round of LBT to check whether the channel is clear or not before uplink transmission [11]. The degradation of uplink channel access due to two rounds of LBT mechanism is investigated in [12–14]. If a scheduled user senses an active WiFi access point (AP) which is hidden to the BS, then the channel cannot be accessed, wasting the reserved uplink resource. Scheduling based approach in uplink eLAA, while there are unexpected interference sources, can significantly affect the utilization of uplink resources.To improve the utilization of unlicensed band resources, several approaches have been suggested. In [15–17], multi-subframe scheduling (MSS), a simple modification of the conventional scheduling, is proposed. MSS enables a single uplink grant to indicate multiple resource allocation across multiple subframes. Providing diverse transmission opportunities may enhance the resource utilization; however, the resources can still be wasted if the user fails to access the channels. In [14, 18], schemes that switch between random access and scheduling are proposed, but their focus is limited to unlicensed spectrum. Joint licensed and unlicensed band resource allocation that takes a hidden node into account is proposed in [19] for the downlink eLAA system. Furthermore in [20], a scheme that does not require uplink grant along with the required enhancement to the existing LTE system is proposed.In this paper, we attempt a new learning approach in which each user makes dynamic band selection (licensed or unlicensed) independently for uplink transmission, without waiting for scheduling from BS. To this end, we implemented each user as a DRL agent that learns the optimal band selection strategy relying only on its own local observation, i.e., without any prior knowledge of WiFi APs’ activities and time-varying channel conditions. Through continuous interactions with the environment, the potential users to be affected by hidden nodes learn the activities of WiFi APs and make use of it in the band selection process. The learned policy not only guarantees channel access but also ensures a transmission rate above a certain threshold, despite the presence of unpredictable hidden nodes. Such a learning approach would be a useful means of handling the underlying resource utilization problems in uplink eLAA.The rest of the paper is organized as follows. Section2 describes the system model considered in the paper. Section 3 gives a brief overview on deep reinforcement learning (DRL), followed by DRL formulation of the band selection problem. The proposed deep neural network architecture and training algorithm are also discussed. Simulation results are presented in Section 4, and finally conclusion is drawn in Section 5.


## 2. System Model

We consider a single cell uplink eLAA system that consists of an eLAA base station (BS) andN user equipment (UE) that can also operate in unlicensed band through carrier aggregation technology. Let N=1,2,⋯,N denote a set of user indices which are uniformly distributed within the cell and ℳ=1,2,⋯,M designate a set of unlicensed band interference sources such as WiFi access points (APs) which are located outside the coverage area of the cell within a certain distance. The system model is shown in Figure 1.Figure 1
Uplink eLAA system model.In order to get uplink access, each UEn∈N makes a scheduling request to the eLAA BS, who is responsible for allocating resources. Before granting uplink resources, the eLAA BS is required to undergo a carrier-sensing procedure within its coverage limit. Once the channel is clear, it reserves resources for uplink transmission. Then, the scheduled user performs another round of listen before talk procedure before transmission. If the user detects transmission from hidden nodes, nearby WiFi APs that are outside the carrier-sensing range of the eLAA BS, then the reserved uplink resources over unlicensed band cannot be accessed.We assume the channel between the BS and then-th UE, denoted as hnt, evolves according to the Gaussian Markov block fading autoregressive model [21] as follows:
(1)hnt=ρnhnt−1+1−ρn2et,where ρn is the normalized channel correlation coefficient between slot t and t−1. From Jake’s fading spectrum, ρn=Jo2πfd,nτo where fd,n, τo, and Jo⋅ are the Doppler frequency, slot duration, and the zeroth-order Bessel function of the first kind, respectively. The error et is a circularly symmetric complex Gaussian variable, i.e., et∼CN0,ϒd/doα, where ϒ is the path loss corresponding to the reference at a distance do and α is the path loss exponent. The channel is initialized as hn0∼CN0,ϒdn/doα, where dn is distance of the n-th user from the BS.LetWU and WL be the total bandwidth in unlicensed and licensed bands, respectively. At time slot t, let the number of users associated with unlicensed and licensed band be NUt and NLt, respectively. If all UEs on licensed band are uniformly allocated to orthogonal uplink resources, then the bandwidth of the UEs is constrained as
(2)BLt=WLNLt.Similarly, expecting that the total unlicensed bandwidth is equally shared among UEs in a virtual sense, then the bandwidth of UEs on unlicensed band can be constrained as(3)BUt=WUNUt.DenotingP and N0 as uplink transmit power and the noise spectral density, we may compute the signal-to-noise ratio (SNR) of the received signal at the BS for unlicensed band user n (assuming it occupies channel) as
(4)SNRn,Ut=Phnt2BUt⋅N0.Likewise, the SNR for licensed band usern is given as
(5)SNRn,Lt=Phnt2BLt⋅N0.The dynamics of each WiFi APs activity are modeled as a discrete-time two-state Markov chain as shown in Figure2. Each AP can be either in active (state=0) or inactive (state=1) state. The transition probability from state j to k is denoted as
(6)Pj,k=Prst+1=k∣st=j,∀j,k∈0,1.Figure 2
Activity model of WiFi AP as a two-state Markov chain.Note that the users do not have the knowledge of the underlying dynamics of WiFi APs’ activities, i.e., transition probabilities.Letτ represent the transmission probability of an active WiFi AP. In slot t, let Nn,contt be the number of contending active APs within the sensing range of n-th UE. Assuming that all activities of WiFi AP’s are independent, the probability of UE n having at least one hidden node is
(7)Pn,hidt=1−1−τNn,contt.In order to calculate the uplink rate (throughput) of the users, we refer to the lookup table, given in Table1, which maps the received SNR to spectral efficiency (SE) [22]. Then, the uplink rate of UE n using unlicensed band is given as
(8)Rn,Ut=BUtSEt1−Pn,hidt.Table 1
Lookup table for SNR-to-spectral efficiency mapping.
Index
Minimum SNR (dB)
Spectral efficiency (bps/Hz)
1
−6.7
0.1523
2
−4.7
0.2344
3
−2.3
0.3770
4
0.2
0.6016
5
2.4
0.8770
6
4.3
1.1758
7
5.9
1.4766
8
8.1
1.9141
9
10.3
2.4063
10
11.7
2.7305
11
14.1
3.3223
12
16.3
3.9023
13
18.7
4.5234
14
21.0
5.1152
15
22.7
5.5547Similarly, the uplink rate of UEn using licensed band is given as
(9)Rn,Lt=BLtSEt.In each time slott, the goal of each UE is to select the band that maximizes the uplink rate. Note that if a certain band, e.g., licensed band, is overloaded by a large number of UEs, the individual rate of the users in the band will be significantly reduced. This will constraint each UE to take advantage of the unlicensed band whenever the APs are inactive. Hence, learning the WiFi APs’ activities and channel conditions is critical to effectively use the uplink resources while boosting individual data rate.


## 3. DRL-Based Decentralized Dynamic Band Selection

### 3.1. Deep Reinforcement Learning (DRL): Overview

In reinforcement learning (RL), an agent learns how to behave by sequentially interacting with the environment. As shown in Figure3, at each time t, the agent observes the state st∈S, where S is the state space, and executes action at∈A from the action space A. The interaction with the environment produces the next state st+1 and scalar reward rt+1.Figure 3
Reinforcement learning framework.The goal of the agent is to learn an optimal policy that maximizes the discounted long-term cumulative reward, expressed as(10)Rt=∑t−1Tγt−1rt+1,where γ∈0,1is the discounting factor and T is the total number of time steps (horizon) [23].One of the most widely used model-free RL methods is Q-learning in which the agent learns policy by iteratively evaluating the state-action value functionQs,a, defined as the expected return starting from the state s, taking the action a, and then, following the policy π. In order to derive the optimal policy, at a given state s, the action that maximizes the state-action value function should be selected, i.e.,
(11)a∗s=argmaxaQs,aand then similarly follow optimal actions in the successor states.In Q-learning, a lookup table is constructed that stores the action valueQs,a for every state-action pair (s, a). The entries of the table are updated by iteratively evaluating the Bellman optimality equation as:
(12)Qst,at⟵Qst,at+βrt+1+γmaxat+1Qst+1,at+1−Qst,atwhere β∈0,1 is the learning rate. However, the look up table approach in Q-learning is not scalable for problems with the large state and action spaces. DRL approximates the value functions with deep neural network (DNN) instead. In deep Q-network (DQN), the action-value function Qs,a;θ is estimated by DNN, parametrized by θ, which takes the state as input. Then, action is selected according to the following ε-greedy policy:
(13)at=randomaction,withaprobabilityofε,argmaxaQs,a;θ,withaprobabilityof1−ε.To stabilize the learning process, it is common to use a replay bufferD that stores transitions e=st,at,rt+1,st+1 and mini batch of samples are randomly drawn from the buffer to train the network. Moreover, a separate quasistatic target network, parametrized by θ′, is used to estimate the target value of the next state. The loss function is computed as
(14)ℒθ=Es,a,r,s′∼Dr+γmaxa′Qs′,a′;θ′−Qs,a;θ2θ is updated by following stochastic gradient of the loss as θ⟵θ−β∇θLθ, while the target parameter θ′ is updated according to θ′⟵θ every C steps [24]. The details of DQN algorithm is summarized in Algorithm 1.Algorithm 1. DQN algorithm.
Initialize replay bufferD
Initialize action value functionQ with parameter θ
Initialize target action value functionQ^ with parameter θ′=θ
Input the initial state to the DQN
for t=1,2,..⋯ do
Execute actionat from Q using ε-greedy policy
Observert+1 and st+1 from the environment.
Store the transitionst,at,rt+1,st+1 into the replay buffer D
Sample random minibatch of transitions fromD
Evaluate the targetyj=rj+γmaxa′Q^sj+1,a′;θ′
Perform a gradient descent step onyj−Qsj,aj;θ2 with respect to θ
EveryC steps, update the target network Q^ according to θ′⟵θ
end for


### 3.2. DRL Formulation for Dynamic Band Selection

Each user is implemented as DRL, specifically by deep Q-network (DQN) agent that relies on the output of their deep neural network to make dynamic band selection decisions between licensed and unlicensed bands. The DRL formulation is presented below.(i)
ActionIn each time slott, the n-th agent samples an action ant from the action set
(15)A=Licensed,Unlicensed.
(ii)
StateAfter executing the actionant, the agent receives binary observation and reward from the environment. The observation is either ont=1 if the uplink rate in the selected band exceeds the minimum threshold rate or ont=0 otherwise. The state of the agent is defined as history of an action-observation pairs with length H:
(16)snt≜ani,onii=t−Ht
(iii)
RewardDepending on the selected action, the agent receives the following scalar reward:(17)Ifant=Unlicensed,rt+1n=Rn,Ut,ifRn,Ut≥RU,min0,otherwiseLicensed,rt+1n=Rn,Lt,ifRn,Lt≥RL,min0,otherwisewhere Rn,Ut and Rn,Lt are given according to Equations (8) and (9), while RU,min and RL,min are the uplink minimum threshold rates on unlicensed and licensed band, respectively.


### 3.3. Deep Neural Network Description

For dynamic band selection, each UE trains independent DQN. The structure of the deep neural network is shown in Figure4.Figure 4
Structure of the proposed deep neural network.The deep neural network consists of long short-term memory (LSTM) layer, fully connected layers, and rectified linear unit (ReLu) activation function.Long short-term memory(LSTM) is one class of recurrent neural networks (RNNs) which are designed to learn a specific pattern in a sequence of data by taking time correlation into account. They were initially introduced to overcome the vanishing (exploding) gradient problem of RNNs in the course of back propagation. Regulated by gate functions, the cell (internal memory) state of an LSTM learns how to aggregate inputs separated by time, i.e., which experiences to keep or throw away [25]. In our formulation, note that the states of the agents, which are histories of action-observation pairs, have long-term dependency (correlation) emanating from the dynamics of WiFi APs’ activities that follow a two-state Markov property, and the time-varying channel conditions according to Gaussian Markov block-fading autoregressive model. LSTM is crucial for the learning process since it can capture the actual state by exploiting the underlying correlation in the history of action-observation pairs. Therefore, the state must pass through this preprocessing step before it is directly fed to the neural network.A deep neural network consists of multiple fully connected layers, in which each of the layers abstracts certain feature of the input. Letx be the input to the layer, while W and b are the weight matrix and bias vector, respectively. The output vector of a layer, denoted as y, in a fully connected layer can be described by the following operation:
(18)y=fWx+b,where f is the element-wise excitation (activation) that adds nonlinearity. In our simulations, we input the states to an LSTM layer with hidden units of 64, whose output is fed to two fully connected hidden layers with 128 and 64 neurons. The output layer produces action values Qs,a for both actions. ReLu activation function is used on all the layers to avoid the vanishing gradient problem [26]. The target network also adopts the same neural network structure.


### 3.4. Training Algorithm Description

The DQNs of the agents are individually trained according to Algorithm2. The loss function given by Equation (14) is used to train the DQN. The hyperparameters are summarized in Table 2.Algorithm 2. DQN training algorithm for dynamic band selection.
for each agent n∈N do
Initialize replay bufferDn
Initialize action value functionQn with parameter θn
Initialize target action value functionQ^n with parameter θn′=θn
Generate initial statesn,1 from the environment simulator
end for
for t=1,2,..⋯ do
for each agent n∈N do
Execute actionan,t from Qn using ε-greedy policy
Collect rewardrn,t+1 and observation on,t+1
Observe the next statesn,t+1 from the environment simulator
Store the transitionsn,t,an,t,rn,t+1,sn,t+1 into Dn
Sample random minibatch of transitions fromDn
Evaluate the targetyn,j=rn,j+γmaxan,j+1Q^nsn,j+1,an,j+1;θn′
Perform a gradient descent step onyn,j−Qnsn,j,an,j;θn2with respect to θn
EveryC steps, update the target network Q^n according to θn′⟵θn
end for
end forTable 2
Hyperparameters.
Parameter
Value
Discount factorγ
0.9
Learning rateβ
0.01
Explorationε in ε-greedy policy
0.05 to 0.01
Target network update frequencyC
300
Mini batch size
32
Replay bufferD size
1000Note that the agents do not have a complete knowledge of the environment, such as the action of other agents, the underlying dynamics of the WiFi APs’ activities, and varying channel conditions. Instead, through sequential interaction with the environment, each agent makes decisions on band selection solely based on local feedbacks (reward and observation) from the base station. This significantly reduces the training complexity (cost) at each user. Moreover, since the training can be conducted in an offline manner, the trained weights can be used in deployment phase. Retraining the weights is done infrequently; for example, if the environment significantly changes.


## 3.1. Deep Reinforcement Learning (DRL): Overview

In reinforcement learning (RL), an agent learns how to behave by sequentially interacting with the environment. As shown in Figure3, at each time t, the agent observes the state st∈S, where S is the state space, and executes action at∈A from the action space A. The interaction with the environment produces the next state st+1 and scalar reward rt+1.Figure 3
Reinforcement learning framework.The goal of the agent is to learn an optimal policy that maximizes the discounted long-term cumulative reward, expressed as(10)Rt=∑t−1Tγt−1rt+1,where γ∈0,1is the discounting factor and T is the total number of time steps (horizon) [23].One of the most widely used model-free RL methods is Q-learning in which the agent learns policy by iteratively evaluating the state-action value functionQs,a, defined as the expected return starting from the state s, taking the action a, and then, following the policy π. In order to derive the optimal policy, at a given state s, the action that maximizes the state-action value function should be selected, i.e.,
(11)a∗s=argmaxaQs,aand then similarly follow optimal actions in the successor states.In Q-learning, a lookup table is constructed that stores the action valueQs,a for every state-action pair (s, a). The entries of the table are updated by iteratively evaluating the Bellman optimality equation as:
(12)Qst,at⟵Qst,at+βrt+1+γmaxat+1Qst+1,at+1−Qst,atwhere β∈0,1 is the learning rate. However, the look up table approach in Q-learning is not scalable for problems with the large state and action spaces. DRL approximates the value functions with deep neural network (DNN) instead. In deep Q-network (DQN), the action-value function Qs,a;θ is estimated by DNN, parametrized by θ, which takes the state as input. Then, action is selected according to the following ε-greedy policy:
(13)at=randomaction,withaprobabilityofε,argmaxaQs,a;θ,withaprobabilityof1−ε.To stabilize the learning process, it is common to use a replay bufferD that stores transitions e=st,at,rt+1,st+1 and mini batch of samples are randomly drawn from the buffer to train the network. Moreover, a separate quasistatic target network, parametrized by θ′, is used to estimate the target value of the next state. The loss function is computed as
(14)ℒθ=Es,a,r,s′∼Dr+γmaxa′Qs′,a′;θ′−Qs,a;θ2θ is updated by following stochastic gradient of the loss as θ⟵θ−β∇θLθ, while the target parameter θ′ is updated according to θ′⟵θ every C steps [24]. The details of DQN algorithm is summarized in Algorithm 1.Algorithm 1. DQN algorithm.
Initialize replay bufferD
Initialize action value functionQ with parameter θ
Initialize target action value functionQ^ with parameter θ′=θ
Input the initial state to the DQN
for t=1,2,..⋯ do
Execute actionat from Q using ε-greedy policy
Observert+1 and st+1 from the environment.
Store the transitionst,at,rt+1,st+1 into the replay buffer D
Sample random minibatch of transitions fromD
Evaluate the targetyj=rj+γmaxa′Q^sj+1,a′;θ′
Perform a gradient descent step onyj−Qsj,aj;θ2 with respect to θ
EveryC steps, update the target network Q^ according to θ′⟵θ
end for


## 3.2. DRL Formulation for Dynamic Band Selection

Each user is implemented as DRL, specifically by deep Q-network (DQN) agent that relies on the output of their deep neural network to make dynamic band selection decisions between licensed and unlicensed bands. The DRL formulation is presented below.(i)
ActionIn each time slott, the n-th agent samples an action ant from the action set
(15)A=Licensed,Unlicensed.
(ii)
StateAfter executing the actionant, the agent receives binary observation and reward from the environment. The observation is either ont=1 if the uplink rate in the selected band exceeds the minimum threshold rate or ont=0 otherwise. The state of the agent is defined as history of an action-observation pairs with length H:
(16)snt≜ani,onii=t−Ht
(iii)
RewardDepending on the selected action, the agent receives the following scalar reward:(17)Ifant=Unlicensed,rt+1n=Rn,Ut,ifRn,Ut≥RU,min0,otherwiseLicensed,rt+1n=Rn,Lt,ifRn,Lt≥RL,min0,otherwisewhere Rn,Ut and Rn,Lt are given according to Equations (8) and (9), while RU,min and RL,min are the uplink minimum threshold rates on unlicensed and licensed band, respectively.


## 3.3. Deep Neural Network Description

For dynamic band selection, each UE trains independent DQN. The structure of the deep neural network is shown in Figure4.Figure 4
Structure of the proposed deep neural network.The deep neural network consists of long short-term memory (LSTM) layer, fully connected layers, and rectified linear unit (ReLu) activation function.Long short-term memory(LSTM) is one class of recurrent neural networks (RNNs) which are designed to learn a specific pattern in a sequence of data by taking time correlation into account. They were initially introduced to overcome the vanishing (exploding) gradient problem of RNNs in the course of back propagation. Regulated by gate functions, the cell (internal memory) state of an LSTM learns how to aggregate inputs separated by time, i.e., which experiences to keep or throw away [25]. In our formulation, note that the states of the agents, which are histories of action-observation pairs, have long-term dependency (correlation) emanating from the dynamics of WiFi APs’ activities that follow a two-state Markov property, and the time-varying channel conditions according to Gaussian Markov block-fading autoregressive model. LSTM is crucial for the learning process since it can capture the actual state by exploiting the underlying correlation in the history of action-observation pairs. Therefore, the state must pass through this preprocessing step before it is directly fed to the neural network.A deep neural network consists of multiple fully connected layers, in which each of the layers abstracts certain feature of the input. Letx be the input to the layer, while W and b are the weight matrix and bias vector, respectively. The output vector of a layer, denoted as y, in a fully connected layer can be described by the following operation:
(18)y=fWx+b,where f is the element-wise excitation (activation) that adds nonlinearity. In our simulations, we input the states to an LSTM layer with hidden units of 64, whose output is fed to two fully connected hidden layers with 128 and 64 neurons. The output layer produces action values Qs,a for both actions. ReLu activation function is used on all the layers to avoid the vanishing gradient problem [26]. The target network also adopts the same neural network structure.


## 3.4. Training Algorithm Description

The DQNs of the agents are individually trained according to Algorithm2. The loss function given by Equation (14) is used to train the DQN. The hyperparameters are summarized in Table 2.Algorithm 2. DQN training algorithm for dynamic band selection.
for each agent n∈N do
Initialize replay bufferDn
Initialize action value functionQn with parameter θn
Initialize target action value functionQ^n with parameter θn′=θn
Generate initial statesn,1 from the environment simulator
end for
for t=1,2,..⋯ do
for each agent n∈N do
Execute actionan,t from Qn using ε-greedy policy
Collect rewardrn,t+1 and observation on,t+1
Observe the next statesn,t+1 from the environment simulator
Store the transitionsn,t,an,t,rn,t+1,sn,t+1 into Dn
Sample random minibatch of transitions fromDn
Evaluate the targetyn,j=rn,j+γmaxan,j+1Q^nsn,j+1,an,j+1;θn′
Perform a gradient descent step onyn,j−Qnsn,j,an,j;θn2with respect to θn
EveryC steps, update the target network Q^n according to θn′⟵θn
end for
end forTable 2
Hyperparameters.
Parameter
Value
Discount factorγ
0.9
Learning rateβ
0.01
Explorationε in ε-greedy policy
0.05 to 0.01
Target network update frequencyC
300
Mini batch size
32
Replay bufferD size
1000Note that the agents do not have a complete knowledge of the environment, such as the action of other agents, the underlying dynamics of the WiFi APs’ activities, and varying channel conditions. Instead, through sequential interaction with the environment, each agent makes decisions on band selection solely based on local feedbacks (reward and observation) from the base station. This significantly reduces the training complexity (cost) at each user. Moreover, since the training can be conducted in an offline manner, the trained weights can be used in deployment phase. Retraining the weights is done infrequently; for example, if the environment significantly changes.


## 4. Simulation Results

### 4.1. Simulation Setup

For each realization, we first distribute 10 users uniformly in a square area of100m×100m. Within 30 m distance from the coverage area of the cell, WiFi APs are distributed in homogeneous Poisson point process (PPP) with rate λ. Figure 5 illustrates the network model of one realization for node deployment of BS, users, and APs.Figure 5
Network model for node deployment (layout).We set the dynamics of each WiFi AP activity according to the following transition matrix:(19)P=0.70.30.20.8,We further assume that the uplink transmission of a user over unlicensed band can be interfered from any active WiFi APm∈ℳ within 30 m range. Table 3 summarizes the values of all simulation parameters used for evaluating the proposed algorithm.Table 3
Simulation parameters.
Parameter
Value
Total bandwidth in unlicensed band (WU)
10MHz
Total bandwidth in licensed band (WL)
10MHz
Uplink transmission power (P)
20dBm
Receiver noise power (N0)
−147dBm
Path loss exponent (α)
3.76
Reference distance (do)
1m
Channel gain at reference distance (ϒ)
−35.3dB
Channel correlation coefficient (ρn)
0.95
Doppler frequency (fd,n)
70Hz
Transmission probability of active WiFi AP (τ)
0.7


### 4.2. Performance Evaluation

We compared the policy learned by the DRL agents to two benchmark schemes: random policy and fixed distance policy. In random policy, each user randomly decides which band to select, while in fixed policy, decision is made based on the location of the user. Assuming the BS knows the location of the users at each slott; hence, the distance from BS, only users within D meters from the base station transmit using unlicensed band resources, since they are less susceptible to interfering WiFi APs. The others transmit using licensed band resources. Since we assumed transmission from a WiFi AP can affect unlicensed band uplink transmission of any user within 30 m distance, according to the node deployment in Figure 5, in fixed policy users with D=20m from the BS are assigned to unlicensed band resources. The trained DRL policy of each agents should learn this distance without any prior assumption while selecting band. Furthermore, by learning the activities of the APs, the agents should make a dynamic selection.Figure6 compares the per user average success rate of the users for different thresholds at history length H=5, λ=0.5×10−2, and RL,min=RU,min=4Mbps. The dynamic DRL agents entertain around 90% of success rate, outperforming the users of the fixed distance-based policy with all of the thresholds we set. The gain from the fixed distance-based policy is attributed to two factors. The first one is that DRL agents, without any prior assumption, learn the optimal distance d∗ from the BS to make a decision on band selection. In other words, if user n∈N is located outside the optimal distance range dn>d∗, then it transmits over licensed band to avoid interference from nearby WiFi APs. The second factor is that the agents capture the dynamics of both time-varying channel and WiFi APs’ activities while making use of it in dynamically selecting band. It implies that during the absence of transmission from nearby WiFi APs, even if dn>d∗, user n∈N exploits the opportunity of transmitting over unlicensed band; hence, avoids overloading other users on licensed band.Figure 6
Per user average success rate with the training process.To further investigate the gain coming from dynamic decision on band selection, we evaluate the per user average success rate of the users for different throughput thresholds in Figure7. As the threshold values (over both bands) increase from 3 to 5, the gap on performance (per user average success rate) also increases. This indicates that capability of the DRL agents is crucial to maintaining appreciable success rate under a stringent requirement on quality of service (QoS).Figure 7
Per user average success rate at varying thresholds with the training process.In Figure8, per user average throughput obtained by the three policies for history length H=5, λ=0.5×10−2, and RL,min=RU,min=4Mbps is compared. As depicted, the per user average throughput achieved by DRL agents outperforms the other two schemes. The ability of DRL to adapt to changing environment and learn robust policy enabled the agents to outperform a fixed distance-based policy which falls short when either of the bands is overloaded. In other words, even if there is an opportunity to transmit on unlicensed band, due to inactivity of nearby WiFi APs, cell edge users in fixed distance-based policy fail to take advantage of it. Further gain can be obtained by tuning the hyperparameters.Figure 8
Per user average throughput of the policies with the training process.The effect of the number of interfering WiFi APs on the performance of the DRL agents is investigated for history lengthH=5, and RL,min=RU,min=4Mbps in Figure 9. As the number of WiFi APs increases (when λ increases), the gain due to dynamic decision on band selection reduces since the number of contenders for unlicensed band resources increases. However, the agents still retain the gain coming from learning the optimal distance for band selection. The performance of the fixed distance-based policy is unaffected by the number of WiFi APs.Figure 9
Effect of number of WiFi APs on per user average success rate.Next, in Figure10, we compare the effect of history size on the performance of the DRL agents. We observe that shorter history sizes tend to converge relatively faster. The variation of convergence time of the learned policy is however marginal. This implies the convergence time of the learned policy is generally less sensitive to history size. Note that all the results are averaged from three numerical simulations.Figure 10
Effect of history size on convergence of the training.


## 4.1. Simulation Setup

For each realization, we first distribute 10 users uniformly in a square area of100m×100m. Within 30 m distance from the coverage area of the cell, WiFi APs are distributed in homogeneous Poisson point process (PPP) with rate λ. Figure 5 illustrates the network model of one realization for node deployment of BS, users, and APs.Figure 5
Network model for node deployment (layout).We set the dynamics of each WiFi AP activity according to the following transition matrix:(19)P=0.70.30.20.8,We further assume that the uplink transmission of a user over unlicensed band can be interfered from any active WiFi APm∈ℳ within 30 m range. Table 3 summarizes the values of all simulation parameters used for evaluating the proposed algorithm.Table 3
Simulation parameters.
Parameter
Value
Total bandwidth in unlicensed band (WU)
10MHz
Total bandwidth in licensed band (WL)
10MHz
Uplink transmission power (P)
20dBm
Receiver noise power (N0)
−147dBm
Path loss exponent (α)
3.76
Reference distance (do)
1m
Channel gain at reference distance (ϒ)
−35.3dB
Channel correlation coefficient (ρn)
0.95
Doppler frequency (fd,n)
70Hz
Transmission probability of active WiFi AP (τ)
0.7


## 4.2. Performance Evaluation

We compared the policy learned by the DRL agents to two benchmark schemes: random policy and fixed distance policy. In random policy, each user randomly decides which band to select, while in fixed policy, decision is made based on the location of the user. Assuming the BS knows the location of the users at each slott; hence, the distance from BS, only users within D meters from the base station transmit using unlicensed band resources, since they are less susceptible to interfering WiFi APs. The others transmit using licensed band resources. Since we assumed transmission from a WiFi AP can affect unlicensed band uplink transmission of any user within 30 m distance, according to the node deployment in Figure 5, in fixed policy users with D=20m from the BS are assigned to unlicensed band resources. The trained DRL policy of each agents should learn this distance without any prior assumption while selecting band. Furthermore, by learning the activities of the APs, the agents should make a dynamic selection.Figure6 compares the per user average success rate of the users for different thresholds at history length H=5, λ=0.5×10−2, and RL,min=RU,min=4Mbps. The dynamic DRL agents entertain around 90% of success rate, outperforming the users of the fixed distance-based policy with all of the thresholds we set. The gain from the fixed distance-based policy is attributed to two factors. The first one is that DRL agents, without any prior assumption, learn the optimal distance d∗ from the BS to make a decision on band selection. In other words, if user n∈N is located outside the optimal distance range dn>d∗, then it transmits over licensed band to avoid interference from nearby WiFi APs. The second factor is that the agents capture the dynamics of both time-varying channel and WiFi APs’ activities while making use of it in dynamically selecting band. It implies that during the absence of transmission from nearby WiFi APs, even if dn>d∗, user n∈N exploits the opportunity of transmitting over unlicensed band; hence, avoids overloading other users on licensed band.Figure 6
Per user average success rate with the training process.To further investigate the gain coming from dynamic decision on band selection, we evaluate the per user average success rate of the users for different throughput thresholds in Figure7. As the threshold values (over both bands) increase from 3 to 5, the gap on performance (per user average success rate) also increases. This indicates that capability of the DRL agents is crucial to maintaining appreciable success rate under a stringent requirement on quality of service (QoS).Figure 7
Per user average success rate at varying thresholds with the training process.In Figure8, per user average throughput obtained by the three policies for history length H=5, λ=0.5×10−2, and RL,min=RU,min=4Mbps is compared. As depicted, the per user average throughput achieved by DRL agents outperforms the other two schemes. The ability of DRL to adapt to changing environment and learn robust policy enabled the agents to outperform a fixed distance-based policy which falls short when either of the bands is overloaded. In other words, even if there is an opportunity to transmit on unlicensed band, due to inactivity of nearby WiFi APs, cell edge users in fixed distance-based policy fail to take advantage of it. Further gain can be obtained by tuning the hyperparameters.Figure 8
Per user average throughput of the policies with the training process.The effect of the number of interfering WiFi APs on the performance of the DRL agents is investigated for history lengthH=5, and RL,min=RU,min=4Mbps in Figure 9. As the number of WiFi APs increases (when λ increases), the gain due to dynamic decision on band selection reduces since the number of contenders for unlicensed band resources increases. However, the agents still retain the gain coming from learning the optimal distance for band selection. The performance of the fixed distance-based policy is unaffected by the number of WiFi APs.Figure 9
Effect of number of WiFi APs on per user average success rate.Next, in Figure10, we compare the effect of history size on the performance of the DRL agents. We observe that shorter history sizes tend to converge relatively faster. The variation of convergence time of the learned policy is however marginal. This implies the convergence time of the learned policy is generally less sensitive to history size. Note that all the results are averaged from three numerical simulations.Figure 10
Effect of history size on convergence of the training.


## 5. Conclusion and Future Works

To improve the underlying resource utilization problem in uplink eLAA, we presented a learning-based fully decentralized dynamic band selection scheme. In particular, employing the deep reinforcement learning algorithm, we have implemented each user as an agent that makes a decision based on the output of the DQN, without waiting for scheduling from BS. It is shown that despite the lack of the knowledge of the underlying dynamics of WiFi APs’ activities, the DRL agents successfully learn a robust policy to make a dynamic decision on band selection. Such dynamic and decentralized learning approach can significantly improve the resource utilization problem associated with unlicensed band, due to hidden nodes, in the uplink eLAA system. In a future study, we want to extend this work to more complicated scenarios that involve joint resource allocation over the two bands. Moreover, to improve the gain presented in this paper, different architectures and hyperparameters should be investigated.



---

*Source: 5937358-2020-03-27.xml*