# A Novel Scheduling Index Rule Proposal for QoE Maximization in Wireless Networks

Authors: Ianire Taboada; Fidel Liberal
Journal: Abstract and Applied Analysis
 (2014)

---

## Abstract

This paper deals with the resource allocation problem aimed at maximizing users’ perception of quality in wireless channels with time-varying capacity.
First of all, we model the subjective quality-aware scheduling problem in the framework of Markovian decision processes. Then, given that the obtaining of the optimal solution of this model is unachievable, we propose a simple scheduling index rule with closed-form expression by using a methodology based on Whittle approach. Finally, we analyze the performance of the achieved scheduling proposal in several relevant scenarios, concluding that it outperforms the most popular existing resource allocation strategies.


---

## Body

## 1. Introduction

Undoubtedly, the use of mobile Internet applications has notably increased over the last years, which has led to the growth of the demand for wireless bandwidth. Hence, one of the fundamental challenges that network providers nowadays face is how to efficiently share radio resources among users’ traffic flows. In wireless links channel capacity evolves over time due to the intrinsic degradations of this medium, which has motivated the investigation of scheduling problems in time-varying channels.Traditional scheduling strategies for resource allocation are oriented to objective quality parameters such as delay. Nevertheless, considering the importance and the necessity of network resource allocation for maximizing users’ subjective quality of service or quality of experience (QoE) [1], QoE-driven scheduling becomes essential for network providers.Thus, motivated by the necessity of obtaining an implementable QoE-aware scheduler in wireless channels with time-varying capacity, in this paper we aim at characterizing in closed-form a novel channel-aware scheduler for the problem of maximizing users’ perceived quality. We focus on a scenario where traffic flows arrive and depart upon service completion.


### 1.1. Background

Channel-aware or opportunistic schedulers give priority to users in good channel conditions. Although several channel-aware strategies exist in the literature, Max Rate and Proportional Fair [2] among the most popular, the achievement of the optimal solution for time-varying scheduling optimization problems is rather difficult and unknown. Without any doubt, most analyzed resource allocation problems aim at minimizing the average delay of user traffic flows [3–7].The best contribution found in the field of optimizing delay in a time-varying context is found in [6], which proposes a simple index-based heuristic scheduler that performs satisfactorily. In this work, the authors consider exponentially distributed flow sizes and formulate the optimal scheduling problem as a Markov decision process (MDP) [8]. Since this model is PSPACE-hard [9], it does not allow obtaining a tractable solution, and the authors use approximations for its resolution. This way, using the theory of restless bandits [10] and Whittle approach [11], the authors develop an index rule called potential improvement (PI), which has the property of being asymptotically optimal and fluid-optimality [12]. Nonetheless, analyzing the problem of minimizing mean delay for generally distributed flow sizes in randomly time-varying channels remains still open.Furthermore, even though a few QoE and channel-aware scheduling proposals exist [13–15], no significant analytical framework is provided for these heuristics. In fact, there is no solution based on relevant mathematical methods such as MDP for modelling and Whittle method for its resolution in order to obtain a simple scheduling index rule.On the other hand, it is worth mentioning the work carried out in [16], which is pioneer in introducing QoE-awareness in a MDP. Moreover, since the proposed model is analytically unsolvable in general, they achieve an easy, tractable, and well-performing QoE-aware index rule based on Gittins approach [4] for constant channel capacity, which they denominate Gittins MOS (GM). It is known that Gittins proposed a method to obtain the so-called Gittins index which originally minimized mean delay for constant channel capacity. However, as concluded in [17], Gittins approach is unsuitable for solving problems with time-varying channel.


### 1.2. Scope and Contributions

Since the stochastic and dynamic resource allocation problem of subjective quality maximization in channels with time-varying capacity is analytically and computationally unfeasible for finding an optimal solution, the main objective of this work is to design a simple and tractable heuristic priority scheduling rule using analytical tools that have had a great contribution in the optimization area.Thus, the main contribution of this paper is threefold.(i)
Firstly, we propose a QoE-aware MDP model in a time-varying channel context.(ii)
Secondly, in order to achieve a tractable solution for the aforementioned QoE and channel-aware model, we focus on designing a simple heuristic index rule using Whittle approach.(iii)
Thirdly, although for many years exponential flow size distributions have been considered for traffic modelling in order to simplify the resolution of scheduling optimization problems, as a step forward, in this work we take into account size distributions that better capture the real world patterns.The remainder of the paper is organized as follows. First of all, we characterize the QoE-aware scheduling problem for the time-varying channel context in Section2. Then, Section 3 describes the extension of the basic QoE-aware MDP model to time-varying channels. In Section 4 we achieve the Whittle index-based solution for the proposed MDP, and its performance is evaluated in Section 5. Finally, Section 6 gathers the main conclusions of this paper.


## 1.1. Background

Channel-aware or opportunistic schedulers give priority to users in good channel conditions. Although several channel-aware strategies exist in the literature, Max Rate and Proportional Fair [2] among the most popular, the achievement of the optimal solution for time-varying scheduling optimization problems is rather difficult and unknown. Without any doubt, most analyzed resource allocation problems aim at minimizing the average delay of user traffic flows [3–7].The best contribution found in the field of optimizing delay in a time-varying context is found in [6], which proposes a simple index-based heuristic scheduler that performs satisfactorily. In this work, the authors consider exponentially distributed flow sizes and formulate the optimal scheduling problem as a Markov decision process (MDP) [8]. Since this model is PSPACE-hard [9], it does not allow obtaining a tractable solution, and the authors use approximations for its resolution. This way, using the theory of restless bandits [10] and Whittle approach [11], the authors develop an index rule called potential improvement (PI), which has the property of being asymptotically optimal and fluid-optimality [12]. Nonetheless, analyzing the problem of minimizing mean delay for generally distributed flow sizes in randomly time-varying channels remains still open.Furthermore, even though a few QoE and channel-aware scheduling proposals exist [13–15], no significant analytical framework is provided for these heuristics. In fact, there is no solution based on relevant mathematical methods such as MDP for modelling and Whittle method for its resolution in order to obtain a simple scheduling index rule.On the other hand, it is worth mentioning the work carried out in [16], which is pioneer in introducing QoE-awareness in a MDP. Moreover, since the proposed model is analytically unsolvable in general, they achieve an easy, tractable, and well-performing QoE-aware index rule based on Gittins approach [4] for constant channel capacity, which they denominate Gittins MOS (GM). It is known that Gittins proposed a method to obtain the so-called Gittins index which originally minimized mean delay for constant channel capacity. However, as concluded in [17], Gittins approach is unsuitable for solving problems with time-varying channel.


## 1.2. Scope and Contributions

Since the stochastic and dynamic resource allocation problem of subjective quality maximization in channels with time-varying capacity is analytically and computationally unfeasible for finding an optimal solution, the main objective of this work is to design a simple and tractable heuristic priority scheduling rule using analytical tools that have had a great contribution in the optimization area.Thus, the main contribution of this paper is threefold.(i)
Firstly, we propose a QoE-aware MDP model in a time-varying channel context.(ii)
Secondly, in order to achieve a tractable solution for the aforementioned QoE and channel-aware model, we focus on designing a simple heuristic index rule using Whittle approach.(iii)
Thirdly, although for many years exponential flow size distributions have been considered for traffic modelling in order to simplify the resolution of scheduling optimization problems, as a step forward, in this work we take into account size distributions that better capture the real world patterns.The remainder of the paper is organized as follows. First of all, we characterize the QoE-aware scheduling problem for the time-varying channel context in Section2. Then, Section 3 describes the extension of the basic QoE-aware MDP model to time-varying channels. In Section 4 we achieve the Whittle index-based solution for the proposed MDP, and its performance is evaluated in Section 5. Finally, Section 6 gathers the main conclusions of this paper.


## 2. Problem Description

We analyze the problem of maximizing average QoE in time-varying channels. Even though this study is applicable to any time-varying channel context, we focus on wireless networks. In particular, we centre on a wireless downlink data channel in a single cell system. In this way, at the beginning of each transmission time interval (TTI), the scheduler located in the base station makes decisions in order to choose a user traffic flow to transmit.


### 2.1. User

Each userk in the system is uniquely associated with the flow related to its requested content, with its QoE characteristic and with its wireless channel. We use a generic concept of flow, which refers to an amount of data that can be displayed to (received by) the end user as a standalone element.We consider a system with a fixed number of users, without arrivals of new users. This assumption simplifies the mathematical model. However, being aware of the impact of arrivals on the performance of scheduling, we will analyze the performance of scheduling strategies in the presence of arrivals.


#### 2.1.1. Flow Size

Flows are characterized by their random sizeX, the total amount of bits to be transferred. Sizes are assumed to be independently distributed with E[X]<∞, where F(x)=P(X≤x), f(x), and h(x)=f(x)/(1-F(x)) are the corresponding cumulative distribution function, density function, and hazard rate, respectively.We focus on the important class of size distributions with a decreasing hazard rate, particularly we assume Pareto distributed flow sizes. It is known that Internet traffic flow sizes are properly modelled by means of Pareto distributions [18], which better fit Internet flow sizes compared with the simplistic exponential approach. We define the Pareto distribution with shape parameter α>1 and scale parameter γ>0 whose density function for all x≥0 is
(1)f(x)=γα(1+γx)α+1.Note that, even though we use Pareto distributed flow sizes, the results provided are valid for any size distribution with a decreasing hazard rate.


#### 2.1.2. QoE Characteristic

The time needed to transmit to channel all the bits from a user flow has a direct impact on user’s perceived quality. We consider that this delay is the main cause of subjective quality distortion. Nowadays, most of the Internet traffic is transported over TCP, which implements the packet recovery mechanisms to avoid application level losses. In TCP-based services, the predominant source of losses at the application is packets arriving later than their playout time. Hence, it is generally useful to provide a delay-driven QoE-awareness.This way, we quantify QoE by a delay dependent Mean Opinion Score (MOS) [19] utility function. User satisfaction is evaluated in MOS scale of 1 (very poor quality) to 5 (excellent quality).As shown in QoE studies [20], perceived quality is maximum until a low delay value, dmin⁡. Then, from this delay threshold, QoE degradation happens until a high delay value, dmax⁡. Above this delay threshold service is unaffordable, resulting in a MOS value of 1. The values of these delay thresholds depend on several factors [21], such as the flow size, the service type, the network type and the user expertise.We illustrate an example of the explained behavior in Figure1.Figure 1
MOS versus delay.


#### 2.1.3. Channel Model

The channel quality associated to a userk evolves randomly and independently of its evolution history. We assume that the channel can be in N∶={1,2,…,Nk} conditions. Therefore, user k is in channel condition n with probability qk,n, having ∑n∈Nqk,n=1. In channel state n, rk,n bits are transmitted to channel, and transmission rates sk,n are sk,1≤sk,2≤⋯≤sk,Nk.


### 2.2. Server

The objective of the scheduler is finding a policyπ that solves the following optimization problem:
(2)max⁡π1K∑k=1KMOSk(dk).
According to the already described aspects about the scheduling problem, we consider a set of admissible policies, Π, which fulfils the next properties.(i)
Delay dependency: the time that a flow remains in the system,d, is known.(ii)
QoE-awareness: the delay-dependent MOS function that characterizes user subjective quality is known.(iii)
Channel-awareness: channel quality indicator (CQI) information sent from mobile users is used as user’s instantaneous channel condition information.(iv)
Nonanticipation: similarly to most current IP systems, flow size is unknown. However, existent nonanticipating size-based disciplines make use of flow attained service [4, 7], the bits that have been transferred of a flow. Due to the memoryless property of the exponential distributions, for these distributions, the hazard rate is independent of attained service. Nevertheless, the considered Pareto distributions have a decreasing hazard rate with attained service [22]; this is, completion probability, μ(a,n), is higher for lower attained service values. This way, attained service, a, is used as size information.(v)
Preemption: we assume that the server is preemptive, that is, at every decision epoch, it is permitted to suspend the service of a flow whose transmission is unfinished.(vi)
Single service: only the transmission of a flow is allowed in each TTI.


## 2.1. User

Each userk in the system is uniquely associated with the flow related to its requested content, with its QoE characteristic and with its wireless channel. We use a generic concept of flow, which refers to an amount of data that can be displayed to (received by) the end user as a standalone element.We consider a system with a fixed number of users, without arrivals of new users. This assumption simplifies the mathematical model. However, being aware of the impact of arrivals on the performance of scheduling, we will analyze the performance of scheduling strategies in the presence of arrivals.


### 2.1.1. Flow Size

Flows are characterized by their random sizeX, the total amount of bits to be transferred. Sizes are assumed to be independently distributed with E[X]<∞, where F(x)=P(X≤x), f(x), and h(x)=f(x)/(1-F(x)) are the corresponding cumulative distribution function, density function, and hazard rate, respectively.We focus on the important class of size distributions with a decreasing hazard rate, particularly we assume Pareto distributed flow sizes. It is known that Internet traffic flow sizes are properly modelled by means of Pareto distributions [18], which better fit Internet flow sizes compared with the simplistic exponential approach. We define the Pareto distribution with shape parameter α>1 and scale parameter γ>0 whose density function for all x≥0 is
(1)f(x)=γα(1+γx)α+1.Note that, even though we use Pareto distributed flow sizes, the results provided are valid for any size distribution with a decreasing hazard rate.


### 2.1.2. QoE Characteristic

The time needed to transmit to channel all the bits from a user flow has a direct impact on user’s perceived quality. We consider that this delay is the main cause of subjective quality distortion. Nowadays, most of the Internet traffic is transported over TCP, which implements the packet recovery mechanisms to avoid application level losses. In TCP-based services, the predominant source of losses at the application is packets arriving later than their playout time. Hence, it is generally useful to provide a delay-driven QoE-awareness.This way, we quantify QoE by a delay dependent Mean Opinion Score (MOS) [19] utility function. User satisfaction is evaluated in MOS scale of 1 (very poor quality) to 5 (excellent quality).As shown in QoE studies [20], perceived quality is maximum until a low delay value, dmin⁡. Then, from this delay threshold, QoE degradation happens until a high delay value, dmax⁡. Above this delay threshold service is unaffordable, resulting in a MOS value of 1. The values of these delay thresholds depend on several factors [21], such as the flow size, the service type, the network type and the user expertise.We illustrate an example of the explained behavior in Figure1.Figure 1
MOS versus delay.


### 2.1.3. Channel Model

The channel quality associated to a userk evolves randomly and independently of its evolution history. We assume that the channel can be in N∶={1,2,…,Nk} conditions. Therefore, user k is in channel condition n with probability qk,n, having ∑n∈Nqk,n=1. In channel state n, rk,n bits are transmitted to channel, and transmission rates sk,n are sk,1≤sk,2≤⋯≤sk,Nk.


## 2.1.1. Flow Size

Flows are characterized by their random sizeX, the total amount of bits to be transferred. Sizes are assumed to be independently distributed with E[X]<∞, where F(x)=P(X≤x), f(x), and h(x)=f(x)/(1-F(x)) are the corresponding cumulative distribution function, density function, and hazard rate, respectively.We focus on the important class of size distributions with a decreasing hazard rate, particularly we assume Pareto distributed flow sizes. It is known that Internet traffic flow sizes are properly modelled by means of Pareto distributions [18], which better fit Internet flow sizes compared with the simplistic exponential approach. We define the Pareto distribution with shape parameter α>1 and scale parameter γ>0 whose density function for all x≥0 is
(1)f(x)=γα(1+γx)α+1.Note that, even though we use Pareto distributed flow sizes, the results provided are valid for any size distribution with a decreasing hazard rate.


## 2.1.2. QoE Characteristic

The time needed to transmit to channel all the bits from a user flow has a direct impact on user’s perceived quality. We consider that this delay is the main cause of subjective quality distortion. Nowadays, most of the Internet traffic is transported over TCP, which implements the packet recovery mechanisms to avoid application level losses. In TCP-based services, the predominant source of losses at the application is packets arriving later than their playout time. Hence, it is generally useful to provide a delay-driven QoE-awareness.This way, we quantify QoE by a delay dependent Mean Opinion Score (MOS) [19] utility function. User satisfaction is evaluated in MOS scale of 1 (very poor quality) to 5 (excellent quality).As shown in QoE studies [20], perceived quality is maximum until a low delay value, dmin⁡. Then, from this delay threshold, QoE degradation happens until a high delay value, dmax⁡. Above this delay threshold service is unaffordable, resulting in a MOS value of 1. The values of these delay thresholds depend on several factors [21], such as the flow size, the service type, the network type and the user expertise.We illustrate an example of the explained behavior in Figure1.Figure 1
MOS versus delay.


## 2.1.3. Channel Model

The channel quality associated to a userk evolves randomly and independently of its evolution history. We assume that the channel can be in N∶={1,2,…,Nk} conditions. Therefore, user k is in channel condition n with probability qk,n, having ∑n∈Nqk,n=1. In channel state n, rk,n bits are transmitted to channel, and transmission rates sk,n are sk,1≤sk,2≤⋯≤sk,Nk.


## 2.2. Server

The objective of the scheduler is finding a policyπ that solves the following optimization problem:
(2)max⁡π1K∑k=1KMOSk(dk).
According to the already described aspects about the scheduling problem, we consider a set of admissible policies, Π, which fulfils the next properties.(i)
Delay dependency: the time that a flow remains in the system,d, is known.(ii)
QoE-awareness: the delay-dependent MOS function that characterizes user subjective quality is known.(iii)
Channel-awareness: channel quality indicator (CQI) information sent from mobile users is used as user’s instantaneous channel condition information.(iv)
Nonanticipation: similarly to most current IP systems, flow size is unknown. However, existent nonanticipating size-based disciplines make use of flow attained service [4, 7], the bits that have been transferred of a flow. Due to the memoryless property of the exponential distributions, for these distributions, the hazard rate is independent of attained service. Nevertheless, the considered Pareto distributions have a decreasing hazard rate with attained service [22]; this is, completion probability, μ(a,n), is higher for lower attained service values. This way, attained service, a, is used as size information.(v)
Preemption: we assume that the server is preemptive, that is, at every decision epoch, it is permitted to suspend the service of a flow whose transmission is unfinished.(vi)
Single service: only the transmission of a flow is allowed in each TTI.


## 3. MDP Formulation

In this section we formulate the problem described in Section2 as a MDP. First, we provide the QoE-aware MDP model of each user k. Then, we formulate the optimization problem for the joint MDP model, which takes into account all the users in the system.


### 3.1. MDP Model of a Job

The action space,B, of user k is binary; action 0 means not serving, and if action 1 is chosen rk,n bits from this user being in channel state n are transmitted to channel. Besides, each user k is defined by tuple (Sk,(Rk,sb)b∈B,(Wk,sb)b∈B,(Pk,sb)b∈B). These elements are defined as follows.(i)
Sk=(Ak×Dk×{1,2,…,Nk})∪{[Dk+1]}∪{[*]} is the set of all s states for a user k, which is classified in three groups.
(a)
Unfinished states: all the states in which flow transmission is uncompleted belong to this group. These states have three dimensions, denoted ass=(a,d,n), with attained service level a∈Ak, delay level d∈Dk, and channel condition n∈{1,2,…,Nk} as components. The attained services that correspond to these attained service levels are multiples of the first not null rk,n transmission rate, whereas the delays that correspond to delay levels are multiples of TTI.(b)
Reward states: theses=[d] states are reached once flow transmission has finished, and in these delay-dependent states the QoE-aware reward is given. This way, aimed at maximizing average MOS, a delay dependant QoE-aware reward giving only happens once the whole flow has been transmitted, since from subjective quality perspective the MOS function only makes sense once flow transmission is completed (see [16] for more details).(c)
Final state: the chain of states ends in the absorbing states=[*].(ii)
Rkb∶=(Rk,sb)s∈Sk, where Rk,sb is the expected one-slot reward received for user k at state s if action b is decided at the beginning of a TTI,
(3)Rk,(a,d,n)b=0,Rk,[d]b=MOS^(d+1),Rk,[*]b=0,whereMOS^(d+1) is normalized MOS in the range [0,1].(iii)
Wkb∶=(Wk,sb)s∈Sk, where Wk,sb is the expected one-slot work done for user k at state s if action b is decided at the beginning of a TTI,
(4)Wk,s0=0,Wk,s1=1;(iv)
Pkb∶=(pkb(s,s′))s,s′∈Sk, where pkb(s,s′) is the probability of user k of moving from state s to state s′ if action b is decided at the beginning of a TTI,
(5)pk0((a,d,n),(a,d+1,m))=qk,m,pk1((a,d,n),(a+rn,d+1,m))=qk,m·(1-μk,(a,n)),pk1((a,d,n),[d+1])=μk,(a,n),pk0([d],[*])=pk0([*],[*])=1,whereμk,(a,n)=P(a<Xk≤a+rk,n∣Xk>a). Observe that the probability of changing from channel state n to channel state m corresponds to the steady-state probability qk,m due to the memoryless property of channel evolution.Thus, the dynamics of userk is captured by state process sk(t)∈Sk and action process bk(t)∈B. Moreover, we highlight that the proposed MDP model has restless property, which admits evolution and rewards even if not chosen for transmission due to the delay increase and the channel stochastic evolution.Figure2 shows part of the per-user state diagram of the new QoE-aware MDP model, which summarizes and relates the aforedescribed components.Figure 2
A part of per-user state diagram.


### 3.2. Optimization Problem

Besides, the optimization problem (2) associated with such MDP can be written as
(6)max⁡π∈ΠE0π[∑t=0∞∑k∈KβtRk,((ak(t),dk(t),nk(t))∪[dk(t)])bk(t)],∑k∈Kbk(t)=1∀t.
Note that we consider the undiscounted reward case, where the discounted factor β is 1, being reward deterioration in time implicit in the delay-dependent MOS function.Nonetheless, optimally solving problem (6) is unfeasible in general. On the one hand, when channel capacity is constant the impossibility of obtaining the exact QoE-aware solution is revealed in [16], and hence, since the time-varying channel model is more complex, the achievement of the optimal solution is unviable in this case. On the other hand, it is known that the allocation constraint causes intractability in similar problems [11], and therefore, in our case having three-dimensional states finding the optimal solution is extremely difficult or even unachievable. In this way, once seeing the impossibility of obtaining the optimal solution for the considered problem in this work, we will focus on achieving an approximate solution. For that purpose, in the next section we will propose a heuristic based on Whittle method.


## 3.1. MDP Model of a Job

The action space,B, of user k is binary; action 0 means not serving, and if action 1 is chosen rk,n bits from this user being in channel state n are transmitted to channel. Besides, each user k is defined by tuple (Sk,(Rk,sb)b∈B,(Wk,sb)b∈B,(Pk,sb)b∈B). These elements are defined as follows.(i)
Sk=(Ak×Dk×{1,2,…,Nk})∪{[Dk+1]}∪{[*]} is the set of all s states for a user k, which is classified in three groups.
(a)
Unfinished states: all the states in which flow transmission is uncompleted belong to this group. These states have three dimensions, denoted ass=(a,d,n), with attained service level a∈Ak, delay level d∈Dk, and channel condition n∈{1,2,…,Nk} as components. The attained services that correspond to these attained service levels are multiples of the first not null rk,n transmission rate, whereas the delays that correspond to delay levels are multiples of TTI.(b)
Reward states: theses=[d] states are reached once flow transmission has finished, and in these delay-dependent states the QoE-aware reward is given. This way, aimed at maximizing average MOS, a delay dependant QoE-aware reward giving only happens once the whole flow has been transmitted, since from subjective quality perspective the MOS function only makes sense once flow transmission is completed (see [16] for more details).(c)
Final state: the chain of states ends in the absorbing states=[*].(ii)
Rkb∶=(Rk,sb)s∈Sk, where Rk,sb is the expected one-slot reward received for user k at state s if action b is decided at the beginning of a TTI,
(3)Rk,(a,d,n)b=0,Rk,[d]b=MOS^(d+1),Rk,[*]b=0,whereMOS^(d+1) is normalized MOS in the range [0,1].(iii)
Wkb∶=(Wk,sb)s∈Sk, where Wk,sb is the expected one-slot work done for user k at state s if action b is decided at the beginning of a TTI,
(4)Wk,s0=0,Wk,s1=1;(iv)
Pkb∶=(pkb(s,s′))s,s′∈Sk, where pkb(s,s′) is the probability of user k of moving from state s to state s′ if action b is decided at the beginning of a TTI,
(5)pk0((a,d,n),(a,d+1,m))=qk,m,pk1((a,d,n),(a+rn,d+1,m))=qk,m·(1-μk,(a,n)),pk1((a,d,n),[d+1])=μk,(a,n),pk0([d],[*])=pk0([*],[*])=1,whereμk,(a,n)=P(a<Xk≤a+rk,n∣Xk>a). Observe that the probability of changing from channel state n to channel state m corresponds to the steady-state probability qk,m due to the memoryless property of channel evolution.Thus, the dynamics of userk is captured by state process sk(t)∈Sk and action process bk(t)∈B. Moreover, we highlight that the proposed MDP model has restless property, which admits evolution and rewards even if not chosen for transmission due to the delay increase and the channel stochastic evolution.Figure2 shows part of the per-user state diagram of the new QoE-aware MDP model, which summarizes and relates the aforedescribed components.Figure 2
A part of per-user state diagram.


## 3.2. Optimization Problem

Besides, the optimization problem (2) associated with such MDP can be written as
(6)max⁡π∈ΠE0π[∑t=0∞∑k∈KβtRk,((ak(t),dk(t),nk(t))∪[dk(t)])bk(t)],∑k∈Kbk(t)=1∀t.
Note that we consider the undiscounted reward case, where the discounted factor β is 1, being reward deterioration in time implicit in the delay-dependent MOS function.Nonetheless, optimally solving problem (6) is unfeasible in general. On the one hand, when channel capacity is constant the impossibility of obtaining the exact QoE-aware solution is revealed in [16], and hence, since the time-varying channel model is more complex, the achievement of the optimal solution is unviable in this case. On the other hand, it is known that the allocation constraint causes intractability in similar problems [11], and therefore, in our case having three-dimensional states finding the optimal solution is extremely difficult or even unachievable. In this way, once seeing the impossibility of obtaining the optimal solution for the considered problem in this work, we will focus on achieving an approximate solution. For that purpose, in the next section we will propose a heuristic based on Whittle method.


## 4. QoE-Aware Index Rule Proposal Based on Whittle Approach

In this part we provide a Whittle index rule type approximate solution to problem (6) based on Whittle method. The idea of Whittle [11] consists in achieving a function that measures the dynamic service priority in order to obtain a simple scheduling index rule. Whittle indices are computed in isolated way for each user, and they measure the expected efficiency of serving a user in each state.Note that we are only interested in the indices of(a,d,n) states as once flow transmission is completed scheduling makes no sense. Thereby, the optimization problem formulated in (6) can be relaxed by requiring to serve a job per slot on average as proposed in [11], which is further approached by Lagrangian methods [23] and can be decomposed into a single-job price-based parameterized optimization problem. For a price or Lagrangiar parameter v, we will therefore study the user-k subproblem:
(7)max⁡πk∈Π∑t=0∞E0πβt·[Rk,(ak(t),dk(t),nk(t))bk(t)-vWk,(ak(t),dk(t),nk(t))bk(t)].
The Lagrangiar parameter v can be interpreted as the per-slot cost of serving.Let us define serving setF⊆Sk, which prescribes to serve a user k if (a,d,n)∈F, while not to serve this user if (a,d,n)∉F. We will refer to states (a,d,n)∈F as active and (a,d,n)∉F as passive. This way, it is possible to rewrite problem (7) as
(8)max⁡F⊆SkRk,(a,d,n)F-vWk,(a,d,n)F,
where R(a,d,n)F and W(a,d,n)F are, respectively, the expected total reward and expected total work in (a,d,n) state.Conjecture 1.
Problem (8) is indexable.Due to the complexity of our model, proving the indexability of Problem (8) is not an easy task. In this way, we assume Conjecture 1, and therefore, we suppose that for each (a,d,n) state of user k the Whittle index exists. From now on, we omit user label k. In spite of considering the undiscounted case, the mathematical analysis is carried out for the discounted case so as to avoid infinite values caused by 1-β terms in denominators for β=1, and then, we obtain the undiscounted index values in the limit β→1.Following Whittle index definition in [10], we formally write the Whittle index for Problem (8), v(a,d,n)*, as
(9)v(a,d,n)*=v(a,d,n)F=R(a,d,n)〈1,F〉-R(a,d,n)〈0,F〉W(a,d,n)〈1,F〉-W(a,d,n)〈0,F〉.Whittle index,v(a,d,n)F, represents the rate between marginal reward and marginal work, where the marginal reward (work) is the difference of the expected reward earned (work required) by serving and not serving at the initial state (a,d,n) and employing policy F afterwards.Lemma 2.
For any state(a,d,n) and under any policy F we obtain
(10)v(a,d,n)F=(β[(1-μ(a,n))∑m∈NqmR(a+rn,d+1,m)F+μ(a,n)mmmm·MOS^(d+1)-∑m∈NqmR(a,d+1,m)F])×(1+β[(1-μ(a,n))∑m∈NqmW(a+rn,d+1,m)Fmmmmmmm-∑m∈NqmW(a,d+1,m)F])-1.Proof.
From the definition of reward and work, respectively, we have(11)R(a,d,n)F={R(a,d,n)〈1,F〉=β[(1-μ(a,n))∑m∈NqmR(a+rn,d+1,m)Fmmmmmml+μ(a,n)·MOS^(d+1)∑m∈N]R(a,d,n)〈0,F〉=β∑m∈NqmR(a,d+1,m)F(a,d,n)∈FR(a,d,n)〈0,F〉=β∑m∈NqmR(a,d+1,m)F(a,d,n)∉F,(12)W(a,d,n)F={W(a,d,n)〈1,F〉=1+β(1-μ(a,n))∑m∈NqmW(a+rn,d+1,m)FW(a,d,n)〈0,F〉=β∑m∈NqmW(a,d+1,m)F(a,d,n)∈FW(a,d,n)〈0,F〉=β∑m∈NqmW(a,d+1,m)F(a,d,n)∉F.
By substituting the expressions (11) and (12) in formula (9), we achieve a more complete v(a,d,n)F (10) formulation.Nevertheless, in order to obtain an analytically tractable Whittle index expression for (10) in the state (a,d,n), it is necessary to determine the policy F of the future states that have influence on the index computation of this state. As can be observed from (10), (11), and (12) expressions, in these future states the value of attained service is maintained or increased, whereas delay grows over time. Since increasing attained service decreases instant completion probability for a Pareto distribution and increasing delay degrades QoE, it would seem intuitively correct to suppose that future states with the same or worse channel conditions are passive. In such a way, we could say that in a state with the same or worse channel condition being attained service and delay higher that the Whittle index value is lower. However, for future states with better channel conditions their passive or active property is not trivial, because the improvement in channel condition does not guarantee being a better state due to the aforementioned behavior of the increasing attained service and delay. Therefore, finding the active set of all the influential states for Whittle index computation becomes a challenging task.In such situation, aimed at achieving the characteristics of the optimal policy and the subsequent active set building evolution we will use an algorithm called Adaptive-Greedy, shortlyAG-algorithm (see [10] for a survey). This algorithm computes Whittle indices numerically taking into account optimal active sets. This way, we have implemented AG-algorithm for our case study in order to verify the fundamental properties of the Whittle index for our problem, which will be useful to determine the active set, and consequently, a closed-form index expression. We provide the operation of the applied algorithm in Algorithm 1.Algorithm 1:
A
G-algorithm for Problem (8).
(1)F0∶=∅
(2)for  i=1→ number of states do
(3)(a,d,n)i∈argmax⁡⁡{v(a,d,n)Fi-1:(a,d,n)∉Fi-1}
(4)v(a,d,n)i*=v(a,d,n)iFi-1;Fi=Fi-1∪{(a,d,n)i}
(5)end forWe have performed several numerical experiments forβ=1 case, in which we have reduced the state space of both delay and channel states so as to do the execution of the algorithm computationally and in time viably. We have only considered two channel conditions and eleven levels of delay in the scenarios. In such a way, from experimental results we conjecture the following fundamental properties.(1)
The Whittle index value depends on attained service, delay, and channel state. Moreover, as can be seen in graphs from Figures3 and 4, we can clearly distinguish the ranges of MOS function; above the upper delay threshold, the index value is null.(2)
For the same channel condition and the same delay level, Whittle index values are decreasing with attained service:v(a,d,m)*>v(a′,d,m)*  ∀a′>a (illustrated in Figures 3 and 4).(3)
For the same channel condition and the same attained service level, Whittle index values are decreasing with delay:v(a,d,m)*>v(a,d+1,m)*. (See Figures 3 and 4).(4)
In a better channel condition, for the same attained service level and the same delay level, the Whittle index value is higher:v(a,d,m1)*>v(a,d,m2)*  ∀m1>m2. (Observe Figure 3).(5)
Opposite to the previous channel-aware work [6], where Whittle indices in the best channel condition are infinite having absolute priority in this channel state, in our case being in the best channel condition does not guarantee that the Whittle index value is higher than in the rest of the channel conditions: it is possible to fulfil v(a,d,m<N)*>v(a′,d′,N)*. If we compare the two graphs in Figure 3, it can be easily seen how for several attained services and/or delayed values the Whittle index values in the worst channel condition are higher than in the best channel condition (taking also into account nonnull values).(6)
Dependency on channel condition probabilities: hile for channel conditionN the Whittle index value remains invariant with channel state probability, for the rest of channel conditions its value decreases as long as channel state probabilities increase in better channel conditions (shown in Figure 4).Whittle index values obtained fromAG-algorithm for two channel conditions and different attained services and delays. For a Pareto distribution with α=1.5 and E[X]=5Mb, q2=0.5, r1=8.4Kb, r2=16.8Kb, dmin⁡=2s, and dmax⁡=9s.
(a)
(b)Whittle index values obtained fromAG-algorithm for different channel state probabilities. For a Pareto distribution with α=1.5 and E[X]=5Mb, r1=8.4Kb, r2=16.8Kb, dmin⁡=2s, and dmax⁡=9s.
(a)
(b)Once we analyzed the properties of the Whittle index, we set out to define the structure of the active set in order to obtain an analytically and computationally tractable Whittle index expression. In such a way, to derive a closed-form characterization of the Whittle index of(a,d,n) state, first of all it is necessary to specify the activity of (a,d+1,m) and (a+rn,d+1,m) states. As concluded from Whittle index properties, it is known that (a,d+1,m≤n)∪(a+rn,d+1,m≤n) states are passive. However, it is not trivial to specify above which attained service level and/or delay level if (a,d+1,m>n)∪(a+rn,d+1,m>n) states with better channel condition are active or passive.Nonetheless, in the best channel the active set is totally specified since all the states that compute in the index are passive, because of nonexisting channel improvement and being in the future attained service equal or higher and delay higher. As shown in Proposition3, for the best channel condition the Whittle index has a closed-form expression, where the index value is the multiplication between completion probability and normalized MOS function. Note that the obtained Whittle index expression in the best channel state is equivalent to the GM index achieved for constant channel [16].Proposition 3.
The Whittle index for Problem (8) in the best channel condition is given by
(13)v(a,d,N)*=μ(a,N)·MOS^(d+1).Proof.
It is known that forv(a,d,N)* index computation that F=∅. In this way, using (11) and (12): R(a,d+1,m)F=β∑m∈NqmR(a,d+2,m)F;R(a,d+2,m)F=β∑m∈NqmR(a,d+3,m)F;…;R(a,d>dmax⁡,m)F=0. Hence, analogolously, R(a+rn,d+1,m)F=0, W(a,d+1,m)F=0, and W(a+rn,d+1,m)F=0. Substituting these null terms in expression (10), and being β=1, we achieve (13).Besides, as stated before, obtaining a computationally tractable Whittle index expression for any channel condition is not possible. In such situation, in order to achieve a tractable Whittle-based QoE-aware index rule for time-varying channels, we propose a Whittle-based approximate heuristic in the next subsection.


### 4.1. Approximate Whittle Index Proposal

In the following we propose an approximate solution for the QoE-aware Whittle index in a time-varying channel context (10).First of all, we will determine the structure of the active set. For the Whittle index computation of(a,d,n) state, as previously concluded, it is known that (a,d+1,m≤n)∪(a+rn,d+1,m≤n) states are passive, but we need to define the unknown activity of (a,d+1,m>n)∪(a+rn,d+1,m>n) states. Thus, so as to determine if these states are active or passive we apply the following two considerations.(1)
μ approximation:
(14)μ(a,m)≈μ(a+rl,m)(2)
MOS approximation:
(15)MOS(d)≈MOS(d+1).On the one hand, we assume that for anyl=1,2,…,N value that μ(a,m)≈μ(a+rl,m) (14). It is known that (a,d,m>n) states are active, and being μ(a,m)≈μ(a+rn,m), we could consider that (a+rn,d,m>n) states are active.On the other hand, we assumeMOS(d)≈MOS(d+1) (15), since QoE degradation in a higher delay level is negligible in the millisecond scale that delay evolves. Taking into account the approximation about MOS function, being (a,d,m>n) states active, we can suppose that (a,d+1,m>n) states are also active. This way, combining approximations (14) and (15), we consider that (a+rn,d+1,m>n) states are active.Then, suppose that the aforementionedμ and MOS approximations cause Rad=∑mqmR(a,d,m)≈∑mqmR(a+rl,d+1,m) and Wad=∑mqmW(a,d,m)≈∑mqmW(a+rl,d+1,m). So, using these reward and work approximations, the first simplified expression for the Whittle index formulation (10) is
(16)v~(a,d,n)*=βμ(a,n)(MOS^(d+1)-Rad)1-βμ(a,n)Wad.In such a way, applying the previous simplifications, we obtain the Whittle index approximation presented in Proposition4, what we call attained service PI MOS (ASPIM). The methodology to compute the closed-form expression of this QoE-aware Whittle-based index is presented in the Appendix.Proposition 4.
The formulation of the ASPIM index is:(17)ASPIM=v~(a,d,n)*=lim⁡β→1βμ(a,n)·MOS^(d+1)(1-β)1-β+β∑m>nqm(μ(a,m)-μ(a,n)).If we analyze expression (17), we observe that this QoE-aware and channel-aware Whittle-based ASPIM index is implementable. This proposal depends on QoE, size, and channel properties. Moreover, the obtained index for the best channel condition is equal to the original Whittle index (13), since the summation in the denominator becomes null. However, as shown in (18), when channel state is not the best the index value goes to zero:
(18)ASPIMn≠N=lim⁡β→1μ(a,n)·MOS^(d+1)(1-β)∑m>nqm(μ(a,m)-μ(a,n))⟶0.Hence, a user in its best channel state with nonnull instantaneous normalized MOS function has priority over a user which is not in its best channel condition. Thus, we summarize the proposed QoE-aware Whittle-based ASPIM index rule in Definition5.Definition 5.
The ASPIM index rule consists of: at every decision slot,(i)
serving the user in its best channel condition with nonnull instantaneous normalized MOS function with the highest value ofμ(a,N)·MOS^(d+1);(ii)
if there is no user in its best channel condition with nonnull instantaneous normalized MOS function, serving the user with the highest value of(μ(a,n)·MOS^(d+1))/∑m>nqm(μ(a,m)-μ(a,n)) (using (18) normalized by (1-β)).So, if channel condition is not the best, the index value is the ratio of the multiplication between the actual service completion and the instantaneous normalized MOS function respect to the expected potential improvement of the completion probability.Therefore, in the context of time-varying channels aimed at maximizing average MOS, we have proposed ASPIM scheduling algorithm, which is a simple and tractable QoE-aware, channel-aware, and size-based index rule. Nevertheless, in order to verify the correct behavior of this heuristic, we analyze its performance in the subsequent section.


## 4.1. Approximate Whittle Index Proposal

In the following we propose an approximate solution for the QoE-aware Whittle index in a time-varying channel context (10).First of all, we will determine the structure of the active set. For the Whittle index computation of(a,d,n) state, as previously concluded, it is known that (a,d+1,m≤n)∪(a+rn,d+1,m≤n) states are passive, but we need to define the unknown activity of (a,d+1,m>n)∪(a+rn,d+1,m>n) states. Thus, so as to determine if these states are active or passive we apply the following two considerations.(1)
μ approximation:
(14)μ(a,m)≈μ(a+rl,m)(2)
MOS approximation:
(15)MOS(d)≈MOS(d+1).On the one hand, we assume that for anyl=1,2,…,N value that μ(a,m)≈μ(a+rl,m) (14). It is known that (a,d,m>n) states are active, and being μ(a,m)≈μ(a+rn,m), we could consider that (a+rn,d,m>n) states are active.On the other hand, we assumeMOS(d)≈MOS(d+1) (15), since QoE degradation in a higher delay level is negligible in the millisecond scale that delay evolves. Taking into account the approximation about MOS function, being (a,d,m>n) states active, we can suppose that (a,d+1,m>n) states are also active. This way, combining approximations (14) and (15), we consider that (a+rn,d+1,m>n) states are active.Then, suppose that the aforementionedμ and MOS approximations cause Rad=∑mqmR(a,d,m)≈∑mqmR(a+rl,d+1,m) and Wad=∑mqmW(a,d,m)≈∑mqmW(a+rl,d+1,m). So, using these reward and work approximations, the first simplified expression for the Whittle index formulation (10) is
(16)v~(a,d,n)*=βμ(a,n)(MOS^(d+1)-Rad)1-βμ(a,n)Wad.In such a way, applying the previous simplifications, we obtain the Whittle index approximation presented in Proposition4, what we call attained service PI MOS (ASPIM). The methodology to compute the closed-form expression of this QoE-aware Whittle-based index is presented in the Appendix.Proposition 4.
The formulation of the ASPIM index is:(17)ASPIM=v~(a,d,n)*=lim⁡β→1βμ(a,n)·MOS^(d+1)(1-β)1-β+β∑m>nqm(μ(a,m)-μ(a,n)).If we analyze expression (17), we observe that this QoE-aware and channel-aware Whittle-based ASPIM index is implementable. This proposal depends on QoE, size, and channel properties. Moreover, the obtained index for the best channel condition is equal to the original Whittle index (13), since the summation in the denominator becomes null. However, as shown in (18), when channel state is not the best the index value goes to zero:
(18)ASPIMn≠N=lim⁡β→1μ(a,n)·MOS^(d+1)(1-β)∑m>nqm(μ(a,m)-μ(a,n))⟶0.Hence, a user in its best channel state with nonnull instantaneous normalized MOS function has priority over a user which is not in its best channel condition. Thus, we summarize the proposed QoE-aware Whittle-based ASPIM index rule in Definition5.Definition 5.
The ASPIM index rule consists of: at every decision slot,(i)
serving the user in its best channel condition with nonnull instantaneous normalized MOS function with the highest value ofμ(a,N)·MOS^(d+1);(ii)
if there is no user in its best channel condition with nonnull instantaneous normalized MOS function, serving the user with the highest value of(μ(a,n)·MOS^(d+1))/∑m>nqm(μ(a,m)-μ(a,n)) (using (18) normalized by (1-β)).So, if channel condition is not the best, the index value is the ratio of the multiplication between the actual service completion and the instantaneous normalized MOS function respect to the expected potential improvement of the completion probability.Therefore, in the context of time-varying channels aimed at maximizing average MOS, we have proposed ASPIM scheduling algorithm, which is a simple and tractable QoE-aware, channel-aware, and size-based index rule. Nevertheless, in order to verify the correct behavior of this heuristic, we analyze its performance in the subsequent section.


## 5. Performance Evaluation

In this section, we evaluate the performance of the proposed ASPIM index rule (see proposal in Definition5). To that end, we compare the QoE performance of our heuristic with the one achieved by well-known scheduling strategies in several simulation scenarios. Below we provide a brief description of the scheduling priority rules used in our experiments.(i)
Max rate (MR): this opportunistic policy consists in serving the user with the best channel capacity.(ii)
Proportional fair (PF): this channel-aware discipline serves the user with the highest ratio between instantaneous transmission rate and current attained throughput.(iii)
Cost and attained service dependantμ (cASμ): although it might seem counter-intuitive, minimizing mean delay does not result in maximizing average MOS [24]. This way, in order to compare the performance with delay-aware solutions, we propose to adapt the classical cμ rule aimed at minimizing mean delay [25] to attained service. This adaptation consists of serving the user with the highest value of cμ(a,n), being c the waiting cost paid for every slot while flow transmission is uncompleted.(iv)
GM [16]: this QoE-aware pioneering index rule, originally proposed for constant channel capacity, chooses the user with the highest value of μ(a,n)·MOS^(d+1).In case of ties, we use random tie-breaking rule.Concerning traffic flows in the system, we use the Pareto size distribution given in (1) in simulations, whose α and γ parameters determine mean flow size. We consider typical mean sizes used in wireless networks: 0.5 Mb (small traffic: a web page, email), 5 Mb (medium traffic: a PDF document, a picture, etc.), and 50 Mb (MP3 audio, a group of pictures from a video sequence, etc.). User traffic flows arrive to the scheduler according to a Poisson process with λ rate.Referring to network characteristics, we use transmission rates that depend on CQIs used in real wireless networks, which are associated with 4G modulation and coding schemes. We show the mapping between CQI values and transmission rates in Table1. The presented values are adapted and extrapolated from [26] for a cell of 20 MHz.Table 1
CQIs and corresponding transmission rates (Mbps).
CQI
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
s
0
4.2
6.72
8.4
11.256
16.8
21.84
25.2
26.88
33.6
44.68
50.4
53.76
67.2
75.8
80.64Moreover, we consider different network loads in order to analyze the behavior of scheduling strategies under different network conditions. Flow arrival rate determines network load,ρ, where ρ=λ·(E[X]/sN). We consider seven network states in this study: low load (ρ=0.25,0.375), medium load (ρ=0.5,0.75), and high load (ρ=0.85,0.9,0.95).Regarding QoE aspects, as typically utilized in perceived quality studies [27], we use a log-shaped MOS function for QoE degradation, presented in (19). This way, employing the MOS curve modelling provided in [21] as starting point and taking into account the flow size, the network type and the user expertise, we have suitably defined dmin⁡ and dmax⁡ delay thresholds used in simulations, and consequently c1 and c2 MOS coefficients. Consider
(19)MOS=c1-c2·log⁡⁡(d).We have implemented the whole network environment for the scheduling of network resources in MATLAB. In relation to the performed experiments, for each scenario, for each combination of scheduling discipline and network load, we have carried out a set of 10 rounds of 10000 s length simulations. These rounds differ in the randomly pregenerated traces (input vector) of sizes and arrivals. Thus, not only the overall average MOS values will be provided but we will also include their 95% confidence intervals.Besides, in order to guarantee that the obtained performance results are generally valid, we will analyze different simulation scenarios. We have chosen five relevant settings, which differ in QoE, size, channel, or/and cost characteristics. The parameters of these scenarios are summarized in Table2. Note that in the first four scenarios, we restrict ourselves to the case of two channel conditions so as to get fundamental performance results.Table 2
Parameter set in the experimental scenarios.
Scenario
MOS
Size
Channel
c
1
d
min
⁡
=
0.5 s, dmax⁡=10 s
α
=
1.5, γ=4·10-7
CQI = {3, 5}
1
c
1
=
4.07, c2=1.33
E
[
X
]
=
5 Mbit
q
2
=
0.5
2
d
min
⁡
=
0.4 s, dmax⁡=0.6 s
α
=
1.5, γ=4·10-7
CQI = {3, 5}
1
c
1
=
-
4.03, c2=9.86
E
[
X
]
=
5 Mbit
q
2
=
0.5
3
d
min
⁡
=
0.5 s, dmax⁡=10 s
α
=
1.5, γ=4·10-7
CQI = {3, 9}
1
c
1
=
4.07, c2=1.33
E
[
X
]
=
5 Mbit
q
2
=
0.5
4
d
min
⁡
=
0.05 s, dmax⁡=1 s
α
=
1.5, γ=4·10-6
CQI = {3, 5}
1
c
1
=
1, c2=1.33
E
[
X
]
=
0.5 Mbit
q
2
=
0.5
5
d
mi
n
1
=
0.5 s, dmax1=5 s
α
1
=
1.5, γ1=4·10-7
see Table3
{
5,1
}
c
1
1
=
3.79, c21=1.73
E
[
X
1
]
=
5 Mbit
d
mi
n
2
=
5 s, dmax2=30 s
α
2
=
1.5, γ2=4·10-8
c
1
2
=
8.59, c22=2.23
E
[
X
2
]
=
50 MbitWe have carefully selected the scenarios used in simulations. First of all, we define a setting (Scenario 1) in which the values of all the parameters are typical or standard; the values of these parameters introduce low error in the approximations considered for our ASPIM scheduler proposal. Then, in order to evaluate worst performance cases for ASPIM, we have chosen three scenarios (Scenario 2, Scenario 3, and Scenario 4) in which the error of the approximation used for ASPIM is high due to a different factor (gradient of MOS, rate and size, resp.).Apart from that, it is worth mentioning that for obtaining the ASPIM index rule that we have considered an isolated MDP model per user. Hence, the achieved solution is based on a unique user type or single class. However, it would be interesting to examine the proposal when there are different types of users, which is analyzed in Scenario 5. Furthermore, this last setting resembles a real 4G wireless network. Note that, even though this technology allows the simultaneous transmission of multiple flows per TTI, we assume that a single flow transmits in each TTI.Next we describe the results we have achieved in the aforementioned scenarios.


### 5.1. Scenario 1: Basic Case

In this first family of simulations we consider a basic or typical scenario, which takes into account the equiprobable channel case and medium-sized self-similar flows. Figure5 shows average MOS results for this setting employing different scheduling policies and network loads. The main conclusion achieved from these results is that ASPIM outperforms the rest of disciplines, which is better than the QoE-aware GM rule proposed for constant channel capacity. Moreover, contrary to the other policies, it holds average MOS values in the same level for all ρ. Referring to the traditional channel-aware disciplines, both PF and MR give bad behavior comparing to the rest of scheduling strategies.Figure 5
E[MOS] for the basic scenario.In order to justify the previous performance results we now analyze delay and MOS statistics for the highest network load considered. In such a way, as regarded in Figure6(a) for the Probability Density Function (PDF) of delay, we highlight the probability mass or peak that appears for some scheduling disciplines before the low delay threshold (0.5 s). The height and weight of a peak belonging to a policy have direct impact on its average MOS. ASPIM discipline shows the highest and the widest peak among the presented ones, followed by GM but with a notable difference. Therefore, our novel QoE-aware policy seems to try to finish the transmission of most of the flows before dmin⁡, which is reflected in the aforementioned peaks. Note that the cASμ rule aimed at minimizing delay behaves in different manner, which shows more fairness among delays in the range shown compared to the QoE-aware ones.Statistics in Scenario 1: delay PDF (a), MOS PDF (b), and MOS CCDF (c).
(a)
(b)
(c)What is previously concluded about delay statistics has a direct reflection on QoE statistics. This way, as can be seen in the PDF of MOS presented in Figure6(b), if the objective is to maximize average QoE it is important, on the one hand, to be the height and the weight of the peaks around the maximum MOS high, and on the other hand, to be the value around the minimum MOS low. ASPIM fulfils these positive properties, and consequently, it is the best in average QoE terms. Besides, if we analyze the Complementary Cumulative Distribution Function (CCDF) of MOS provided in Figure 6(c), we observe that ASPIM gives the highest probability of being MOS value higher. Considering, for example, a QoE satisfaction threshold, satisth, of MOS=4, in this point ASPIM improves GM a 10%, whereas the deterioration of MR and PF is higher than a 50%.


### 5.2. Scenario 2: High Demanding Users Case

This scenario reflects the case of users with low tolerance to delay, in which the MOS function in the QoE degradation range shows a great slope. Thus, in this setting the error of the MOS approximation (15) is high since the gradient of MOS among consecutive delay levels becomes significant. In Figure 7 we compare the results of this second scenario and the basic scenario. From the illustrated graph we could say that our QoE-aware proposal adapts to any MOS function. This is, whereas for the rest of disciplines (excluding GM) the negative effects due to the abrupt degradation of the QoE utility function are appreciable, ASPIM manages to avoid this, adapting to any MOS function.Figure 7
E[MOS] for abrupt MOS scenario.


### 5.3. Scenario 3: Higher Error Case due to Rate

During the simplification process carried out for obtaining the ASPIM index rule in Section4.1, the μ approximation used (see (14)) introduces a rate dependant error. Note that increasing the rate in the good channel state makes bigger the error of the μ approximation used. Therefore, we will analyze the obtained performance for our new QoE-aware proposal under this higher error due to rate. In the previous scenarios we have assumed that the transmission rate in the good channel condition is twice the rate in the bad channel state. In this setting we consider that the transmission rate in the good channel is four times the transmission rate in the bad channel. This way, according to the QoE results presented in Figure 8, in spite of a higher error caused by rate, ASPIM is superior to the rest of policies.Figure 8
E[MOS] under higher error due to rate.


### 5.4. Scenario 4: High Error Case due to Size

Besides, it is known that for a distribution with a decreasing hazard rate such as Pareto that the gradient of completion probability increases as long as the mean size decreases, which makes the error of theμ approximation (14) for the ASPIM rule achievement bigger. In this way, in this scenario we verify that ASPIM gives satisfactory behavior under a more notable error due to size, where we consider a mean size ten times lower. We conclude from Figure 9 that even under a high error caused by size that ASPIM shows good performance, being the best alternative.Figure 9
E[MOS] under high error due to size.


### 5.5. Scenario 5: A Real Network Case

This last scenario reflects a real 4G wireless network context, but with the simplification that a single user transmits in each TTI. To that end, we use CQI traces obtained from a system-level radio access simulator [28], considering users moving at 5 km/h in the cell. For this setting with dynamic users, we define channel propagation by the Extended Pedestrian A model. As in current 4G networks, we consider 16 channel conditions. The state probabilities of these channel states are collected in Table 3, which are decreasing with the improvement of channel quality. In this scenario we consider two classes of users, which differ in size, QoE and cost. The mean size in class 2 is ten times bigger than in class 1, and we give priority to the small-sized class by having bigger cost and QoE requirement.Table 3
Channel state probabilities in Scenario 5.
CQI
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
q
0.28
0.12
0.09
0.08
0.08
0.08
0.07
0.06
0.05
0.04
0.03
0.01
0.009
0.0005
0.0003
0.0002Under this mixture of classes and realistic channel models, in Figure10(a) we observe that scheduling strategies show the same tendency as in the previous scenarios. It is remarkable that the difference between ASPIM and GM is bigger, especially for high ρ values. Moreover, under the highest network load ASPIM achieves MOS values above QoE satisfaction threshold (MOS=4). Furthermore, average MOS values for MR and PF dramatically decrease comparing with the previous settings.E
[
MOS
] for 4G scenario: aggregate (a), class 1 (b), class 2 (c).
(a)
(b)
(c)Concerning class behaviour, we show per-class results in Figures10(b) and 10(c). As can be concluded from the presented results, QoE-aware policies guarantee fairness among classes, and ASPIM is the best choice also inside class. Aside from that, for the cost dependant cASμ discipline the deterioration of the class with no priority is significant.


## 5.1. Scenario 1: Basic Case

In this first family of simulations we consider a basic or typical scenario, which takes into account the equiprobable channel case and medium-sized self-similar flows. Figure5 shows average MOS results for this setting employing different scheduling policies and network loads. The main conclusion achieved from these results is that ASPIM outperforms the rest of disciplines, which is better than the QoE-aware GM rule proposed for constant channel capacity. Moreover, contrary to the other policies, it holds average MOS values in the same level for all ρ. Referring to the traditional channel-aware disciplines, both PF and MR give bad behavior comparing to the rest of scheduling strategies.Figure 5
E[MOS] for the basic scenario.In order to justify the previous performance results we now analyze delay and MOS statistics for the highest network load considered. In such a way, as regarded in Figure6(a) for the Probability Density Function (PDF) of delay, we highlight the probability mass or peak that appears for some scheduling disciplines before the low delay threshold (0.5 s). The height and weight of a peak belonging to a policy have direct impact on its average MOS. ASPIM discipline shows the highest and the widest peak among the presented ones, followed by GM but with a notable difference. Therefore, our novel QoE-aware policy seems to try to finish the transmission of most of the flows before dmin⁡, which is reflected in the aforementioned peaks. Note that the cASμ rule aimed at minimizing delay behaves in different manner, which shows more fairness among delays in the range shown compared to the QoE-aware ones.Statistics in Scenario 1: delay PDF (a), MOS PDF (b), and MOS CCDF (c).
(a)
(b)
(c)What is previously concluded about delay statistics has a direct reflection on QoE statistics. This way, as can be seen in the PDF of MOS presented in Figure6(b), if the objective is to maximize average QoE it is important, on the one hand, to be the height and the weight of the peaks around the maximum MOS high, and on the other hand, to be the value around the minimum MOS low. ASPIM fulfils these positive properties, and consequently, it is the best in average QoE terms. Besides, if we analyze the Complementary Cumulative Distribution Function (CCDF) of MOS provided in Figure 6(c), we observe that ASPIM gives the highest probability of being MOS value higher. Considering, for example, a QoE satisfaction threshold, satisth, of MOS=4, in this point ASPIM improves GM a 10%, whereas the deterioration of MR and PF is higher than a 50%.


## 5.2. Scenario 2: High Demanding Users Case

This scenario reflects the case of users with low tolerance to delay, in which the MOS function in the QoE degradation range shows a great slope. Thus, in this setting the error of the MOS approximation (15) is high since the gradient of MOS among consecutive delay levels becomes significant. In Figure 7 we compare the results of this second scenario and the basic scenario. From the illustrated graph we could say that our QoE-aware proposal adapts to any MOS function. This is, whereas for the rest of disciplines (excluding GM) the negative effects due to the abrupt degradation of the QoE utility function are appreciable, ASPIM manages to avoid this, adapting to any MOS function.Figure 7
E[MOS] for abrupt MOS scenario.


## 5.3. Scenario 3: Higher Error Case due to Rate

During the simplification process carried out for obtaining the ASPIM index rule in Section4.1, the μ approximation used (see (14)) introduces a rate dependant error. Note that increasing the rate in the good channel state makes bigger the error of the μ approximation used. Therefore, we will analyze the obtained performance for our new QoE-aware proposal under this higher error due to rate. In the previous scenarios we have assumed that the transmission rate in the good channel condition is twice the rate in the bad channel state. In this setting we consider that the transmission rate in the good channel is four times the transmission rate in the bad channel. This way, according to the QoE results presented in Figure 8, in spite of a higher error caused by rate, ASPIM is superior to the rest of policies.Figure 8
E[MOS] under higher error due to rate.


## 5.4. Scenario 4: High Error Case due to Size

Besides, it is known that for a distribution with a decreasing hazard rate such as Pareto that the gradient of completion probability increases as long as the mean size decreases, which makes the error of theμ approximation (14) for the ASPIM rule achievement bigger. In this way, in this scenario we verify that ASPIM gives satisfactory behavior under a more notable error due to size, where we consider a mean size ten times lower. We conclude from Figure 9 that even under a high error caused by size that ASPIM shows good performance, being the best alternative.Figure 9
E[MOS] under high error due to size.


## 5.5. Scenario 5: A Real Network Case

This last scenario reflects a real 4G wireless network context, but with the simplification that a single user transmits in each TTI. To that end, we use CQI traces obtained from a system-level radio access simulator [28], considering users moving at 5 km/h in the cell. For this setting with dynamic users, we define channel propagation by the Extended Pedestrian A model. As in current 4G networks, we consider 16 channel conditions. The state probabilities of these channel states are collected in Table 3, which are decreasing with the improvement of channel quality. In this scenario we consider two classes of users, which differ in size, QoE and cost. The mean size in class 2 is ten times bigger than in class 1, and we give priority to the small-sized class by having bigger cost and QoE requirement.Table 3
Channel state probabilities in Scenario 5.
CQI
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
q
0.28
0.12
0.09
0.08
0.08
0.08
0.07
0.06
0.05
0.04
0.03
0.01
0.009
0.0005
0.0003
0.0002Under this mixture of classes and realistic channel models, in Figure10(a) we observe that scheduling strategies show the same tendency as in the previous scenarios. It is remarkable that the difference between ASPIM and GM is bigger, especially for high ρ values. Moreover, under the highest network load ASPIM achieves MOS values above QoE satisfaction threshold (MOS=4). Furthermore, average MOS values for MR and PF dramatically decrease comparing with the previous settings.E
[
MOS
] for 4G scenario: aggregate (a), class 1 (b), class 2 (c).
(a)
(b)
(c)Concerning class behaviour, we show per-class results in Figures10(b) and 10(c). As can be concluded from the presented results, QoE-aware policies guarantee fairness among classes, and ASPIM is the best choice also inside class. Aside from that, for the cost dependant cASμ discipline the deterioration of the class with no priority is significant.


## 6. Conclusions

This paper goes into detail about obtaining a scheduling algorithm aimed at maximizing users’ perception of quality in channels with time-varying capacity. Furthermore, this work considers traffic flows with nonexponential size distributions, contrary to previous approaches that assume memoryless distributions.As first contribution, we provide a MDP-based model for the scheduling problem presented. This model combines QoE-awareness, size-awareness and channel-awareness. Nevertheless, since this model can not be solved either analytically or computationally, from the proposed MDP model we derive a simple, tractable and implementable well-performing scheduling index rule by applying a methodology based on Whittle approach.The proposed scheduling solution still combines QoE-, size- and channel-awareness. Our scheduling proposal gives priority to users in their best channel condition that are out of the service unavoidable range. As verified from simulation results, the proposed scheduling strategy shows suitable subjective quality performance in a wide range of scenarios, including the case of a simplified 4G network with heterogeneous users.Therefore, the results of this work present a relevant mathematical basis for developing scheduling algorithms aimed at maximizing QoE in current and future networks with time-varying channel capacity. This way, the proposed channel-aware ASPIM discipline will be useful for network operators in order to guarantee to their customers service satisfaction in time-varying wireless networks. Apart from that, the approximations and simplifications used in Whittle method are applicable to any area when the gradient of system components is considerably small among consecutive decision slots.As future research, we will extend our work to a multiuser approach, considering the simultaneous transmission of multiple flows per TTI.



---

*Source: 647157-2014-08-05.xml*