Deep Reinforcement Learning For Wireless Scheduling with Multiclass Services

In this paper , we investigate the problem of scheduling and resource allocation over a time varying set of clients with heterogeneous demands . This problem appears when service providers need to serve traffic generated by users with different classes of requirements . We thus have to allocate bandwidth resources over time to efficiently satisfy these demands within a limited time horizon . This is a highly intricate problem and solutions may involve tools stemming from diverse fields like combinatorics and optimization . Recent work has successfully proposed Deep Reinforcement Learning ( DRL ) solutions , although not yet for heterogeneous user traffic . We propose a deep deterministic policy gradient algorithm combining state of the art techniques , namely Distributional RL and Deep Sets , to train a model for heterogeneous traffic scheduling . We test on diverse number scenarios with different time dependence dynamics , users ’ requirements , and resources available , demonstrating consistent results . We evaluate the algorithm on a wireless communication setting and show significant gains against state-of-theart conventional algorithms from combinatorics and optimization ( e.g . Knapsack , Integer Linear Programming , Frank-Wolfe ) . 1 INTRODUCTION . User scheduling ( i.e. , which user to be served when ) and associated resource allocation ( i.e. , which and how many resources should be assigned to scheduled users ) are two long-standing fundamental problems in communications , which have recently attracted vivid attention in the context of next generation communication systems ( 5G and beyond ) . The main reason is the heterogeneity in users ’ traffic and the diverse Quality of Service ( QoS ) requirements required by the users . The goal of this paper is to design a scheduler and resource assigner , which takes as inputs the specific constraints of the traffic/service class each user belongs in order to maximize the number of satisfied users . This problem is hard to solve since we have at least two main technical challenges : ( i ) except for some special cases , there is no simple closed-form expression for the problem and a fortiori for its solution ; ( ii ) the problem solving algorithm has to be scalable with the number of users . Current solutions rely on combinatorial approaches or suboptimal solutions , which seem to work satisfactorily in specific scenarios , failing though to perform well when the number of active users is large . This motivates the quest for alternative solutions ; we propose to resort to Deep Reinforcement Learning ( DRL ) to tackle this problem . In the context of DRL , we propose to combine together several ingredients in order to solve the aforementioned challgening problem . In particular , we leverage on the theory of Deep Sets to design permutation equivariant and invariant models , which solves the scalability issue , i.e. , the number of users can be increased without having to increase the number of parameters . We also stabilize the learning process by adding in a new way the distributional dimension marrying it with Dueling Networks to ” center the losses ” . Finally , we compare the proposed DRL-based algorithm with conventional solutions based on combinatorial or suboptimal optimization approaches . Our experiments and simulation results clearly show that our DRL method significanlty outperforms conventional state-of-the-art algorithms . 2 RELATED WORK . The scheduling problem is a well known problem appearing in various fields and as technologies progress and more people want to take advantage of the new services , how to schedule them in an efficient way becomes more intricate . This is exactly the case in wireless communication systems . Researchers are resorting to new methods , such as deep reinforcement learning , which have shown impressive results Mnih et al . ( 2015 ) ; Silver et al . ( 2016 ) . For example in ( Chinchali et al. , 2018 ) they perform scheduling on a cellular level using Deep Reinforcement learning ( DRL ) . Also ideas using DRL in a distributed way to perform dynamic power allocation has appeared in ( Naparstek & Cohen , 2018 ; Nasir & Guo , 2019 ) . Nevertheless , to the best of our knowledge , the problem of scheduling on traffic of users with heterogeneous performance requirements has not been appropriately addressed . To solve this hard problem , one can resort to distributional Reinforcement Learning researched in Jaquette ( 1973 ) and followed by ( Dabney et al. , 2018a ; b ) in order to have richer representations of the environment and obtain better solutions . Also techniques like noisy network for better explorations ( Fortunato et al. , 2018 ) or architectures like dueling networks ( Wang et al. , 2016 ) have greatly improved stability of the trained models . Finally ideas ( Zaheer et al. , 2017 ) managed to simplify and improve neural network models when permutation invariance properties apply . We combine those ideas with a deep deterministic policy gradient method ( Lillicrap et al. , 2016 ) to reach a very efficient scheduling algorithm . 3 THE SCHEDULING AND RESOURCE ALLOCATION PROBLEM . 3.1 THE PROBLEM . The problem we consider here involves a set of randomly arriving users that communicate wirelessly with a base station ( service provider ) ; users require that their traffic is served according to the quality of service ( QoS ) requirements imposed by the service class they belong to . We consider the case where users belong to different service classes with heterogeneous requirements . Each class specifies the amount of data to be delivered , the maximum tolerable latency , and the “ importance/priority ” of the user . A centralized scheduler ( at the base station ) at each time step takes as input this time varying set of users belonging to different service classes and has to decide how to allocate its limited resources per time step in order to maximize the long-term ” importance ” weighted sum of satisfied users . A user is considered to be satisfied whenever it successfully received its data within the maximum tolerable latency specified by its service class . The hard problem of scheduling and resource allocation - which is combinatorial by nature - is exacerbated by the wireless communication , which in turn brings additional uncertainty due to time-varying random connection quality . The scheduler that assigns resources does not exclude the possibility of a bad connection ( low channel quality ) which renders data transmission unsuccessful . In order to mitigate that effect , some protocols make use of channel state information ( CSI ) to the transmitter , i.e. , the base station/scheduler knows in advance the channel quality and adapts the allocated resources to the instantaneous channel conditions . We consider here two extreme cases of channel knowledge : ( i ) full-CSI , in which perfect ( instantaneous , error free ) CSI is provided to the scheduler enabling accurate estimation of exact resources each user needs ; and ( ii ) no-CSI , in which the scheduler is agnostic to the channel quality . In case of unsuccessful/erroneous data reception , we employ a simple retransmission protocol ( HARQ-type I ) . A widely used way to model the channel connection dynamics is to make the wireless channel quality depend on the distance of the user from the base station and evolve in a Markovian way from the quality on the channel realization of the previous time step . The mathematical description of the traffic generator model and the channel dynamics is provided in the appendix A . To better understand the problem , we draw the following analogy . Imagine a server having a water pitcher that is full at every time step and has to distribute it across a set of people . Every person has a glass and leaves satisfied only if its glass is filled ( or overfilled ) at any time instant prior to certain maximum waiting time . As mentioned before , in this work we consider a retransmission protocol ( HARQ-type I ) , which in our analogy means that the server can not fill a glass using multiple trials ; if a glass is not filled completely at a time step , then it will be emptied and the server has to retry . The wireless communication setting brings the additional complication that the sizes of the glasses are not actually fixed but fluctuate ( due to the randomness of the connection quality of each user ) . In the full-CSI case , the server could know at every time step the size of glasses and therefore the exact amount of resources required . On the other hand , with no-CSI , the server can only roughly estimate the size , mainly using the amount of data requested and the distance between user and base station . The problem can be modeled as a Markov Decision Process ( MDP ) ( Bellman , 1957 ) ( S , A , R , P , γ ) 1 , where S is the state space of the environment ( described in detail in appendix A.2 ) , and A is the action space ( in our case the set of all feasible allocations ) . After action at ∈ A at state st ∈ S a reward rt ∼ R ( ·|st , at ) is obtained and the next state follows the probability st+1 ∼ P ( ·|st , at ) . The discount factor is γ ∈ [ 0 , 1 ) . Under a fixed policy π : S → A the return is a random variable defined as Zπt = ∑∞ i=0 γ t+irt+i representing the discounted sum of rewards when a trajectory of states is taken following the policy π . An agent ( scheduler ) ideally aims to find the optimal policy π ? maximizing the mean reward Eπ [ Zπ ] . Being more rigorous , only in the full-CSI case the states st are fully observed , whereas for no-CSI , the channel qualities are unknown to the agent and the observation ot ⊂ st is a part of the state , leading to a Partially Observable MDP ( POMDP ) ( Åström , 1965 ) . A way to reduce POMDP to MDP is by substituting the states with the ” belief ” of the states ( Kaelbling et al. , 1998 ) of the value of st. Another way is using the complete history { o0 , a0 , o1 , a1 , · · · , at−1 , ot−1 } , which fortunately in our case it works since only the most recent part is relevant , i.e. , the one representing if and how many resources have been previously allocated to only the currently active users . 3.2 THE DEEP REINFORCEMENT LEARNING APPROACH . Deep reinforcement learning ( DRL ) has shown impressive results in many problems modeled as MDP , but mainly in cases where the environment is close to deterministic due to game rules ( Atari , Chess , Go ( Mnih et al. , 2015 ; Silver et al. , 2017 ; 2016 ) ) or physical laws ( robotics and physics tasks ( Kober et al. , 2013 ; Lillicrap et al. , 2015 ) ) . A very relevant question to ask is whether we can develop a DRL algorithm coping successfully with environments exhibiting high variance randomness as in our case due to the channel dynamics and heterogeneous traffic . Existing applications of DRL to problems with similar properties in other fields are encouraging ( trading , pricing , vehicle routing ( Nazari et al. , 2018 ; Charpentier et al. , 2020 ) ) . 3.2.1 POLICY NETWORK . Our objective is to a scheduler that can handle a large number of users K , say K = 100 , in which case the action space becomes infeasibly large for a traditional Deep Q-learning Network approach . For that , we propose to employ a deep deterministic policy gradient ( DDPG ) method ( Lillicrap et al. , 2016 ) , with which we aim at training a policy πθ : S → A modelled as a Neural Network ( NN ) with parameters θ . Moreover , our method should work in both full-CSI and no-CSI cases with minor , if any , modification . With full-CSI the exact amount of required resources ( bandwidth ) per user is known , so the ( discrete ) action is just to select the subset of user to satisfy , while for noCSI , it is continuous since on top of selecting the users , the scheduler has to decide on the portion of resources each user takes . For no-CSI those portion are exactly the output of πθ but for fullCSI we do a continuous relaxation2 and the output provides the value ( related to importance ) per resources ; that way , a user ranking is obtained , which allows the scheduler to proceed sequentially : the scheduler serves/satisfies as many of the most “ valuable ” ( highest rank ) users as possible subject to available resources . This discrepancy in the output process is the only difference in the model between full-CSI and no-CSI . Setting Zπ ( st , at ) = rt + Zπt+1 with rt ∼ R ( ·|st , at ) being the return if at t the action at is taken followed by policy π and Qπ ( st , at ) = E [ Zπ ( st , at ) ] be the expected return conditioned on the action at st is at then the objective of the agent to maximize is J ( θ ) = Est0∼pt0 [ Q πθst0 , πθ ( st0 ) ) ] with pt0 being the probability of the initial state st0 at time t0 . The gradient can be written ( Silver et al. , 2014 ) : ∇θJ ( θ ) = Est0∼pt0 , s∼ρπθst0 [ ∇θπθ ( s ) ∇aQ πθ ( s , a ) |a = πθ ( s ) ] ( 1 ) 1The only discrepancy is that the scheduler aims to ideally maximize the sum of rewards , i.e. , for γ = 1 , and not the discounted one . 2The continuous relaxation is also mandatory for a DDPG approach to work so that the gradients can pass from the value network . with ρπθst0 the discounted state ( improper ) distribution defined as ρ πθ st0 ( s ) = ∑∞ i=0 γ i P ( st+i = s|st0 , πθ ) . In practice ρπθst0 is approximated by the ( proper ) distribution % πθ st0 ( s ) : = ∑∞ i=0P ( st+i = s|st0 , πθ ) . To compute the gradient , the function Qπθ ( s , a ) is needed which is approximated by another NN Qψ ( s , a ) , named value network , described in the next subsection . We now explain the architecture of the model πθ . The policy falls in a category of permutation equivariant functions meaning that permuting the users should only result in permuting likewise the resource allocation . In ( Zaheer et al. , 2017 ) necessary and sufficient conditions are shown for permutation equivariance in neural networks ; we adopt their model with minor changes . At first the characteristics xi ∈ RNx , i ∈ { 1 , · · ·K } of each ( active ) user are processed individually by the same function φuser : RNx → RHx modeled as a two layer fully connected network . Then all those features per user are aggregated with the permutation equivariant fσ : RK×H → RK×H ′ of H/H ′ input/output channels : fσ ( x ) = σ ( xΛ + 11 ᵀxΓ ) , 1 = [ 1 , · · · , 1 ] ∈ RK , Λ , Γ ∈ RD×D ′ and σ ( · ) an element wise non-linear function . We stack two of those , one frelu : RK×Hx → R K×H′x with σ ( ) being the relu ( x ) = max ( 0 , x ) and a second flinear : RK×H ′ x → RK×1 without any non-linearity σ ( ) . This structure on top of preserving the desirable permutation equivariance property it also brings a significant reduction of parameters since an increase of the number of users does not necessitate additional parameters with bigger network prone to overfitting traps . Before the final non-linearity , which is a smooth approximation of ReLU , namely softplus ( x ) = log ( 1 + ex ) , guaranteeing the output is positive , there is a critical normalization step x → x−E [ x ] ||x||2 with || · ||2 being the ` 2 norm . To better understand the criticality of that step , consider the case of full-CSI where the output denotes how valuable each user is . Without the normalization step , the value network perceives that the higher the value assigned to a user , the more probable is to get resources , be satisfied and take reward , leading to a pointless attempt of increasing every user ’ s value . However , by subtracting the mean value , whenever the value of a user increases , the value of the rest decreases , giving hence the sense of the total resources being limited . In the case of no-CSI , there is an additional benefit . Here there is an extra final operation , i.e. , x→ x||x||1 , see Figure 1 , so as to signify portions ( of the total bandwidth ) adding up to 1 . Having done the normalization step previously ( dividing by ||x||2 ) , helps keeping the denominator ||x||1 stable . A final note regards the exploration . The output has to satisfy properties ( like positivity and/or adding to 1 ) which makes the common approach of adding noise on the actions cumbersome . An easy way out is through noisy networks ( Fortunato et al. , 2018 ) , which introduce noise to the weights of a layer , resulting to changed decision of the policy network . The original approach considers the variance of the noise to be learnable ; we keep it constant though since it provides better results . The noise is added at φusers parameters , resulting to altered output features per user and therefore different allocations .

Summary: Basically, it seems that the proposed method is interesting and meaningful. The scheduling problem in this paper is based on the analogy to a server having a water pitcher, and the deep reinforcement learning approach for the scheduling problem has been designed. However, the scheduling problem in wireless networks is a very famous issue. Of course, applying DRF to it is quite interesting. However, the authors need to describe the conventional well-known scheduling algorithms and compare them with the proposed scheme (now, the current paper only focuses on applying the DRF to the scheduling and evaluating its performance in aspects of an optimization problem.). Further,  typically, in scheduling problems, efficiency (total data rate) and fairness are the key factors and it is needed to describe the relationship between these conventional performance metrics and the satisfaction probability.