Deep Reinforcement Learning For Wireless Scheduling with Multiclass Services
Authors: Apostolos Avranas
Marios Kountouris
Philippe Ciblat
Venue:  (2021)

Abstract: In this paper, we investigate the problem of scheduling and resource allocation over a time varying set of clients with heterogeneous demands. This problem appears when service providers need to serve traffic generated by users with different classes of requirements. We thus have to allocate bandwidth resources over time to efficiently satisfy these demands within a limited time horizon. This is a highly intricate problem and solutions may involve tools stemming from diverse fields like combinatorics and optimization. Recent work has successfully proposed Deep Reinforcement Learning (DRL) solutions, although not yet for heterogeneous user traffic. We propose a deep deterministic policy gradient algorithm combining state of the art techniques, namely Distributional RL and Deep Sets, to train a model for heterogeneous traffic scheduling. We test on diverse number scenarios with different time dependence dynamics, users’ requirements, and resources available, demonstrating consistent results. We evaluate the algorithm on a wireless communication setting and show significant gains against state-of-the-art conventional algorithms from combinatorics and optimization (e.g. Knapsack, Integer Linear Programming, Frank-Wolfe).

Under review as a conference paper at ICLR 2021
DEEP REINFORCEMENT LEARNING FOR WIRELESS SCHEDULING WITH MULTICLASS SERVICES
Anonymous authors Paper under double-bling review
ABSTRACT
In this paper, we investigate the problem of scheduling and resource allocation over a time varying set of clients with heterogeneous demands. This problem appears when service providers need to serve traffic generated by users with dif- ferent classes of requirements. We thus have to allocate bandwidth resources over time to efficiently satisfy these demands within a limited time horizon. This is a highly intricate problem and solutions may involve tools stemming from diverse fields like combinatorics and optimization. Recent work has successfully pro- posed Deep Reinforcement Learning (DRL) solutions, although not yet for hetero- geneous user traffic. We propose deep deterministic policy gradient algorithm combining state of the art techniques, namely Distributional RL and Deep Sets, train a model for heterogeneous traffic scheduling. We test diverse number scenarios with different time dependence dynamics, users' requirements, and re- sources available, demonstrating consistent results. We evaluate the algorithm on a wireless communication setting and show significant gains against state-of-the- art conventional algorithms from combinatorics and optimization (e.g. Knapsack, Integer Linear Programming, Frank- Wolfe).
INTRODUCTION
User scheduling (i.e., which user to served when) and associated resource allocation (i.e., which and how many resources should be assigned to scheduled users) two long-standing fundamental problems in communications, which have recently attracted vivid attention in the context next generation communication systems (5G and beyond). The main reason is the heterogeneity in users' traffic and the diverse Quality of Service (QoS) requirements required by the users. The goal of this paper to design scheduler and resource assigner, which takes inputs the specific constraints of the traffic/service class each user belongs in order to maximize the number of satisfied users.
This problem is hard to solve since have at least two main technical challenges: (i) except for some special cases, there is no simple closed-form expression for the problem and a fortiori for its solution; (ii) the problem solving algorithm has be scalable with the number of users. Current solutions rely on combinatorial approaches or suboptimal solutions, which seem to work satisfactorily in specific scenarios, failing though to perform well when the number of active users large. This motivates the quest for alternative solutions; we propose to resort to Deep Reinforcement Learning (DRL) to tackle this problem.
In the context of DRL,, we propose to combine together several ingredients in order to solve the aforementioned challgening problem. In particular, we leverage on the theory of Deep Sets to design permutation equivariant and invariant models, which solves the scalability issue, i.c., the number of users can be increased without having to increase the number of parameters. We also stabilize the learning process by adding in Networks to "center the losses".
way the distributional dimension marrying it with Dueling
Finally, we compare the proposed DRL-based algorithm with conventional solutions based com- binatorial or suboptimal optimization approaches. Our experiments and simulation results clearly show that our DRL method significanlty outperforms conventional otate-of-the-art algorithms.

---

Under review as a conference paper at ICLR 2021
2
RELATED WORK
The scheduling problem is a well known problem appearing in various fields and as technologies progress and more people want to take advantage of the new services, how to schedule them in an efficient way becomes more intricate. This is exactly the case in wireless communication systems. Researchers are resorting to new methods, such as deep reinforcement learning, which have shown impressive results Mnih et al.] (2015); [Silver et al. (2016). For example in (Chinchali et al., 2018) they perform scheduling on a cellular level using Deep Reinforcement learning (DRL). Also ideas using DRL in distributed way perform dynamic power allocation has appeared in (Naparstek) & Cohen 2018: Nasir & Guo 2019). Nevertheless, to the best of our knowledge, the problem of scheduling on traffic of users with heterogeneous performance requirements has not been appropri- ately addressed. To solve this hard problem, one can resort to distributional Reinforcement Learning researched in Daquette (1973) and followed by (Dabney et al. 20 b) in order to have richer rep- resentations of the environment and obtain better solutions. Also techniques like noisy network for better explorations (Fortunato et al. 2018) or architectures like dueling networks (Wang et al. 2016) have greatly improved stability of the trained models. Finally ideas (Zaheer et al.) 2017) managed to simplify and improve neural network models when permutation invariance properties apply. We combine those ideas with a deep deterministic policy gradient method (Lillicrap et al. 2016 to reach a very efficient scheduling algorithm.
3 THE SCHEDULING AND RESOURCE ALLOCATION PROBLEM
3.1 THE PROBLEM
The problem consider here involves set of randomly arriving users that communicate wire- lessly with a base station (service provider); users require that their traffic is served according to the quality of service (QoS) requirements imposed by the service class they belong to. We consider the case where users belong to different service classes with heterogeneous requirements. Each class specifies the amount of data to be delivered, the maximum tolerable latency, and the "impor- tance/priority" of the user. A centralized scheduler (at the base station) at each time step takes as input this time varying set of users belonging to different service classes and has decide how to al- locate its limited resources per time step in order to maximize the long-term "importance" weighted sum of satisfied users. A user is considered to be satisfied whenever successfully received its data within the maximum tolerable latency specified by its service class.
The hard problem of scheduling and resource allocation which is combinatorial by nature- is exacerbated by the wireless communication, which in turn brings additional uncertainty due to time-varying random connection quality. The scheduler that assigns resources does not exclude the possibility of bad connection (low channel quality) which renders data transmission unsuc- cessful. order to mitigate that effect, some protocols make use of channel state information (CSI) to the transmitter, i.e., the base station/scheduler knows in advance the channel quality and adapts the allocated resources to the instantaneous channel conditions. We consider here two extreme cases of channel knowledge: (i) full-CSI, in which perfect (instantaneous, error free) CSI is provided to the scheduler enabling accurate estimation of exact resources each user needs; and (ii) no-CSI, in which the scheduler is agnostic to the channel quality. In case of unsuccessfullerrrocooss data re- ception, we employ simple retransmission protocol (HARQ-type I). A widely used way to model the channel connection dynamics is to make the wireless channel quality depend on the distance of the user from the base station and evolve in Markovian way from the quality on the channel realization of the previous time step. The mathematical description of the traffic generator model and the channel dynamics is provided in the appendix A
To better understand the problem, we draw the following analogy. Imagine a server having a water pitcher that is full at every time step and has to distribute it across set of people. Every person has a glass and leaves satisfied only if its glass is filled (or overfilled) at any time instant prior to certain maximum waiting time. As mentioned before, in this work we consider retransmission protocol (HARQ-type I). which in our analogy means that the server cannot fill a glass using multiple trials; if a glass is not filled completely at a time step, then it will be emptied and the server has retry- The wireless communication setting brings the additional complication that the sizes of the glasses not actually fixed but fluctuate (due to the randomness of the connection quality of each user). In
2

---

Under review as a conference paper at ICLR 2021
the full-CSI case, the server could know every time step the size of glasses and therefore the exact amount of resources required. On the other hand, with no-CSI, the server can only roughly estimate the size, mainly using the amount of data requested and the distance between user and base station.
The problem can be modeled as a Markov Decision Process(MDP) (Bellman 1957) (S. A, R, P. P I [A.2). and A is the action space (in our case the set of all feasible allocations). After action a, E A state St S a reward r R(-Ist, ae) is obtained and the next state follows the probability 81411 ~ P(-Is,,a2) at). The discount factor is 7 E [0, 1). Under fixed policy : S + A the return random variable defined as Z Ei=o 7 't+s representing the discounted sum of rewards when a trajectory of states is taken following the policy π. An agent (scheduler) idcally aims to find the optimal policy x'' maximizing the mean reward FF, [. Being more rigorous, only in the full-CSI case the states 8t are fully observed, whereas for no-CSI, the channel qualities are unknown to the agent and the observation Or C 8, is a part of the state, leading to a Partially Observable MDP (POMDP)(Astöö
where S is the state space of the enviromment (described in detail in appendix
1965). A way to reduce POMDP to MDP is by substituting the states with the "belief" of the states (Kaelbling et al. 1998) of the value of st. Another way is using the complete history foo, ao 01. a1, ... 0,-1, 9t-1 J. which fortunately in case it works since only the most recent part is relevant, i.e., the one representing if and how many resources have been previously allocated to only the currently active users.
3.2 THE DEEP REINFOR LEARNING APPROACH
Deep reinforcement learning (DRL) has shown impressive results in many problems modeled as MDP, but mainly in cases where the environment is close to deterministic due to game rules (Atari, Chess, Go (Mnih et al. 2015 Silver et al. 2017 2016)) or physical laws (robotics and physics tasks (Kober et al. 2013] Lillicrap et al. 2015)). A very relevant question to ask is whether we can develop a DRL algorithm coping successfully with environments exhibiting high variance random- ness as in our case due to the channel dynamics and heterogeneous traffic. Existing applications of DRL to problems with similar properties in other fields are encouraging (trading, pricing, vehicle routing Nazari et al. 2018 Charpentier et al. 2020)).
3.2.1 POLICY NETWORK
Our objective is to a scheduler that can handle large number of users K. say K 100, in which case the action space becomes infeasibly large for a traditional Deep Q-learning Network approach. For that, we propose to employ a deep deterministic policy gradient (DDPG) method (Lillierap et al. 2016). with which we aim at training policy Ze S + AA modelled as Neural Network (NN) with parameters 0. Moreover, our method should work in both full-CSI and no-CSI cases with minor, if any, modification. With full-CSI the exact amount of required resources (bandwidth) per user is known, so the (discrete) action is just to select the subset of user to satisfy, while for no- CSI, it is continuous since on top of selecting the users, the scheduler has decide on the portion of resources each user takes. For no-CSI those portion are exactly the output of Th but for full- CSI do continuous relaxatior and the output provides the value (related importance) per resources; that way, a user ranking is obtained, which allows the scheduler to proceed sequentially: the scheduler serves/satisfies as many of the most "valuable" (highest rank) users as possible subject to available resources. This discrepancy in the output process is the only difference in the model between full-CSI and no-CSI.
r. + ZF+1 with r ~ R(-|st. az) being the return if at the action a2 is taken followed by policy π and (s) action at 8; is ae then the objective of the agent to maximize is J(O) Ie,, ~Pto 'Q*0 Sto R (2())) ))
Setting ("
E 2 (st, a:)] be the expected return conditioned on the
with Ptg being the probability of the initial state 8t at time to. The gradient can be written (Silver
ct al. 2014):
VoJ(O) = -E -Eng SPo Vo (s,a)la ms()]]
1 'The only discrepancy is that the scheduler aims ideally maximize the sum of rewards, i.c., for ry and not the discounted one. Thh continuous relaxation is also mandatory for a DDPG approach to work so that the gradients pass from the value network.
3

---

Under review as a conference paper at ICLR 2021
with Para the discounted state (improper) distribution defined Paty (s) E P(stt
s/ To). In practice Per, is approximated by the (proper) distribution 0% (s) = Ei= P(
a|8tg To). To compute the gradient, the function (s, a) is needed which is approximated by another NN (v a), named value network, described in the next subsection.
We now explain the architecture of the model The policy falls in a category of permutation equivariant functions meaning that permuting the users should only result in permuting likewise the resource allocation. In (Zaheer et al.) 2017) necessary and sufficient conditions are shown for permutation equivariance in neural networks; we adopt their model with minor changes. At first the characteristics I; R No, i {1. K} of each (active) user processed individually by the same function Quser : No + R fí modeled as a two layer fully connected network. Then all those features per user are aggregated with the permutation equivariant fo RKXH - RK*H of H/Hl' input/output channels:
fx) (xA + 11TxT), 1- (1, 1.t....../EER I A.TERDDD
and a--) an element wise non-linear function. We stack two of those, one frelu RKHH,
R Kx Ha with o() being the relu(z)- max(0, z) and second flinear Rx - RK without any non-linearity o(). This structure on top of preserving the desirable permutation equivariance property it also brings significant reduction of parameters since an increase of the number of users does not necessitate additional parameters with bigger network prone to overfitting traps.
Before the final non-linearity, which is smooth approximation of ReLU, namely softplus(1) log(1 e²). guaranteeing the output is positive, there a critical normalization step - with || 1/2 being the b2 norm. To better understand the criticality of that step, consider the case of full-CSI where the output denotes how valuable each user Without the normalization step, the value network perceives that the higher the value assigned to user, the more probable is to get resources, satisfied and take reward, leading to a pointless attempt of increasing every user's value. However, by subtracting the mean value, whenever the value of user increases, the value of the rest decreases, giving hence the sense of the total resources being limited. In the case of nS-CSI, there is an additional benefit. Here there is extra final operation, i.c., | see Figure so signify portions (of the total bandwidth) adding up to Having done the normalization step previously (dividing by ||æ])))) helps keeping the denominator ||=||] stable
15/l]]]
A final note regards the exploration. The output has satisfy properties (like positivity and/or adding to I) which makes the common approach of adding noise on the actions cumbersome. An easy way out through noisy networks (Fortunato al.) 2018), which introduce noise to the weights of a layer, resulting to changed decision of the policy network. The original approach considers the variance of the noise be learnable; we keep it constant though since provides better results. The noise is added at dusers parameters, resulting to altered output features per user and therefore different allocations.
3.2.2 VALUE NETWORK
As mentioned previously (s, is used for computing the gradient (3.2.]. however as it is intractable to compute, neural network, called value network, is employed to approximate it. The common approach is through the Bellman operator
THH Q(t a.) = IE[ at) ] P (Q (Q(s)+1,
minimize the temporal difference error, i.e., the difference between before and after applying the Bellman operator. This leads to the minimization of the loss
(2(2) = E s.g P
[(Qu(s, a) T (F
where (Irgr, ) correspond to separate networks called target policy and target value networks, respectively. used for stabilizing the learning, and are periodically (or gradually) updated as copies of the current actor and value networks.
Another approach the following: instead of only approximating the expected value of the return, approximate its distribution in (Barth-Maron et al. 2018). Algorithmically, it is impossible to
4

---

Under review as a conference paper at ICLR 2021
represent the full space of probability distributions with finite number of parameters the value neural network 2 :S x + N2 must approximate with a discrete representation the actual n Among other variations (Bellemare et al., all| 22017 Dabney et al., 201 one can choose the repre- sentation to be uniform (discrete) probability distribution supported at ((g") );, i {I. .Nq) where (2 ) is the i-th element of the output. More rigorously, the distribution that the value neu-
ral network represents, is Nq > ((2))) where d₂ is Dirac delta function Dabney al.
2018b), Minimizing the I-Wasserstein distance between this distribution and the actual one of Z can be done by minimizing the quantile regression loss
No
(1(()=
E sig PPg somppio ZNT 2n (n, Us(2 (27))))
where Ji(x)- = x 22-1 - 1 (Eco)) with 1 being the indicator function, the distributional Bellman 2NQ
operator is T 7 (st; ae) - R(st, at) + 72t ((+)) 88+1 ~ P(st, 0z) and n being the target policy network (defined in the same way as before).
(s, a), what we An important observation is that even though we approximate the distribution of =
need then end only its expected value, approximated as (s,(s, a) 1 Σ: (27);: ): There- No
fore a natural question that arises here is why using z instead of a simpler Q (s. a) approximates straight away the needed expected value. If instead of having a scheduler and its users, we consider teacher and its students, things become evident. Even though the objective of the teacher crease the mean "knowledge" of its students, using the distribution of the capacity/knowledge of the students enable for example deciding whether distribute its attention uniformly among students or focus more on a fraction of them.
in-
Even though intuitively we expected to observe gains by using the distribution, that was not the case at first. The main problem was that the distribution 2 was far away from 0 making it very difficult for for the policy network to well approximate them. One way to solve this could have been through scaling the rewards (i.e., the rewards are divided by the standard deviation of a rolling discounted sum of rewards). We came out with a new proposal and we propose to center the distribution through a dueling architecture (Wang et al. 2016). As shown in Figure I just before the output there is the dueling architecture with:
No
(2 2 To Mean + (2 Shapp
z Shape
Vi, f1,...,,,,
NQ
which effectively pushes 2 Moan to approximate QT used for training the policy. To further encourage the decomposition of the shape and the center of the distribution, we add loss term 2 Shapee around zero.
Lehape || .Shapee 12. centering
In Figure - we provide additional element to support the choice of distributional reinforcement learning. We use the traffic model described in Table Ia showing the two classes of users with different requirements. In Figure 2a (which is the mean taken over five experiments) we see that all approaches finally converge to approximately the same value; nevertheless, the combination of distributional with dueling is faster. Figures 2b and 2c focus on two (out of the five) experiments, where it is evident the advantage of the distributional approach. This approach is able to detect the existence of two difference classes with different requirements, thus gradually improving the satisfaction rate for both of them. On the other hand, trying only to learn the expected value leads to a training where one class is improved at the expense of the other.
A final remark is that the architecture should be designed in way preserving the permutation invariance. If we associate every user's characteristics with the resources given by the agent, i.c., the action corresponding to it, then permuting the users and accordingly the allocation should not influence the assessment of the success of the agent. To build such an architecture, we adopt the one of the Policy Network using ideas from (aaheer et al. 2017).
5

---

Under review as a conference paper at ICLR 2021
Input per user
Policy Network
Features per user
Value resources
rou
relu6
softpiuel x- 8/sl ||x]]
Sretu Sioscar
Portion of
no-
total resources
Value Ne Network
Input and action
Festures
2 Maan
Shape
fiincar
frata
2
Mean
Puser
Stap
Figure 1: The Neural Network architecture.
0.9
Expected
Distr&
Satisfaction Probabiltty
0.8
0.8
0.8
0.7
0.6
0.6
0.6
Class exp. 1 exp.
Class 1, exp.
0.5
Expected
Duel
0.4
Class exp. 2
0.4
Class 2,
Class 1, exp.
Distr
exp.
Class 2,
0.2
2 Millions of Samples
6
4
Millions of Samples
Millions of Samples
(b)
Figure 2: Comparison between distributional and traditional (non-distributionaa) approach. We con- ducted five experiments (with different seeds) for no-CSI using the traffic model of table Ia and a maximum number of users = 75. In the first figure we depict the average over those five experiments; in the other figures consider two specific experiments (named exp. and exp. 2).
3.3 BENCHMARKS USING CONVENTIONAL APPROACHES
3.3.1 FULL-CSI
For explanation convenience, we use once again the analogy introduced in section 3.1 Having full- CSI means that the server knows for all people (active users) at each time, the size of their glasses and their importance. If we try solve the problem myopically ignoring the impact on the future steps, we have reformulated knapsack problem with the size of the glasses being the weight of the objects, whose value the importance of its "holder". The size of the server's 's pitcher is the capacity of the knapsack that we try to fill with objects this benchmark simply by myopic Knapsack. More details are given in appendix B.2.
maximize the
of their value. We refer to
Accounting for the impact on the future is not trivial (discussed in appendix B.2.2). One way is to assume that the scheduler acts as an oracle and knows in advance for the future T time steps which users will appear and what will be their channel qualities. In that case, the problem can be written as an Integer Linear Programming and can solved by standard Branch and Cut approach. We obtain therefore upper bound which we call oracle IIP. More details in appendix B.2.2

---

Under review as a conference paper at ICLR 2021
Table Classses description for scenarios
(b) Prioritized and normal users
(a) Equal importance users
Data
Lat.
Imp.
Prob.
Data
Lat.
Imp.
Prob.
class I
256 Bytes
2
0,15
class class 2
256 Bytes 2048 Bytes
0.3
class I+
256 Bytes
2
0.05
10
1
0.2
class 2
2048 Bytes
10
0.3
class 2+
2048 Bytes
0.05
3.3.2 NO-CSI
Without CSI, one cannot much so one may want least to learn (usually with autoregression) the channel model. Using DRL approach, the dynamics of the environment do not change through the training (and testing) and the agent learns how to react well under those conditions. Therefore, though it acts in model-free manner, in way it learns the statistics of the problem. For fair comparison, we consider a benchmark that knows the statistics of the channel and traffic dynamics. Based on knowing the distributions of the problem, one reach to optimization problem with multiple local optimums. The Frank- Wolfe algorithm guarantees reaching to a local optimum, so this method Ninit times and pick the best found local optimum. More details in appendix B.I
EXPERIMENTS
We consider two for the traffic, described in table The first one has two classes, with requiring a low amount of data but with a stringent latency constraint (of just two time slots) and another one with the opposite. Every class has the same importance which is the main difference with the second scenario where certain part of users are of highest priority. An important remark that when users are all of equal importance, then the objective of the scheduler coincides with indiscriminately increasing the satisfaction probability of every user. Finally the Prob. column describes the probability a user of the class to appear at a time slot (note that they do not add up to one since it is possible that user appears).
The channel dynamics is described through a parameter p. For = the dynamics behaves in an i..d. manner increasing the unpredictability of the future but also the chances recover from bad quality connection. We consider users appear with distance from the base station varying from 0.05 km to 1 km. We keep the energy per symbol(/andwiith)) equal to I,J. For more information on the architecture of the networks and the setting, the interested reader is referred to appendix C Hereafter we refer to the DRL approach as "Deep Scheduler".
In Figures 3a 3b we demonstrate the high performance of the Deep Scheduler for the full CSI case with maximum number of K 100 users. consistently manages outperform the myopic Knapsack approach, which is myopically optimal. We emphasize that solving Knapsack problem non-polynomial. We also show that actually the performance of Deep Scheduler close optimal since it is close to the "oracle ILP", which is an upper bound since the scheduler knows in advance the future.
In Figures 3e and 3d we focus on the no-CSI case with K 60 users. In that case, we know that suboptimal solution. We repeat the algorithm for many initialization (for each time step) to get the best possible solution among suboptimal ones. Note that this process is unfortunately very slow (see appendix B.T) and gets slower for increasing K. So even if the method shows considerable improvements with higher Ninits were obliged to stop at 20. Moreover, as K increases, unfortunately so does the number of local optimums and and that of solutions with poor performance. This is why we the Deep Scheduler to substantially outperform the FW for even a moderate = 60.
the Frank Wolfe (FW) method reaches
Finally include Figure 3e] which showcases that the Deep Scheduler consistently keeps its high performance even if the traffic model gets more complicated so as to represent more accurately a real-world scenario.

---

Under review as a conference paper at ICLR 2021
Satisfaction Probability
0.95
0.95
0.95
Deep Schoduler
0.9
FW Non
0.9
FW Neur
5
FW Nam
oracle ILP
0.85
myopic Knapeack Scheduler
0.8
Deep Scheduler
0.85
myopic Knapsack
0.8
oracle ILP
0.75
0.2
0.4
0.8constant
1.5
2
2.5
L.d.
0.2
0.4
0.6
0.8 constant
Channel Dynamics P
Bandwidth Resources (MHz)
Channel Dynamics
(b)
0.85
Satisfaction Probability
0.9
Rewards
0.8
Deep Scheduler
PW Nusi 20
0.6
Deep
0.7
myopr Krepsacx
3.5
4.5
5 5.5
Bandwictth Resources (MHz)
Resources (MHz))
(d)
(c)
Figure 3: Performance Deep Scheduler versus benchmarks on different scenarios. The first four correspond to the case of Table Tap and the last to the other. Figures 3a 3b refer to the case of full-CSI and the rest to no-CSI.
CONCLUSION
The problem scheduling and resource allocation for a time varying set of clients with heteroge- neous traffic and QoS requirements in wireless networks has been investigated here. Leveraging on deep reinforcement learning, we have proposed a deep deterministic policy gradient algorithm, which builds upon distributional reinforcement learning and Deep Sets. Experimental evaluation of our proposed method scenarios with different traffic and wireless channel dynamics, shows significant gains against state-of-the-art conventional combinatorial optimization methods.
REFERENCES
Technical Report: 3GPP TR 36.913 v15.0.0: Requirements for further advancements for E-UTRA (LTE-Advanced), 6 2018. URL https: h/www//wwwwwww wwwpwwwwpppp.popoooomccoo..p..p org/ http://tp//eeec/cccccee////////
seri series/36. 913/
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference on Learning Representations, ICLR, Vancouver, Canada, 5 2018.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference Machine Learning, ICML, Syndey, Australia, 8 2017.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pp. 679- 684, 1957.
Arthur Charpentier, Romuald Elie, and Carl Remlinger. Reinforcement learning in economics and finance. arXiv preprint arXiv:2003. 10014, 2020.
Sandeep Chinchali, Pan Hu, Tianshu Chu, Manu Sharma, Manu Bansal, Rakesh Misra, Marco Pavone, and Sachin Katti. Cellular network traffic scheduling with deep reinforcement learning. In AAAI, pp. 766-774, 2018.
8

---

Under review as a conference paper at ICLR 2021
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. arXiv preprint arXiv:1806 06923, 2018a.
Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference Artificial Intelligence, New Orleans, USA, 2 2018b.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, et al. Noisy networks for exploration. International Conference on Learning Representations, 2018.
Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval Research
Logistics Quarterly, 3(1-2)955-110, 1956. doi: 10. 00.10002/nv.33800000100 URL https://
onl nel iibrar.r.... wi wtw.://wwwww.....o/ooooooo.oo. www./oom..oo..............oo.... abs/ 1002 nav. 3800030109
Stratton C. Jaquette. Markov decision processes with a new optimality criterion: Discrete time. The
Amnals of Statistics, ((3):49-5555) 1973.
Leslie Pack Kaclbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-13, 1998.
Jens Kober, J. Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The Interna- tional Journal of Robotics Research, 32: 12:1223-1 09 2013.
Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint M.KKv:5559.00999 2015.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations, ICLR, San Juan, Puerto Rico, 5 2016.
John Mitchell. Branch-and -nt-cut algorithms for combinatorial optimization problems. Handbook of applied optimization, 1:65-77, 2002.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 ((7754)):529-533
2015. 2015.
Oshri Naparstek and Kobi Cohen. Deep multi-user reinforcement learning for distributed dynamic spectrum access. IEEE Transactions on Wireless Communications, 18(1):310-33) 2018.
Yasar Sinan Nasir and Dongning Guo. Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks. IEEE Journal on Selected Areas in Communications, 37(10): 2239-2250, 2019.
Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takáe. Reinforcement learning for solving the vehicle routing problem. In Advances in Neural Information Processing Systems, pp. 9839-9849, 2018.
Albert H. Nuttall. Some integrals involving the q.m function. IEEE Transactions on Information Theory, 21(1):95-96, 4 1975.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministie policy gradient algorithms. Proceedings of the 31st International Conference Machine Learning, PMLR, 32(1):387-395, 2014.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484449) 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv: 171200112,, 2017.

---

Under review as a conference paper at ICLR 2021
Christopher C. Tan and Norman C. Beaulieu. On first-order markov modeling for the rayleigh fading channel. IEEE Transactions on Communications, 48(12):2032-25 204400000 2000.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning, ICML, New York, USA, 6 2016.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3391-3401. Curran Associates, Inc., 2017. URL http::/papers. wiws.ipp....... cc/pper// 6931 dee-ddeep--stseee
Karl Johan Åström. Optimal control of markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10:174-205, 1965. URL https: / //wpp.....// lub.
uu. se/ att////arcc///////ee///3333333/2
10


Reviews:
[{"review_id": "phqAnt6FLhf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the long standing problem of scheduling and resource allocation in wireless networks using modern Deep Reinforcement Learning techniques. \nIt is clearly written and easy to follow but suffers from several minor typos.\nThe methodology is well justified and thoroughly motivated.\nExperimental evaluation seems thorough and provides convincing results.\n\nThe MDP is not described thoroughly enough: \nWhat is your reward, action space, state space, observations?\nHow is the allocation deadline incorporated into the reward?\nIt would be nice to have these details listed in a sub-section somewhere in Section 3.\n\nRegarding the evaluation, \"synthetic\" traffic patterns are used.\nCan you use real world traces with a simulator for evaluation (similar to https://github.com/hongzimao/pensieve)?\nAlso real world applicability is not addressed?\nWill the inference times for the deep network lead to any significant overheads when measured at the time scale of wireless communications?\nOverall, the evaluation setup seems preliminary to me and needs more work to provide assurance of real world usability.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TYXWyLwuJj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary\nThis paper investigated the problem of scheduling and resource allocation for a time-varying set of clients with heterogeneous traffic and QoS requirements in wireless networks. It proposed to solve this problem with distributional based DDPG with Deep Sets, and conducted experiments showing performance gains against conventional methods.\n\nPaper Strength\n1.\tThe paper considered a complex scheduling scenario, which is a hard problem by conventional optimization methods. The problem setting takes into account traffic model, geometry model, channel model, and rate model. Both full-CSI and partial-CSI scenarios are considered. \n2.\tThe paper adopted state-of-the-art techniques and works fine. Specifically, Distributional RL and Deep Sets for speeding up the convergence and reducing neural network parameters, respectively. \n3.\tThe proposed algorithm outperforms conventional combinatorial optimization methods.\n\nPaper Weakness\n1.\tThe presentation of the paper should be improved. Right now all the model details are placed in the appendix. This can cause confusion for readers reading the main text. \n2.\tThe necessity of using techniques includes Distributional RL and Deep Sets should be explained more thoroughly. From this paper, the illustration of Distributional RL lacks clarity.\n3.\tThe details of state representation are not explained clear. For an end-to-end method like DRL, it is crucial for state representation for training a good agent, as for network architecture.\n4.\tThe experiments are not comprehensive for validating that this algorithm works well in a wide range of scenarios. The efficiency, especially the time efficiency of the proposed algorithm, is not shown. Moreover, other DRL benchmarks, e.g., TD3 and DQN, should also be compared with. \n5.\tThere are typos and grammar errors.\n\nDetailed Comments\n1.\tSection 3.1, first paragraph, quotation mark error for \"importance\".\n2.\tAppendix A.2 does not illustrate the state space representation of the environment clearly.\n3.\tThe authors should state clearly as to why the complete state history is enough to reduce POMDP for the no-CSI case.\n4.\tSection 3.2.1: The first expression for $J(\\theta)$ is incorrect, which should be $Q(s_{t_0},\\pi_\\theta(s_{t_0}))$.\n5.\tThe paper did not explain Figure 2 clearly. In particular, what does the curve with the label \"Expected\" in Fig. 2(a) stand for? Not to mention there are multiple misleading curves in Fig. 2(b)&(c). The benefit of introducing distributional RL is not clearly explained. \n6.\tIn Table 1, only 4 classes of users are considered in the experiment sections, which might not be in accordance with practical situations, where there can be more classes of users in the real system and more user numbers.\n7.\tIn the experiment sections, the paper only showed the Satisfaction Probability of the proposed method is larger than conventional methods. The algorithm complexity, especially the time complexity of the proposed method in an ultra multi-user scenario, is not shown. \n8.\tThere is a large literature on wireless scheduling with latency guarantees from the networking community, e.g., Sigcomm, INFOCOM, Sigmetrics. Representative results there should also be discussed and compared with. \n\n======\npost rebuttal: My concern regarding the experiments remains. I will keep my score unchanged. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "F6U_XD_jIjg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Basically, it seems that the proposed method is interesting and meaningful. The scheduling problem in this paper is based on the analogy to a server having a water pitcher, and the deep reinforcement learning approach for the scheduling problem has been designed. However, the scheduling problem in wireless networks is a very famous issue. Of course, applying DRF to it is quite interesting. However, the authors need to describe the conventional well-known scheduling algorithms and compare them with the proposed scheme (now, the current paper only focuses on applying the DRF to the scheduling and evaluating its performance in aspects of an optimization problem.). Further,  typically, in scheduling problems, efficiency (total data rate) and fairness are the key factors and it is needed to describe the relationship between these conventional performance metrics and the satisfaction probability. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9uAeDXOu9LE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a deep RL solution for the communication problem of user scheduling and resource allocation. The deep RL solution uses deterministic policy gradient + quantile regression + dueling + deep sets and the authors demonstrate it outperforms classical solutions on benchmark tasks. \n\nRecommendation:\n\nI quite frankly have no prior knowledge of the task this paper aims to solve. It didn't come across as a grandstanding AI challenge in any capacity (feel free to debate this) so I'm looking at this paper from the angle of significance to the deep RL community. In general, the questions I'm asking are: Does this paper introduce a novel deep RL algorithm? Is the knowledge produced by this paper generalizable to other problems or algorithms? Does this paper provide value to anyone who isn't concerned with the specific application? \n\nAs of now, I feel like the answer is no to all three and so I would recommend rejection to this conference. However, there are presumably venues which are a better fit and I hope the authors consider submitting elsewhere. \n\nThere were other issues with the paper. In general, clarity and organization were a big issue for me. The authors were very rigorous with their supplementary material, so I believe most of the information is there, but was not presented in an easily digestible manner. \n\nStrengths:\n- Thorough supplementary material and code is provided. \n- The use of deep RL to tackle the problem felt well-motived and a good fit. \n- The performance of the agent seems strong but I'm not clear on the significance of some of the results. \n\nWeaknesses:\n- I think the problem set up was clear in the sense that I understood the overarching objective. However, the specifics of the problem, specifically in the context of RL was not. The paragraph about 3.2 is a generic description of the RL problem and left me wondering the connection to the actual application. I realize many of the details are contained in the supplementary material but the statement \"the problem can be modeled as an MDP\" was not defended & the following description did not clarify the problem statement. For example, immediately after, in 3.2, \"high variance randomness\" is discussed but its not clear to me why this is the case or how this randomness affects the problem- reward? transitions?\n- The structure of the paper did not feel helpful to me. 3.2 is categorized into \"Policy Network\" and \"Value Network\" for somebody who is comfortable with RL a lot of the details felt unnecessary but more importantly, this organization doesn't provide a solid presentation of the algorithm. Somebody who is interested in the application and is not an RL expert, won't necessarily follow why \"Policy network\" is being presented or what that even means necessarily. There isn't a clear overview of the algorithm. \n- Novelty of the algorithm is low in the sense that it is a combination of prior, existing ideas. \n- I felt like many of the algorithmic choices were not well-justified. For example, the use of QR is justified by an analogy? The use of the dueling architecture also seems unusual when the authors also propose a much simpler solution. It was also unclear what the issue with \"The main problem was that the distribution Z was far away from 0 making it very difficult for the policy network to well approximate them\" exactly meant. \n\nMinor Comments:\n- There are a few latex issues with reversed quotations. \n- \"In Figure 2 we provide additional element to support the choice\" -> an additional\n- The objective of Figure 2 is nice but its confusing to have two sets of experiments presented in the same graph. Class is not explained in the description of the graph and the significance of the graph is not explained in the figure description.\n\n**Post-Rebuttal\n\nI appreciate the authors taking the time to respond. Unfortunately, my belief that this paper is not strong enough from the deep RL perspective to warrant acceptance has not changed. If other combinations of tools do not work, then the authors should improve this justification, with ablation studies or stronger theoretical motivation. I’ll add that my score is not influenced by my concern that this paper may not be a good fit for ICLR (I leave that choice to the AC).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers mostly agree that this paper presents a new deep reinforcement learning-based approach to solving a challenging problem in the communications domain -- wireless scheduling. However, the main concern, expressed almost unanimously, is about the novelty of the ideas in the paper beyond the assembly of existing deep RL techniques and the translation of the scheduling problem to the language of MDPs in a careful manner that respects modern communication systems standards such as 5G (e.g., URLLC and eMBB traffic demands). A secondary concern, also expressed during the author rebuttal discussion, is about adequate comparison to competing approaches motivated from the literature in wireless scheduling. In view of these issues, I suggest that the author(s) explore more appropriate avenues to submit this piece of valuable translational work, including venues that address the specific topic of wireless communication where a more comprehensive evaluation and comparison could be possible. 

(NOTE: The comments and evaluation above disregard the "enhanced" draft submitted by the author(s) during the rebuttal phase. I was informed that the submission was reverted to the original draft due to space constraints being exceeded in the enhanced version.)

Decision: Reject