NeurWIN: Neural Whittle Index Network for Restless Bandits via Deep RL

1 INTRODUCTION . Many sequential decision problems can be modeled as multi-armed bandit problems . A bandit problem models each potential decision as an arm . In each round , we play M arms out of a total of N arms by choosing the corresponding decisions . We then receive a reward from the played arms . The goal is to maximize the long-term total discounted reward . Consider , for example , displaying advertisements on an online platform with the goal to maximize the long-term discounted clickthrough rates . This can be modeled as a bandit problem where each arm is a piece of advertisement and we choose which advertisements to be displayed every time a particular user visits the platform . It should be noted that the reward , i.e. , click-through rate , of an arm is not stationary , but depends on our actions in the past . For example , a user that just clicked on a particular advertisement may be much less likely to click on the same advertisement in the near future . Such a problem is a classic case of the restless bandit problem , where the reward distribution of an arm depends on its state , which changes over time based on our past actions . The restless bandit problem is notoriously intractable ( Papadimitriou & Tsitsiklis , 1999 ) . Most recent efforts , such as recovering bandits ( Pike-Burke & Grunewalder , 2019 ) , rotting bandits ( Seznec et al. , 2020 ) , and Brownian bandits ( Slivkins & Upfal , 2008 ) , only study some special instances of the restless bandit problem . The fundamental challenge of the restless bandit problem lies in the explosion of state space , as the state of the entire system is the Cartesian product of the states of individual arms . A powerful tool to address the explosion of state space is the Whittle index policy ( Whittle , 1988 ) . In a nutshell , the Whittle index policy calculates a Whittle index for each arm based on the arm ’ s current state , where the index loosely corresponds to the amount of cost that we are willing to pay to play the arm , and then plays the arm with the highest index . It has been shown that the Whittle index policy is either optimal or asymptotically optimal in many settings . In this paper , we present Neural Whittle Index Network ( NeurWIN ) , a principled machine learning approach that finds the Whittle indices for virtually all restless bandit problems . We note that the Whittle index is an artificial construct that can not be directly measured . Finding the Whittle index is typically intractable . As a result , the Whittle indices of many practical problems remain unknown except for a few special cases . We are able to circumvent the challenges of finding the Whittle indices by leveraging an important mathematical property of the Whittle index : Consider an alternative problem where there is only one arm and we decide whether to play the arm in each time instance . In this problem , we need to pay a constant cost of λ every time we play the arm . The goal is to maximize the long-term discounted net reward , defined as the difference between the rewards we obtain from the arm and the costs we pay to play it . Then , the optimal policy is to play the arm whenever the Whittle index becomes larger than λ . Based on this property , a neural network that produces the Whittle index can be viewed as one that finds the optimal policy for the alternative problem for any λ . Using this observation , we propose a deep reinforcement learning method to train NeurWIN . To demonstrate the power of NeurWIN , we employ NeurWIN for three recently studied restless bandit problems , namely , recovering bandit ( Pike-Burke & Grunewalder , 2019 ) , wireless scheduling ( Aalto et al. , 2015 ) , and stochastic deadline scheduling ( Yu et al. , 2018 ) . There is no known Whittle index for the first problem , and there is only an approximation of the Whittle index under some relaxations for the second problem . Only the third problem has a precise characterization of the Whittle index . For the first two problems , the index policy using our NeurWIN achieves better performance than existing studies . For the third problem , the index policy using our NeurWIN has virtually the same performance as the Whittle index policy . The rest of the paper is organized as follows : Section 2 reviews related literature . Section 3 provides formal definitions of the Whittle index and our problem statement . Section 4 introduces our training algorithm for NeurWIN . Section 5 demonstrates the utility of NeurWIN by evaluating its performance under three recently studied restless bandit problems . Finally , Section 6 concludes the paper . 2 RELATED WORK . Restless bandit problems were first introduced in ( Whittle , 1988 ) . They are known to be intractable , and are in general PSPACE hard ( Papadimitriou & Tsitsiklis , 1999 ) . As a result , many studies focus on finding the Whittle index policy for restless bandit problems , such as in ( Le Ny et al. , 2008 ; Meshram et al. , 2018 ; Tripathi & Modiano , 2019 ; Dance & Silander , 2015 ) . However , these studies are only able to find the Whittle indices under various specific assumptions about the bandit problems . There has been a lot of studies on applying RL methods for bandit problems . ( Dann et al. , 2017 ) proposed a tool called Uniform-PAC for contextual bandits . ( Zanette & Brunskill , 2018 ) described a framework-agnostic approach towards guaranteeing RL algorithms ’ performance . ( Jiang et al. , 2017 ) introduced contextual decision processes ( CDPs ) that encompass contextual bandits for RL exploration with function approximation . ( Riquelme et al. , 2018 ) compared deep neural networks with Bayesian linear regression against other posterior sampling methods . However , none of these studies are applicable to restless bandits , where the state of an arm can change over time . Deep RL algorithms have been utilized in problems that resemble restless bandit problems , including HVAC control ( Wei et al. , 2017 ) , cyber-physical systems ( Leong et al. , 2020 ) , and dynamic multichannel access ( Wang et al. , 2018 ) . In all these cases , a major limitation for deep RL is scalability . As the state spaces grows exponentially with the number of arms , these studies can only be applied to small-scale systems , and their evaluations are limited to cases when there are at most 5 zones , 6 sensors , and 8 channels , respectively . An emerging research direction is applying machine learning algorithms to learn Whittle indices . ( Borkar & Chadha , 2018 ) proposed employing the LSPE ( 0 ) algorithm ( Yu & Bertsekas , 2009 ) coupled with a polynomial function approximator . The approach was applied in ( Avrachenkov & Borkar , 2019 ) for scheduling web crawlers . However , this work can only be applied to restless bandits whose states can be represented by a single number , and it only uses a polynomial function approximator , which may have low representational power ( Sutton & Barto , 2018 ) . ( Fu et al. , 2019 ) proposed a Q-learning based heuristic to find Whittle indices . However , as shown in its experiment results , the heuristic may not produce Whittle indices even when the training converges . 3 PROBLEM SETTING . In this section , we provide a brief overview of restless bandit problems and the Whittle index . We then formally define the problem statement . 3.1 RESTLESS BANDIT PROBLEMS . A restless bandit problem consists of N restless arms . In each round t , a control policy observes the state of each arm i , denoted by si [ t ] , and selects M arms to activate . We call the selected arms as active and the others as passive . We use ai [ t ] to denote the policy ’ s decision on each arm i , where ai [ t ] = 1 if the arm is active and ai [ t ] = 0 if it is passive at round t. Each arm i generates a stochastic reward ri [ t ] with distribution Ri , act ( si [ t ] ) if it is active , and with distribution Ri , pass ( si [ t ] ) if it is passive . The state of each arm i in the next round evolves by the transition kernel of either Pi , act ( si [ t ] ) or Pi , pass ( si [ t ] ) , depending on whether the arm is active . The goal of the control policy is to maximize the total discounted reward , which can be expressed as ∑∞ t=1 ∑N i=1 β tri [ t ] with β being the discount factor . A control policy is effectively a function that takes the vector ( s1 [ t ] , s2 [ t ] , . . . , sN [ t ] ) as the input and produces the vector ( a1 [ t ] , a2 [ t ] , . . . , aN [ t ] ) as the output . It should be noted that the space of input is exponential inN . If each arm can be in one ofK possible states , then the number of possible inputs isKN . This feature , which is usually referred to as the curse of dimensionality , makes finding the optimal control policy intractable . 3.2 THE WHITTLE INDEX . An index policy seeks to address the curse of dimensionality through decomposition . In each round , it calculates an index , denoted by Wi ( si [ t ] ) , for each arm i based on its current state . The index policy then selects the M arms with the highest indices to activate . It should be noted that the index of an arm i is independent from the states of any other arms . Obviously , the performance of an index policy depends on the design of the index functionWi ( · ) . A popular index with solid theoretical foundation is the Whittle index , which is defined below . Since we only consider one arm at a time , we drop the subscript i for the rest of the paper . Consider a system with only one arm , and a control policy that determines whether to activate the arm in each round t. Suppose that the policy needs to pay an activation cost of λ every time it chooses to activate the arm . The goal of the control policy is to maximize the total discounted net reward , ∑∞ t=1 β t ( r [ t ] − λa [ t ] ) . The optimal control policy can be expressed by the set of states in which it would activate this arm for a particular λ , and we denote this set by A ( λ ) . Intuitively , the higher the cost , the less likely the optimal control policy would activate the arm in a given state , and hence the set A ( λ ) should decrease monotonically . When an arm satisfies this intuition , we say that the arm is indexable . Definition 1 ( Indexability ) . An arm is said to be indexable if A ( λ ) decreases monotonically from the set of all states to the empty set as λ increases from −∞ to∞ . A restless bandit problem is said to be indexable if all arms are indexable . Definition 2 ( The Whittle Index ) . If an arm is indexable , then its Whittle index of each state s is defined as W ( s ) : = supλ { λ : s ∈ A ( λ ) } . Even when an arm is indexable , finding its Whittle index can still be intractable , especially when the transition kernel of the arm is convoluted1 . Our NeurWIN finds the Whittle index by leveraging the following property of the Whittle index : Consider the single-armed bandit problem . Suppose the initial state of an indexable arm is s at round one . Consider two possibilities : The first is that the control policy activates the arm at round one , and then uses the optimal policy starting from round two ; and the second is that the control policy does not activate the arm at round one , and then uses the optimal policy starting from round two . Let Qλ , act ( s ) and Qλ , pass ( s ) be the expected discounted net reward for these two possibilities , respectively , and let Ds ( λ ) : = ( Qλ , act ( s ) − Qλ , pass ( s ) ) be their difference . Clearly , the optimal policy should activate an arm under state s and activation cost λ if Ds ( λ ) ≥ 0 . We then have the following : Theorem 1 . ( Zhao , 2019 , Thm 3.14 ) If an arm is indexable , then , for every state s , Ds ( λ ) ≥ 0 if and only if λ ≤W ( s ) . 1Niño-Mora ( 2007 ) described a generic approach for finding the Whittle index . The complexity of this approach is at least exponential to the number of states . Our NeurWIN uses Thm . 1 to train neural networks that predict the Whittle index for any indexable arms . From Def . 1 , a sufficient condition for indexability is when Ds ( λ ) is a decreasing function . Thus , we define the concept of strong indexability as follows : Definition 3 ( Strong Indexability ) . An arm is said to be strongly indexable if Ds ( λ ) is strictly decreasing in λ for every state s .

Summary: This paper considers the problem of learning how to control restless bandits.  When all parameters of the system are known, Whittle index policy usually offers a good performance. The main contribution of the paper is to propose an algorithm, NeurWIN, that uses a neural network architecture to learn the Whittle indices. Most of the paper is devoted to the description and the derivation of the algorithm. At the end of the paper, the authors present four illustrations of how the algorithm works. The algorithm is compared to an index-based policy and an (old?) RL algorithm REINFORCE for the small systems. The learning behavior of NeurWIN is very good in all tested cases.