NeurWIN: Neural Whittle Index Network For Restless Bandits Via Deep RL

1 INTRODUCTION . Many sequential decision problems can be modeled as multi-armed bandit problems . A bandit problem models each potential decision as an arm . In each round , we play M arms out of a total of N arms by choosing the corresponding decisions . We then receive a reward from the played arms . The goal is to maximize the long-term total discounted reward . Consider , for example , displaying advertisements on an online platform with the goal to maximize the long-term discounted clickthrough rates . This can be modeled as a bandit problem where each arm is a piece of advertisement and we choose which advertisements to be displayed every time a particular user visits the platform . It should be noted that the reward , i.e. , click-through rate , of an arm is not stationary , but depends on our actions in the past . For example , a user that just clicked on a particular advertisement may be much less likely to click on the same advertisement in the near future . Such a problem is a classic case of the restless bandit problem , where the reward distribution of an arm depends on its state , which changes over time based on our past actions . The restless bandit problem is notoriously intractable ( Papadimitriou & Tsitsiklis , 1999 ) . Most recent efforts , such as recovering bandits ( Pike-Burke & Grunewalder , 2019 ) , rotting bandits ( Seznec et al. , 2020 ) , and Brownian bandits ( Slivkins & Upfal , 2008 ) , only study some special instances of the restless bandit problem . The fundamental challenge of the restless bandit problem lies in the explosion of state space , as the state of the entire system is the Cartesian product of the states of individual arms . A powerful tool to address the explosion of state space is the Whittle index policy ( Whittle , 1988 ) . In a nutshell , the Whittle index policy calculates a Whittle index for each arm based on the arm ’ s current state , where the index loosely corresponds to the amount of cost that we are willing to pay to play the arm , and then plays the arm with the highest index . It has been shown that the Whittle index policy is either optimal or asymptotically optimal in many settings . In this paper , we present Neural Whittle Index Network ( NeurWIN ) , a principled machine learning approach that finds the Whittle indices for virtually all restless bandit problems . We note that the Whittle index is an artificial construct that can not be directly measured . Finding the Whittle index is typically intractable . As a result , the Whittle indices of many practical problems remain unknown except for a few special cases . We are able to circumvent the challenges of finding the Whittle indices by leveraging an important mathematical property of the Whittle index : Consider an alternative problem where there is only one arm and we decide whether to play the arm in each time instance . In this problem , we need to pay a constant cost of λ every time we play the arm . The goal is to maximize the long-term discounted net reward , defined as the difference between the rewards we obtain from the arm and the costs we pay to play it . Then , the optimal policy is to play the arm whenever the Whittle index becomes larger than λ . Based on this property , a neural network that produces the Whittle index can be viewed as one that finds the optimal policy for the alternative problem for any λ . Using this observation , we propose a deep reinforcement learning method to train NeurWIN . To demonstrate the power of NeurWIN , we employ NeurWIN for three recently studied restless bandit problems , namely , recovering bandit ( Pike-Burke & Grunewalder , 2019 ) , wireless scheduling ( Aalto et al. , 2015 ) , and stochastic deadline scheduling ( Yu et al. , 2018 ) . There is no known Whittle index for the first problem , and there is only an approximation of the Whittle index under some relaxations for the second problem . Only the third problem has a precise characterization of the Whittle index . For the first two problems , the index policy using our NeurWIN achieves better performance than existing studies . For the third problem , the index policy using our NeurWIN has virtually the same performance as the Whittle index policy . The rest of the paper is organized as follows : Section 2 reviews related literature . Section 3 provides formal definitions of the Whittle index and our problem statement . Section 4 introduces our training algorithm for NeurWIN . Section 5 demonstrates the utility of NeurWIN by evaluating its performance under three recently studied restless bandit problems . Finally , Section 6 concludes the paper . 2 RELATED WORK . Restless bandit problems were first introduced in ( Whittle , 1988 ) . They are known to be intractable , and are in general PSPACE hard ( Papadimitriou & Tsitsiklis , 1999 ) . As a result , many studies focus on finding the Whittle index policy for restless bandit problems , such as in ( Le Ny et al. , 2008 ; Meshram et al. , 2018 ; Tripathi & Modiano , 2019 ; Dance & Silander , 2015 ) . However , these studies are only able to find the Whittle indices under various specific assumptions about the bandit problems . There has been a lot of studies on applying RL methods for bandit problems . ( Dann et al. , 2017 ) proposed a tool called Uniform-PAC for contextual bandits . ( Zanette & Brunskill , 2018 ) described a framework-agnostic approach towards guaranteeing RL algorithms ’ performance . ( Jiang et al. , 2017 ) introduced contextual decision processes ( CDPs ) that encompass contextual bandits for RL exploration with function approximation . ( Riquelme et al. , 2018 ) compared deep neural networks with Bayesian linear regression against other posterior sampling methods . However , none of these studies are applicable to restless bandits , where the state of an arm can change over time . Deep RL algorithms have been utilized in problems that resemble restless bandit problems , including HVAC control ( Wei et al. , 2017 ) , cyber-physical systems ( Leong et al. , 2020 ) , and dynamic multichannel access ( Wang et al. , 2018 ) . In all these cases , a major limitation for deep RL is scalability . As the state spaces grows exponentially with the number of arms , these studies can only be applied to small-scale systems , and their evaluations are limited to cases when there are at most 5 zones , 6 sensors , and 8 channels , respectively . An emerging research direction is applying machine learning algorithms to learn Whittle indices . ( Borkar & Chadha , 2018 ) proposed employing the LSPE ( 0 ) algorithm ( Yu & Bertsekas , 2009 ) coupled with a polynomial function approximator . The approach was applied in ( Avrachenkov & Borkar , 2019 ) for scheduling web crawlers . However , this work can only be applied to restless bandits whose states can be represented by a single number , and it only uses a polynomial function approximator , which may have low representational power ( Sutton & Barto , 2018 ) . ( Fu et al. , 2019 ) proposed a Q-learning based heuristic to find Whittle indices . However , as shown in its experiment results , the heuristic may not produce Whittle indices even when the training converges . 3 PROBLEM SETTING . In this section , we provide a brief overview of restless bandit problems and the Whittle index . We then formally define the problem statement . 3.1 RESTLESS BANDIT PROBLEMS . A restless bandit problem consists of N restless arms . In each round t , a control policy observes the state of each arm i , denoted by si [ t ] , and selects M arms to activate . We call the selected arms as active and the others as passive . We use ai [ t ] to denote the policy ’ s decision on each arm i , where ai [ t ] = 1 if the arm is active and ai [ t ] = 0 if it is passive at round t. Each arm i generates a stochastic reward ri [ t ] with distribution Ri , act ( si [ t ] ) if it is active , and with distribution Ri , pass ( si [ t ] ) if it is passive . The state of each arm i in the next round evolves by the transition kernel of either Pi , act ( si [ t ] ) or Pi , pass ( si [ t ] ) , depending on whether the arm is active . The goal of the control policy is to maximize the total discounted reward , which can be expressed as ∑∞ t=1 ∑N i=1 β tri [ t ] with β being the discount factor . A control policy is effectively a function that takes the vector ( s1 [ t ] , s2 [ t ] , . . . , sN [ t ] ) as the input and produces the vector ( a1 [ t ] , a2 [ t ] , . . . , aN [ t ] ) as the output . It should be noted that the space of input is exponential inN . If each arm can be in one ofK possible states , then the number of possible inputs isKN . This feature , which is usually referred to as the curse of dimensionality , makes finding the optimal control policy intractable . 3.2 THE WHITTLE INDEX . An index policy seeks to address the curse of dimensionality through decomposition . In each round , it calculates an index , denoted by Wi ( si [ t ] ) , for each arm i based on its current state . The index policy then selects the M arms with the highest indices to activate . It should be noted that the index of an arm i is independent from the states of any other arms . Obviously , the performance of an index policy depends on the design of the index functionWi ( · ) . A popular index with solid theoretical foundation is the Whittle index , which is defined below . Since we only consider one arm at a time , we drop the subscript i for the rest of the paper . Consider a system with only one arm , and a control policy that determines whether to activate the arm in each round t. Suppose that the policy needs to pay an activation cost of λ every time it chooses to activate the arm . The goal of the control policy is to maximize the total discounted net reward , ∑∞ t=1 β t ( r [ t ] − λa [ t ] ) . The optimal control policy can be expressed by the set of states in which it would activate this arm for a particular λ , and we denote this set by A ( λ ) . Intuitively , the higher the cost , the less likely the optimal control policy would activate the arm in a given state , and hence the set A ( λ ) should decrease monotonically . When an arm satisfies this intuition , we say that the arm is indexable . Definition 1 ( Indexability ) . An arm is said to be indexable if A ( λ ) decreases monotonically from the set of all states to the empty set as λ increases from −∞ to∞ . A restless bandit problem is said to be indexable if all arms are indexable . Definition 2 ( The Whittle Index ) . If an arm is indexable , then its Whittle index of each state s is defined as W ( s ) : = supλ { λ : s ∈ A ( λ ) } . Even when an arm is indexable , finding its Whittle index can still be intractable , especially when the transition kernel of the arm is convoluted1 . Our NeurWIN finds the Whittle index by leveraging the following property of the Whittle index : Consider the single-armed bandit problem . Suppose the initial state of an indexable arm is s at round one . Consider two possibilities : The first is that the control policy activates the arm at round one , and then uses the optimal policy starting from round two ; and the second is that the control policy does not activate the arm at round one , and then uses the optimal policy starting from round two . Let Qλ , act ( s ) and Qλ , pass ( s ) be the expected discounted net reward for these two possibilities , respectively , and let Ds ( λ ) : = ( Qλ , act ( s ) − Qλ , pass ( s ) ) be their difference . Clearly , the optimal policy should activate an arm under state s and activation cost λ if Ds ( λ ) ≥ 0 . We then have the following : Theorem 1 . ( Zhao , 2019 , Thm 3.14 ) If an arm is indexable , then , for every state s , Ds ( λ ) ≥ 0 if and only if λ ≤W ( s ) . 1Niño-Mora ( 2007 ) described a generic approach for finding the Whittle index . The complexity of this approach is at least exponential to the number of states . Our NeurWIN uses Thm . 1 to train neural networks that predict the Whittle index for any indexable arms . From Def . 1 , a sufficient condition for indexability is when Ds ( λ ) is a decreasing function . Thus , we define the concept of strong indexability as follows : Definition 3 ( Strong Indexability ) . An arm is said to be strongly indexable if Ds ( λ ) is strictly decreasing in λ for every state s .

Summary: The paper proposes a method to automatically learn the Whittle Indices for a Restless Multi-Armed Bandit (RMAB) problem.   Contributions:  The Algorithm: The algorithm tries to learn a neural network that takes as input the state of a given arm and provides as output the whittle index of that state. It does this by showing a connection between the whittle indices for an arm and the indices of an optimal index policy for a family of MDPs (Env(𝝀)) based on that arm’s MDP (Env). It then learns a good index policy for this family of MDPs using a REINFORCE-type method.  Theoretical Justification: Towards substantiating the claim that learning a good policy for Env(𝝀) is equivalent to learning a good whittle index, the paper provides a proof about an epsilon-delta relationship between learning a good policy and a good whittle index.  Empirical Justification: The paper shows that the proposed algorithm works well on 3 previously published RMAB instances, when compared with 1 similar ‘learning a whittle index’-type baseline and 2 reinforcement learning baselines.