Regime Switching Bandits

√ log T ) for the proposed learning algorithm where T is the learning horizon . Finally , we conduct proof-of-concept experiments to illustrate the performance of the learning algorithm . 1 Introduction . The multi-armed bandit ( MAB ) problem is a popular model for sequential decision making with unknown information : the decision maker makes decisions repeatedly among I different options , or arms . After each decision she receives a random reward having an unknown probability distribution that depends on the chosen arm . The objective is to maximize the expected total reward over a finite horizon of T periods . The MAB problem has been extensively studied in various fields and applications including Internet advertising , dynamic pricing , recommender systems , clinical trials and medicine [ 12 , 13 , 43 ] . In the classical MAB problem , it is typically assumed that the random reward of each arm is i.i.d . ( independently and identically distributed ) over time and independent of the rewards from other arms . However , these assumptions do not necessarily hold in practice [ 10 ] . To address the drawback , a growing body of literature studies MAB problems with non-stationary rewards to capture temporal changes in the reward distributions in applications , see e.g . [ 11 , 16 , 20 ] . In this paper , we study a non-stationary MAB model with Markovian regime-switching rewards . We assume that the random rewards associated with all the arms are modulated by a common unobserved state ( or regime ) { Mt : t = 1 , 2 , . . . } modeled as a finite-state discrete-time Markov chain . This chain makes a transition at each period regardless of which arm is pulled and its transition probabilities are independent of the action chosen . Given Mt = m , the reward of arm i is i.i.d. , whose distribution is denoted Q ( ·|m , i ) . Such structural change of the environment is usually referred to as regime switching in finance [ 33 ] . The agent doesn ’ t observe or control the underlying state Mt , and has to learn the transition probability matrix P of { Mt } as well as the distribution of reward of each arm Q ( ·|m , i ) , based on the observed historical rewards . The goal of the agent is to design a learning policy that decides which arm to pull in each period to minimize the expected regret over T periods . ∗The first two authors ( Xiang Zhou and Yi Xiong ) have equal contribution . †Department of Systems Engineering and Engineering Management , The Chinese University of Hong Kong ; 1911606962 @ qq.com ‡Department of Systems Engineering and Engineering Management , The Chinese University of Hong Kong ; yxiong @ se.cuhk.edu.hk §The Rotman School of Management , University of Toronto ; ningyuan.chen @ utoronto.ca ¶Department of Systems Engineering and Engineering Management , The Chinese University of Hong Kong ; xfgao @ se.cuhk.edu.hk 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The regime-switching models are widely used in various industries . For example , in finance , the sudden changes of market environments are usually modeled as hidden Markov chains . In revenue management and marketing , a firm may face a shift in consumer demand due to undetected changes in sentiment or competition . In such cases , when the agents ( traders or firms ) take actions ( trading financial assets with different strategies or setting prices ) , they need to learn the reward and the underlying state at the same time . Our setup is designed to tackle such problems . Our Contribution . Our study features novel designs in three regards : we propose a new bandit problem formulation , develop a new learning algorithm , and prove regret bounds with new techniques . In terms of problem formulation , online learning with unobserved states has attracted some attention recently [ 8 , 19 ] . We consider the strongest oracle among the studies , who knows P and Q ( ·|m , i ) , but doesn ’ t observe the hidden state Mt . The oracle thus faces a partially observable Markov decision process ( POMDP ) [ 26 ] ( with a long-run average reward objective ) . By reformulating it as a Markov decision process ( MDP ) with a continuous belief space ( i.e. , a distribution over hidden states ) , the oracle then solves the optimal policy ( mapping belief states to actions ) using the Bellman equation . Having sublinear regret benchmarked against the strong oracle , our algorithm has better theoretical performance than others with weaker oracles such as the best fixed arm [ 19 ] or memoryless policies ( the action only depends on the current observation ) [ 8 ] . In terms of algorithmic design , we propose a learning algorithm ( see Algorithm 2 ) with two key ingredients . First , it builds on the recent advance on the estimation of the parameters of hidden Markov models ( HMMs ) using spectral method-of-moments methods [ 3 , 2 , 8 ] . It benefits from the theoretical finite-sample bound of spectral estimators , while the finite-sample guarantees of other alternatives such as maximum likelihood estimators remain an open problem [ 30 ] . Second , it builds on the well-known “ upper confidence bound ” ( UCB ) method in reinforcement learning [ 7 , 25 ] . There are two difficulties here as the oracle uses the optimal ( belief-based ) policy of the POMDP . First , the spectral method can not use the non i.i.d . samples generated from the belief-based policy due to the complex history dependency . Second , the belief of the hidden state is subject to the estimation error . Hence , we divide the horizon into nested exploration and exploitation phases . We use spectral estimators in the exploration phase to gauge the estimation error of P and Q ( ·|m , i ) . We use the UCB method to control the regret in the exploitation phase . Different from other learning problems , we re-calibrate the belief at the beginning of each exploitation phase based on the parameters estimated in the most recent exploration phase using previous exploratory samples . In terms of technical analysis , we establish a regret bound of O ( T 2/3 √ log ( T ) ) for our proposed learning algorithm where T is the learning horizon . Our regret analysis draws inspirations from [ 25 , 35 ] for learning MDPs and undiscounted reinforcement learning problems , but the analysis differs significantly from theirs since there are two main technical challenges in our problem . First , to control the regret , we need to control of the error of the belief state , which itself is not directly observed and needs to be estimated . This is in stark contrast to learning MDPs [ 25 , 35 ] with observed states . Specifically , we need to bound the estimation error of belief states by the estimation errors of the model parameters . In addition , since the belief state is un-observable , we also need to bound the error in the belief transition kernel , which measures the distance between the belief transition kernel under the optimistic belief MDP model ( from the UCB component of our algorithm ) at each episode and the belief transition kernel under the true model . These bounds are not trivial since the transition kernel of the belief state depends on the model parameters in a complex way via Bayesian updating . We overcome the difficulties by building on [ 18 ] and a delicate analysis of the belief transition kernel to control the errors in the estimations of belief states and the belief transitions . Second , to establish regret bound , we need an explicit bound for the span of the bias function ( also referred as the relative value function ) for the belief MDP which has a continuous state space . Such a bound is often critical in the regret analysis of undiscounted reinforcement learning of continuous MDP , but it is either taken as an assumption [ 39 ] or proved under Hölder continuity assumptions that do not hold for the belief transitions in our setting [ 35 , 27 ] . We overcome this challenge and bound the bias span by developing a novel approach , which could be of independent interest for learning continuous-state MDPs . Specifically , we bound the bias span by bounding the Lipschitz module of the bias function for our infinite-horizon undiscounted problem ( with a long-run average reward objective ) . To achieve this , we rely on a non-trivial application of the results in [ 23 ] which provide general tools for proving Lipschitz continuity of value functions in finite-horizon discounted MDPs . One key step is to bound the Lipschitz module of the belief transitions using the Kantorovich metric , which allows us to establish a bound on the Lipschitz module of the value function of the finite-horizon discounted problem uniformly over the discounting factors . Exploiting the connection with the infinite-horizon undiscounted problem then yields an explicit bound on the bias span for our problem . We also mention that our bound on the bias span is unrelated to the diameter of the POMDP discussed in [ 8 ] . The diameter in [ 8 ] is only for observation-based policies , not for belief-state based policies we consider . That also explains why we need a new approach to bound the bias span . Related Work . Studies on non-stationary/switching MAB investigate the problem when the rewards of all arms may change over time [ 5 , 20 , 10 , 6 ] . Our formulation can be regarded as non-stationary MAB with a special structure . They consider an even stronger oracle than ours , the best arm in each period . However , the total number of changes or the changing budget have to be sublinear in T to achieve sublinear regret . In our formulation , the total number of transitions of the underlying state is linear in T , and the algorithms proposed in these papers fail even considering our oracle ( See Section 5 ) . Other studies focus on linear changing budget with some structure such as seasonality [ 15 ] , which is not present in this paper . Our work is also related to the restless Markov bandit problem [ 21 , 36 , 44 ] in which the state of each arm evolves according to independent Markov chains . In contrast , our regime-switching MAB model assumes a common underlying Markov chain so that the rewards of all arms are correlated , and the underlying state is unobservable to the agent . In addition , our work is related to MAB studies where rewards of all arms depend on a common unknown parameter or a latent random variable , see , e.g. , [ 4 , 22 , 28 , 32 , 34 ] . Our model differs from them in that the common latent state variable follows a dynamic stochastic process which introduces difficulties in algorithm design and analysis . Two papers have similar settings to ours . [ 19 ] studies MAB problems whose rewards are modulated by an unobserved Markov chain and the transition matrix may depend on the action . However , their oracle is the best fixed arm when defining the regret , which is much weaker than the optimal policy of the POMDP ( the performance gap is linear between the two oracles ) . Therefore , their algorithm is expected to have linear regret when using our oracle . [ 8 ] proposes an algorithm based on spectral estimators for learning POMDPs . Their algorithm will generally suffer linear regret in our problem setting . This is because we consider a stronger oracle which is the optimal POMDP policy , whereas their oracle is the optimal memoryless policy , i.e. , a policy that only depends on the current reward observation instead of using all historical observations to form the belief of the underlying state . It is known that memoryless policies are in general not optimal and hence the gap between their oracle and our oracle can be linear . Note that considering the memoryless policy allows [ 8 ] to circumvent the introduction of the belief entirely . Indeed , the dynamics under the memoryless policy can be viewed as a finite-state ( modified ) HMM and spectral estimators can be applied . Instead , because we consider the optimal belief-based policy , such reduction is not available . Our algorithm and regret analysis hinge on the interaction between the estimation of the belief and the spectral estimators . Our algorithm needs separate exploration to apply the spectral methods and uses the belief-based policy for exploitation . For the regret analysis , unlike [ 8 ] , we have to carefully control the belief error and bound the span of the bias function from the optimistic belief MDP in each episode . The comparison of our study with related papers is summarized in Table 1 . Note that we only present the regret in terms of T .

Summary: The authors study a variation of the multi-armed bandit problem where the rewards are neither adversarial nor stochastic. The rewards are actually random variables drawn from distributions that change based on an unknown state.  This changing state is modeled by a finite-state Markov chain, and the learner does not know the current state, nor the matrix and reward distribution. The difficulty of this problem seems to be controlled by two different aspects: how strong is the oracle you compare the learner to, and how much budget has the environment to change the state that it is in.    The oracle is the best action in hindsight, then results for adversarial MAB would apply, which give  $ \tilde O(\sqrt T)$ types of regret bounds. The authors consider a much stronger oracle that is aware of the transition and the reward matrix of the MDP, but not of the current underlying state. In their problem setting, the changing budget is linear in T.   The authors present an algorithm for this problem and derive a O(T^{2/3}\sqrt{\ln{T}}) regret bound against the previously mentioned oracle. Their algorithm combines several concepts such as the spectral method of moments methods to estimate the parameters of the MDP and the commonly used UCB method.