Multi-armed Bandit Requiring Monotone Arm Sequences

1 Introduction . In online learning problems such as the multi-armed bandits ( MAB ) , the decision maker chooses from a set of actions/arms with unknown reward . The goal is to learn the expected rewards of the arms and choose the optimal one as much as possible within a finite horizon . The framework and its many variants have been successfully applied to a wide range of practical problems , including news recommendation [ 31 ] , dynamic pricing [ 9 ] and medical decision-making [ 8 ] . Popular algorithms such as UCB [ 4 , 5 ] and Thompson sampling [ 3 , 34 ] typically explore the arms sufficiently and as more evidence is gathered , converge to the optimal arm . These algorithms are designed to handle the classic MAB problems , in which the arms are discrete and do not have any order . However , in some practical problems , the arms are ordinal and even continuous . This is the focus of the study on the so-called continuum-armed bandit [ 2 , 26 , 6 , 11 , 38 ] . By discretizing the arm space properly , algorithms such as UCB and Thompson sampling can still be applied . Therefore , the continuous decision space has readily been incorporated into the framework . Another practical issue , which is not studied thoroughly but emerges naturally from some applications , is the requirement that the action/arm sequence is has to be monotone . Such a requirement is usually imposed due to external reasons that can not be internalized to be part of the regret . We give two concrete examples below . Dynamic pricing . When launching a new product , how does the market demand reacts to the price is typically unknown . Online learning is a popular tool for the firm to learn the demand and make profits simultaneously . See [ 21 ] for a comprehensive review . In this case , the arms are the prices charged by the firm . However , changing prices arbitrarily over time may have severe adverse effects . First , early adopters of the product are usually loyal customers . Price drops at a later stage may cause a backlash in this group of customers , whom the firm is the least willing to offend . Second , customers 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . are usually strategic , as studied extensively in the literature [ 39 , 32 , 44 , 31 , 16 ] . The possibility of a future discount may make consumers intentionally wait and hurt the revenue . A remedy for both issues is a markup pricing policy , in which the firm increases the price over time . This is particularly common for digital services growing rapidly : both Dropbox and Google Drive went through the phases from free , to low fees , to high fees . In a slightly different context , markup pricing is often used in successful crowdfunding campaigns : to reward early backers , the initial products may be given for free ; later after some goals are achieved , backers may get the product at a discount ; after the release , the product is sold at the full price . It is not a good idea to antagonize early backers by lowering the price later on . In the language of multi-armed bandit , it is translated to the requirement that the pulled arms in each period are increasing over time . Clinical trials . Early stage clinical trials for the optimal dose are usually conducted in Phase I/II trials sequentially . For Phase I , the toxicity of the doses is examined to determine a safe dose range . In this phase , the dose allocation commonly follows the “ dose escalation ” principle [ 30 ] and the “ 3+3 design ” is the most common dose allocation approach . For Phase II , the efficacy of the doses are tested within the safe dose range ( by random assignment , for example ) . Recently , especially in oncology , there are proposals to integrate both phases [ 42 ] which is called “ seamless Phase I/II trials. ” In this case , dose escalation has to be observed when looking for the most efficacious dose . Our study is thus motivated by seamless trials . In the online learning framework , the dosage is treated as arms and the efficacy is the unknown reward . Multi-armed bandit has been applied to the dose allocation to identify the optimal dose [ 41 , 7 ] . In particular , after a dose has been assigned to a cohort of patients , their responses are recorded before the next cohort is assigned a higher dose . If any of the patients experiences dose limiting toxicities ( DLTs ) , then the trial stops immediately . The dose escalation principle requires the allocated doses to be increasing over time . A recent paper incorporates the principle in the design of adaptive trials [ 13 ] . To accommodate the above applications , in this paper , we incorporate the requirement of monotone arm sequences in online learning , or more specifically , the continuum-armed bandit framework . We consider an unknown continuous function f ( x ) and the decision maker may choose an action/arm Xt in period t. The observed reward Zt is a random variable with mean f ( Xt ) . The requirement of monotone arm sequences is translated to X1 ≤ X2 ≤ · · · ≤ XT , where T is the length of the horizon . The goal of the decision maker is to minimize the regret T max f ( x ) − ∑T t=1 E [ f ( Xt ) ] . Main results . Next we summarize the main results of the paper informally . We first show that for a general continuous function f ( x ) , the extra requirement makes learning impossible . Result . For the class of continuous functions f ( x ) , the regret is at least T/4 . This diverges from the results in the continuum-armed bandit literature . In particular , [ 26 ] shows that the optimal regret for continuous functions scales with T 2/3 without the monotonicity requirement . Therefore , to prove meaningful results , we need to restrict the class of functions f ( x ) . From both the practical and theoretical points of view , we find that assuming f ( x ) to be unimodal or quasiconcave is reasonable . In practice , for the two applications mentioned above , the revenue function in dynamic pricing is typically assumed to be quasiconcave in the price [ 47 ] ; for clinical trials , the majority of the literature assumes the dose-response curve to be increasing [ 10 ] , increasing with a plateau effect [ 33 ] , or unimodal [ 45 ] , all of which satisfy the assumption . In theory , with the additional assumption , we can show that Result . There is an algorithm that has monotone arm sequences and achieves regret O ( log ( T ) T 3/4 ) for continuous and quasiconcave functions f ( x ) . Note that the rate T 3/4 is still different from that in the continuum-armed bandit literature . We then proceed to close the case by arguing that this is the fundamental limit of learning , not due to the design of the algorithm . Result . No algorithm can achieve less regret than T 3/4/32 for the instances we construct . The constructed instances that are hard to learn ( see Figure 2 ) are significantly different from the literature . In fact , the classic construction of “ needle in a haystack ” [ 27 ] does not leverage the requirement of monotone arm sequences and does not give the optimal lower bound . We prove the lower bound by using a novel random variable that is only a stopping time under monotone arm sequences . By studying the Kullback-Leibler divergence of the probabilities measures involving the stopping time , we are able to obtain the lower bound . We also show numerical studies that complement the theoretical findings . Related literature . The framework of this paper is similar to the continuum-armed bandit problems , which are widely studied in the literature . The earlier studies focus on univariate Lipschitz continuous functions [ 2 , 26 ] and identify the optimal rate of regret O ( T 2/3 ) . Later papers study various implications including smoothness [ 6 , 15 , 24 ] , the presence of constraints and convexity [ 9 , 43 ] , contextual information and high dimensions [ 38 , 12 ] , and so on . The typical reduction used in this stream of literature is to adopt a proper discretization scheme and translate the problem into discrete arms . In the regret analysis , the Lipschitz continuity or smoothness can be used to control the discretization error . Our setting is similar to the study in [ 26 ] with quasiconcave objective functions and the additional requirement of monotone arm sequences . The optimal rate of regret , as a result of the difference , deteriorates from T 2/3 to T 3/4 . This deterioration can be viewed as the cost of the monotonicity requirement . In a recent paper , [ 35 ] study a similar problem while assuming the function f ( x ) is strictly concave . The motivation is from the notion of “ fairness ” in sequential decision making [ 46 ] . The concavity allows them to develop an algorithm based on the gradient information . The resulting regret is O ( T 1/2 ) , the same as continuum-armed bandit with convexity/concavity [ 20 , 1 ] . Comparing to this set of results , the strong concavity prevents the deterioration of regret rates under the monotonicity requirement . In contrast , [ 18 , 19 ] show that the same rate of regret O ( T 1/2 ) can be achieved if the function is not strictly concave , but just unimodal or quasiconcave , under additional technical assumptions . It is worth noting that many results in this paper are discovered independently in a recent paper [ 25 ] . The comparison of the regret rates are shown in Table 1 . Other requirements on the arm sequence motivated by practical problems are also considered in the literature . [ 17 , 36 , 37 ] are motivated by a similar concern in dynamic pricing : customers are hostile to frequent price changes . They propose algorithms that have a limited number of switches and study the impact on the regret . Motivated by Phase I clinical trials , [ 22 ] study the best arm identification problem when the reward of the arms are monotone and the goal is to identify the arm closest to a threshold . [ 14 ] consider the setting when the reward is nonstationary and periodic , which is typical in the retail industry as demand exhibits seasonality . Notations . We use Z+ and R to denote all positive integers and real numbers , respectively . Let T ∈ Z+ be the length of the horizon . Let [ m ] : = { 1 , . . . , m } for any m ∈ Z+ . Let x∗ = argmaxx∈ [ 0,1 ] f ( x ) be the set of maximizers of a function f ( x ) in [ 0 , 1 ] . The indicator function is represented by 1 . 2 Problem Setup . The objective function f ( x ) : [ 0 , 1 ] → [ 0 , 1 ] is unknown to the decision maker . In each period t ∈ [ T ] , the decision maker chooses Xt ∈ [ 0 , 1 ] and collects a random reward Zt which is independent of everything else conditional on Xt and satisfies E [ Zt|Xt ] = f ( Xt ) . After T periods , the total ( random ) reward collected by the decision maker is thus ∑T t=1 Zt . Since the decision maker does not know f ( x ) initially , the arm chosen in period t only depends on the previously chosen arms and observed rewards . That is , for a policy π of the decision maker , we have Xt = πt ( X1 , Z1 , . . . , Xt−1 , Zt−1 , Ut ) , where Ut is an independent random variable associated with the internal randomizer of πt ( e.g. , Thompson sampling ) . Before proceeding , we first introduce a few standard assumptions . Assumption 1 . The function f ( x ) is c-Lipschitz continuous , i.e. , |f ( x1 ) − f ( x2 ) | ≤ c|x1 − x2| , for x1 , x2 ∈ [ 0 , 1 ] . This is a standard assumption in the continuum-armed bandit literature [ 26 , 6 , 11 , 38 ] . Indeed , without continuity , the decision maker has no hope to learn the objective function well . In this study , we do not consider other smoothness conditions . Assumption 2 . The reward Zt is σ-subgaussian for all t ∈ [ T ] . That is , conditional on Xt , E [ exp ( λ ( Zt − f ( Xt ) ) ) |Xt ] ≤ exp ( λ2σ2/2 ) for all λ ∈ R. Subgaussian random variables are the focus of most papers in the literature , due to their benign theoretical properties . See [ 29 ] for more details . We can now define an oracle that has the knowledge of f ( x ) . The oracle would choose arm x ∈ argmaxx∈ [ 0,1 ] f ( x ) and collect the total reward with mean T maxx∈ [ 0,1 ] f ( x ) . Because of Assumption 1 , the maximum is well defined . We define the regret of a policy π as Rf , π ( T ) = T max x∈ [ 0,1 ] f ( x ) − T∑ t=1 E [ f ( Xt ) ] . Note that the regret depends on the unknown function f ( x ) , which is not very meaningful . As a standard setup in the continuum-armed bandit problem , we consider the worst-case regret for all f satisfying Assumption 2 . Rπ ( T ) : = max f Rf , π ( T ) . Next we present the key requirement in this study : the sequence of arms must be increasing . Requirement 1 ( Increasing Arm Sequence ) . The sequence { Xt } Tt=1 must satisfy X1 ≤ X2 ≤ · · · ≤ XT almost surely under π . Note that “ increasing ” can be relaxed to “ monotone ” without much effort . For the simplicity of exposition , we only consider the increasing arm sequence . As explained in the introduction , this is a practical requirement in several applications . In phase I/II clinical trials , the quantity Xt is the dose applied to the t-th patient and f ( x ) is the average efficacy of dose x . The dose escalation principle protects the patients from DLTs . More precisely , the doses applied to the patients must increase gradually over time such that the trial can be stopped before serious side effects occur to a batch of patients under high doses . In dynamic pricing , Xt is the price charged for the t-th customer and f ( x ) is the expected revenue under price x . The increasing arm sequence corresponds to a markup pricing strategy that is typical for new products . Because of the increasing price , early adopters , who are usually loyal to the product , wouldn ’ t feel betrayed by the firm . Moreover , it deters strategic consumer behavior that may cause a large number of customers to wait for promotions , hurting the firm ’ s profits . We point out that under Assumption 1 and 2 , which typically suffice to guarantee sublinear regret , the regret Rπ ( T ) is at least T/4 under Requirement 1 for all policies π . Proposition 1 . For all π satisfying Requirement 1 , we have Rπ ( T ) ≥ T/4 under Assumptions 1 with c = 2 and Assumption 2 . The result can be easily illustrated by the two functions ( f1 ( x ) in the left and f2 ( x ) in the right ) in Figure 1 . Because of Requirement 1 and the fact that f1 ( x ) = f2 ( x ) for x ∈ [ 0 , 0.5 ] , we must have Rf1 , π ( T ) ≥ T 2 − 1 2 T∑ t=1 1Xt≤0.5 , Rf2 , π ( T ) ≥ T − T∑ t=1 1Xt > 0.5 − 1 2 T∑ t=1 1Xt≤0.5 . Recall the fact ∑T t=1 1Xt≤0.5 + ∑T t=1 1Xt > 0.5 = T . The above inequalities lead to Rf1 , π ( T ) + Rf2 , π ( T ) ≥ T/2 , which implies Proposition 1 . Therefore , to obtain sublinear regret , we need additional assumptions for f ( x ) . In particular , we assume Assumption 3 . The function f ( x ) is quasiconcave , i.e. , for any x1 , x2 , λ ∈ [ 0 , 1 ] , we have f ( λx1 + ( 1− λ ) x2 ) ≥ min { f ( x1 ) , f ( x2 ) } . In other words , a quasiconcave function is weakly unimodal and does not have multiple separated local maximums . Therefore , f2 ( x ) in Figure 1 is not quasiconcave while f1 ( x ) is . Note that a quasiconcave function may well have a plateau around the local maximum . We impose Assumption 3 on the function class due to two reasons . First , quasiconcavity is probably the weakest technical assumption one can come up with that is compatible with Requirement 1 . It is much weaker than concavity [ 35 ] , and it is easy to see from Figure 1 that multiple local maximums tend to cause linear regret under Requirement 1 . Moreover , unimodal functions are the focus of other studies in the online learning literature [ 18 , 19 ] . Second , objective functions in practice are commonly assumed to be quasiconcave . For the dynamic pricing example , the revenue function is typically assumed to be quasiconcave in the price [ 47 ] , supported by theoretical and empirical evidence . In clinical trials , most literature assumes the dose-response curve to be quasiconcave [ 10 , 45 , 33 ] .

Summary: The paper studies the continuum-armed bandit problem with additional requirement that the selected arms/actions must be monotone sequences over the course of the game. With this requirement, the authors first show that simply considering a continuous reward function (namely, an unknown function that maps the action to the reward) will suffer linear regrets, in contrast to the sublinear regret $O(T^{2/3})$ in the literature without the monotonicity requirement.  Then the authors consider a more structured reward function which is quasiconcave. The authors then provide an algorithm and prove that a regret $O(T^{3/4})$ of this algorithm for this class of functions. Furthermore, this regret is tight as authors also provide a matching lower bound. Lastly, the authors conduct numerical studies to compare the regret of proposed algorithm with a few benchmark, validating the analysis.