A Reduction-Based Framework for Conservative Bandits and Reinforcement Learning

1 INTRODUCTION . This paper studies online sequential decision making problems such as bandits and reinforcement learning ( RL ) subject to a conservative constraint . Specifically , the agent is given a reliable baseline policy that may not be optimal but still satisfactory . In conservative bandits and RL , the agent is asked to perform nearly as well ( or better ) as the baseline policy at all time . This setting is a natural formalization of many real-world problems such as digital marketing , healthcare , finance , etc . For example , a company may want to explore new strategies to maximize profit while simultaneously maintaining profit above a fixed baseline at any time , in order not to be bankrupted . See ( Wu et al. , 2016 ) for more discussions on the motivation of the conservative constraint . Analogously to the non-conservative case , conservative bandit/RL problems also require us to balance exploration and exploitation carefully . Meanwhile , to ensure the obtained policies outperform the baseline policy , we need to provide a tractable approach to keep the exploration not too aggressive . Solving these two problems simultaneously is the key challenge in conservative bandits and RL . Existing work proposed algorithms for different settings , including bandits ( Wu et al. , 2016 ; Kazerouni et al. , 2016 ; Garcelon et al. , 2020b ; Katariya et al. , 2019 ; Zhang et al. , 2019 ; Du et al. , 2020 ; Wang et al. , 2021 ) and tabular RL ( Garcelon et al. , 2020a ) . However , lower bound exists only for the multiarmed bandit ( MAB ) setting ( Wu et al. , 2016 ) , and there is no lower bound for other widely-adopted settings , such as linear bandits , tabular Markov Decision Process ( MDP ) and low-rank MDP . In Section 1.3 , we provide a more detailed discussion of the related work . For each of the different settings considered in the literature ( i.e. , multi-armed bandits , linear bandits , tabular MDPs ) , existing approaches rely on ad-hoc algorithm design and analysis of the trade-off between the setting-specific regret analysis and the conservative constraint . Furthermore , it is hard to argue about the optimality of the proposed algorithms because it would require clever constructions of the hard instances to prove the non-trivial regret lower bounds under the conservative constraint . 1.1 OUR CONTRIBUTIONS . In this paper , we address these limitations and make significant progress in studying the general problem of online sequential decision-making with conservative constraint . We propose a unified framework that is generally applicable to online sequential decision-making problems . The common theme underlying our framework is to calculate the necessary and sufficient budget required to enable non-conservative exploration . Such a budget is obtained by running the baseline policy ( cf . Section 3 ) . With the new framework , we obtain a novel upper bound on tabular MDPs , which improves the previous result . And we prove a new upper bound on low-rank MDPs . Also , we derive the first lower bounds for linear bandits , tabular and low-rank MDPs , which shows that our upper bound is tight . Lower Bounds . For any specific problem ( e.g. , multi-armed bandits , linear bandits ) , our framework immediately turns a minimax lower bound of the non-conservative setting to a non-trivial lower bound for the conservative case ( cf . Section 4 ) . We list some examples to showcase the power of our framework for lower bounds . Full results are given in Table 1 . • We derive a novel lower bound for multi-armed bandits that works on a wider range of parameters than the one derived in ( Wu et al. , 2016 ) . In particular , our lower bound shows a more refined dependence on the value of the baseline policy . • We derive the first regret lower bound for conservative exploration in linear bandits , tabular MDPs and low-rank MDPs . These results allow to establish or disprove the optimality of the algorithms currently available in the literature . We emphasize our technique for deriving lower bounds is simple and generic , so we believe it can be used to obtain lower bounds for other problems as well . Upper Bounds . Our novel view of conservative exploration can also be used to derive high probability regret upper-bounds . When the suboptimality gap 0 and the expected return µ0 of the baseline policy are known , we show that the Budget-Exploration algorithm ( Alg . 1 ) attains minimax optimal regret in a wide variety of sequential decision-making problems , when associated to any minimax optimal non-conservative algorithm specific to the problem at hand . In the more realistic ( and challenging ) scenario where 0 and µ0 are unknown , we show how to simply convert an entire class of algorithms with a sublinear non-conservative regret bound into a conservative algorithms with a sublinear regret bound . We obtain the following results , full details are given in Table 1 . • In the MAB setting , we obtain a regret upper-bound that matches our refined lower-bound , thus improving on existing analysis . In the linear bandit setting , we match existing bounds that are already minimax optimal . • In the RL setting , we provide two novel results . First , we provide the first minimax optimal result for tabular MDPs , improving over ( Garcelon et al. , 2020a ) . Second , we derive the first upper bound for conservative exploration in low-rank MDPs . Our bound matches the rate of existing non-conservative algorithms though it is not minimax optimal . How to achieve minimax optimality in low rank MDPs is an open question even in non-conservative exploration . Again , our reduction technique is simple and generic , and can be used to obtain new results in previously unstudied settings , like we did for low-rank MDPs . 1.2 MAIN DIFFICULTIES AND TECHNIQUE OVERVIEW . 1.2.1 LOWER BOUNDS . The only lower bound for conservative exploration is by Wu et al . ( 2016 ) who followed a classical approach in the bandit literature . They constructed a class of hard environments and used an information-theoretic argument to prove the lower bound . Construction of hard environments is highly non-trivial because one needs to incorporate the hardness from the conservative constraint . It is also non-trivial to generalize Wu et al . ( 2016 ) ’ s lower bound to other settings such as conservative linear bandits and RL because one will need new constructions of hard environments for different settings . We note that new constructions are needed even for non-conservative settings , because simply embedding the hard instances of MAB to other settings can not give the tightest lower bounds . See , e.g. , Chapter 24 of Lattimore & Szepesvári ( 2020 ) and Domingues et al . ( 2021 ) . In this paper , We use a completely different approach . Our key insights are 1 ) relating the necessary budget to the regret lower bounds of non-conservative sequential decision-making problems , and 2 ) obtaining sharp lower bounds in the conservative settings via maximizing a quadratic function ( cf . Equation ( 6 ) ) . Comparing with the classical approach , our approach is simpler and more general : ours does not need problem-specific constructions and can automatically transform any lower bound in a non-conservative problem to the corresponding conservative problem . See Section 4 for details . 1.2.2 UPPER BOUNDS . Improvement over Wu et al . ( 2016 ) when 0 is known . When 0 is known , Wu et al . ( 2016 ) proposed an algorithm ( BudgetFirst ) which first plays the baseline policy for enough times and then plays an non-conservative MAB algorithm . However , their regret bound is not tight because their analysis on the required budget is loose : they accumulate enough budget to play T -step exploration where T is the total number of rounds . Our main technical insight to obtain the tight regret bound is a sharp analysis on the required budget : by relating the minimax regret upper bounds of UCB algorithms , we show the required budget can be independent of T . See Section 5 and F for details . Sharp upper bounds with unknown 0 . When 0 is unknown , the paper by Wu et al . ( 2016 ) , its follow-up papers ( Kazerouni et al. , 2016 ; Garcelon et al. , 2020b ; Zhang et al. , 2019 ; Garcelon et al. , 2020a ) , and our paper , all adopt the same algorithmic template : 1 ) build an online estimate on the lower bound performance of each possible exploration policy , and 2 ) based on the estimated lower bounds , choose an exploration policy or play the baseline policy . The key difference and the most non-trivial part in different papers is how to analyze T0 ( the number of times of executing the baseline policy ) . Exiting works upper bound T0 by relating it to the decision criterion for whether to choose the baseline policy or not . Since for different problem settings , the criteria have different forms , existing papers adopt different problem-specific analyses , and in some settings , the analyses are not tight ( e.g. , MAB and tabular RL ) . Our analysis approach is different from existing ones : we bound T0 via maximizing a quadratic function that depends on the minimax regret bounds of non-conservative algorithms and the conservative constraint . See Section 5 for more details . 1.3 RELATED WORK . Non-conservative exploration has been widely studied in bandits , and minimax optimal algorithms have been provided for the settings considered in this paper ( e.g . Lattimore & Szepesvári , 2020 ) . The exploration problem has been widely studied also in RL but minimax optimal algorithms have not been provided for all the settings . For any finite-horizon time-inhomogeneous MDP with S states , A actions and horizon H , the minimax regret lower bound is ⌦ ( p H3SAT ) ( Domingues et al. , 2021 ) , where T denotes the number of episodes . For any time-inhomogeneous low-rank MDP with d-dimensional linear representation , the lower-bound is ⌦ ( p d2H3T ) ( Zhou et al. , 2020 , Remark 2Although the lower bound in Wu et al . ( 2016 ) seems tighter , they require a condition 0 ↵µ0+ 0 0.9 . Under this condition , our lower bound is the same as theirs . Thus ours is more general . See Appendix E. 3In ( Garcelon et al. , 2020a ) , the upper bound scales with rb = mins2S , ⇢0 ( s ) > 0 V ⇡0 1 ( s ) ( with ⇢0 the distribution of the starting state ) , the minimum of the baseline ’ s value function at the first step over the potential starting states.Here , we assume there is a unique starting state hence rb = V ⇡0 . 5.8 ) . While several minimax optimal algorithms have been provided for tabular MDPs ( e.g . Azar et al. , 2017 ; Zanette & Brunskill , 2019 ; Zhang et al. , 2020a ; b ; Ménard et al. , 2021 ) , the gap between upper bound and lower bound is still open in low-rank MDPs , where LSVI-UCB ( Jin et al. , 2020 ) attains a eO ( p d3H4T ) , while ELEANOR ( Zanette et al. , 2020 ) improves to eO ( p d2H4T ) . In conservative exploration , previous works focus on designing specific conservative algorithms for different settings . This conservative scenario was studied in multi-armed bandits ( Wu et al. , 2016 ) , contextual linear bandits ( Kazerouni et al. , 2016 ; Garcelon et al. , 2020b ) , contextual combinatorial bandits ( Zhang et al. , 2019 ) and tabular MDPs ( Garcelon et al. , 2020a ) . All these works focused on providing an upper-bound to the regret of a conservative algorithm . Other problems that have been considered in conservative exploration are combinatorial semi-bandit with exchangeable actions ( Katariya et al. , 2019 ) and contextual combinatorial cascading bandits ( Wang et al. , 2021 ) . Du et al . ( 2020 ) have recently considered conservative exploration with sample-path constraint . Our work is also related to safe bandits/RL ( Amani et al. , 2019 ; Pacchiano et al. , 2021 ; Amani et al. , 2021 ) and constrained RL ( Altman , 1999 ; Efroni et al. , 2020 ; Ding et al. , 2020 ; 2021 ) . The setting of safe bandits/RL is different from conservative bandits/RL . Specifically , the safety constraint requires that the expected cost at each stage is below a certain threshold . This constraint is stage-wise , and is independent of the history . On the contrary , the conservative constraint requires that the total reward is not too small . For the constrained MDP , the goal is to maximize the expected reward value subject to a constraint on the expected utility value ( value function with respect to another reward function ) . In conservative RL , however , the agnet aims to maximize the expected reward value subject to the constaint that the ( same ) reward value is not significantly worse that of the baseline policy .

Summary: This paper studies bandits and RL settings subject to a conservative constraint where the agent has to perform at least as well as a given baseline policy. It improves the existing lower bound for conservative MAB, and as the main contribution, obtains new lower bounds for conservative linear bandits, tabular RL and low-rank MDP. It also provides new upper bounds matching existing ones with different analyses.