A Reduction-Based Framework for Conservative Bandits and Reinforcement Learning

1 INTRODUCTION . This paper studies online sequential decision making problems such as bandits and reinforcement learning ( RL ) subject to a conservative constraint . Specifically , the agent is given a reliable baseline policy that may not be optimal but still satisfactory . In conservative bandits and RL , the agent is asked to perform nearly as well ( or better ) as the baseline policy at all time . This setting is a natural formalization of many real-world problems such as digital marketing , healthcare , finance , etc . For example , a company may want to explore new strategies to maximize profit while simultaneously maintaining profit above a fixed baseline at any time , in order not to be bankrupted . See ( Wu et al. , 2016 ) for more discussions on the motivation of the conservative constraint . Analogously to the non-conservative case , conservative bandit/RL problems also require us to balance exploration and exploitation carefully . Meanwhile , to ensure the obtained policies outperform the baseline policy , we need to provide a tractable approach to keep the exploration not too aggressive . Solving these two problems simultaneously is the key challenge in conservative bandits and RL . Existing work proposed algorithms for different settings , including bandits ( Wu et al. , 2016 ; Kazerouni et al. , 2016 ; Garcelon et al. , 2020b ; Katariya et al. , 2019 ; Zhang et al. , 2019 ; Du et al. , 2020 ; Wang et al. , 2021 ) and tabular RL ( Garcelon et al. , 2020a ) . However , lower bound exists only for the multiarmed bandit ( MAB ) setting ( Wu et al. , 2016 ) , and there is no lower bound for other widely-adopted settings , such as linear bandits , tabular Markov Decision Process ( MDP ) and low-rank MDP . In Section 1.3 , we provide a more detailed discussion of the related work . For each of the different settings considered in the literature ( i.e. , multi-armed bandits , linear bandits , tabular MDPs ) , existing approaches rely on ad-hoc algorithm design and analysis of the trade-off between the setting-specific regret analysis and the conservative constraint . Furthermore , it is hard to argue about the optimality of the proposed algorithms because it would require clever constructions of the hard instances to prove the non-trivial regret lower bounds under the conservative constraint . 1.1 OUR CONTRIBUTIONS . In this paper , we address these limitations and make significant progress in studying the general problem of online sequential decision-making with conservative constraint . We propose a unified framework that is generally applicable to online sequential decision-making problems . The common theme underlying our framework is to calculate the necessary and sufficient budget required to enable non-conservative exploration . Such a budget is obtained by running the baseline policy ( cf . Section 3 ) . With the new framework , we obtain a novel upper bound on tabular MDPs , which improves the previous result . And we prove a new upper bound on low-rank MDPs . Also , we derive the first lower bounds for linear bandits , tabular and low-rank MDPs , which shows that our upper bound is tight . Lower Bounds . For any specific problem ( e.g. , multi-armed bandits , linear bandits ) , our framework immediately turns a minimax lower bound of the non-conservative setting to a non-trivial lower bound for the conservative case ( cf . Section 4 ) . We list some examples to showcase the power of our framework for lower bounds . Full results are given in Table 1 . • We derive a novel lower bound for multi-armed bandits that works on a wider range of parameters than the one derived in ( Wu et al. , 2016 ) . In particular , our lower bound shows a more refined dependence on the value of the baseline policy . • We derive the first regret lower bound for conservative exploration in linear bandits , tabular MDPs and low-rank MDPs . These results allow to establish or disprove the optimality of the algorithms currently available in the literature . We emphasize our technique for deriving lower bounds is simple and generic , so we believe it can be used to obtain lower bounds for other problems as well . Upper Bounds . Our novel view of conservative exploration can also be used to derive high probability regret upper-bounds . When the suboptimality gap 0 and the expected return µ0 of the baseline policy are known , we show that the Budget-Exploration algorithm ( Alg . 1 ) attains minimax optimal regret in a wide variety of sequential decision-making problems , when associated to any minimax optimal non-conservative algorithm specific to the problem at hand . In the more realistic ( and challenging ) scenario where 0 and µ0 are unknown , we show how to simply convert an entire class of algorithms with a sublinear non-conservative regret bound into a conservative algorithms with a sublinear regret bound . We obtain the following results , full details are given in Table 1 . • In the MAB setting , we obtain a regret upper-bound that matches our refined lower-bound , thus improving on existing analysis . In the linear bandit setting , we match existing bounds that are already minimax optimal . • In the RL setting , we provide two novel results . First , we provide the first minimax optimal result for tabular MDPs , improving over ( Garcelon et al. , 2020a ) . Second , we derive the first upper bound for conservative exploration in low-rank MDPs . Our bound matches the rate of existing non-conservative algorithms though it is not minimax optimal . How to achieve minimax optimality in low rank MDPs is an open question even in non-conservative exploration . Again , our reduction technique is simple and generic , and can be used to obtain new results in previously unstudied settings , like we did for low-rank MDPs . 1.2 MAIN DIFFICULTIES AND TECHNIQUE OVERVIEW . 1.2.1 LOWER BOUNDS . The only lower bound for conservative exploration is by Wu et al . ( 2016 ) who followed a classical approach in the bandit literature . They constructed a class of hard environments and used an information-theoretic argument to prove the lower bound . Construction of hard environments is highly non-trivial because one needs to incorporate the hardness from the conservative constraint . It is also non-trivial to generalize Wu et al . ( 2016 ) ’ s lower bound to other settings such as conservative linear bandits and RL because one will need new constructions of hard environments for different settings . We note that new constructions are needed even for non-conservative settings , because simply embedding the hard instances of MAB to other settings can not give the tightest lower bounds . See , e.g. , Chapter 24 of Lattimore & Szepesvári ( 2020 ) and Domingues et al . ( 2021 ) . In this paper , We use a completely different approach . Our key insights are 1 ) relating the necessary budget to the regret lower bounds of non-conservative sequential decision-making problems , and 2 ) obtaining sharp lower bounds in the conservative settings via maximizing a quadratic function ( cf . Equation ( 6 ) ) . Comparing with the classical approach , our approach is simpler and more general : ours does not need problem-specific constructions and can automatically transform any lower bound in a non-conservative problem to the corresponding conservative problem . See Section 4 for details . 1.2.2 UPPER BOUNDS . Improvement over Wu et al . ( 2016 ) when 0 is known . When 0 is known , Wu et al . ( 2016 ) proposed an algorithm ( BudgetFirst ) which first plays the baseline policy for enough times and then plays an non-conservative MAB algorithm . However , their regret bound is not tight because their analysis on the required budget is loose : they accumulate enough budget to play T -step exploration where T is the total number of rounds . Our main technical insight to obtain the tight regret bound is a sharp analysis on the required budget : by relating the minimax regret upper bounds of UCB algorithms , we show the required budget can be independent of T . See Section 5 and F for details . Sharp upper bounds with unknown 0 . When 0 is unknown , the paper by Wu et al . ( 2016 ) , its follow-up papers ( Kazerouni et al. , 2016 ; Garcelon et al. , 2020b ; Zhang et al. , 2019 ; Garcelon et al. , 2020a ) , and our paper , all adopt the same algorithmic template : 1 ) build an online estimate on the lower bound performance of each possible exploration policy , and 2 ) based on the estimated lower bounds , choose an exploration policy or play the baseline policy . The key difference and the most non-trivial part in different papers is how to analyze T0 ( the number of times of executing the baseline policy ) . Exiting works upper bound T0 by relating it to the decision criterion for whether to choose the baseline policy or not . Since for different problem settings , the criteria have different forms , existing papers adopt different problem-specific analyses , and in some settings , the analyses are not tight ( e.g. , MAB and tabular RL ) . Our analysis approach is different from existing ones : we bound T0 via maximizing a quadratic function that depends on the minimax regret bounds of non-conservative algorithms and the conservative constraint . See Section 5 for more details . 1.3 RELATED WORK . Non-conservative exploration has been widely studied in bandits , and minimax optimal algorithms have been provided for the settings considered in this paper ( e.g . Lattimore & Szepesvári , 2020 ) . The exploration problem has been widely studied also in RL but minimax optimal algorithms have not been provided for all the settings . For any finite-horizon time-inhomogeneous MDP with S states , A actions and horizon H , the minimax regret lower bound is ⌦ ( p H3SAT ) ( Domingues et al. , 2021 ) , where T denotes the number of episodes . For any time-inhomogeneous low-rank MDP with d-dimensional linear representation , the lower-bound is ⌦ ( p d2H3T ) ( Zhou et al. , 2020 , Remark 2Although the lower bound in Wu et al . ( 2016 ) seems tighter , they require a condition 0 ↵µ0+ 0 0.9 . Under this condition , our lower bound is the same as theirs . Thus ours is more general . See Appendix E. 3In ( Garcelon et al. , 2020a ) , the upper bound scales with rb = mins2S , ⇢0 ( s ) > 0 V ⇡0 1 ( s ) ( with ⇢0 the distribution of the starting state ) , the minimum of the baseline ’ s value function at the first step over the potential starting states.Here , we assume there is a unique starting state hence rb = V ⇡0 . 5.8 ) . While several minimax optimal algorithms have been provided for tabular MDPs ( e.g . Azar et al. , 2017 ; Zanette & Brunskill , 2019 ; Zhang et al. , 2020a ; b ; Ménard et al. , 2021 ) , the gap between upper bound and lower bound is still open in low-rank MDPs , where LSVI-UCB ( Jin et al. , 2020 ) attains a eO ( p d3H4T ) , while ELEANOR ( Zanette et al. , 2020 ) improves to eO ( p d2H4T ) . In conservative exploration , previous works focus on designing specific conservative algorithms for different settings . This conservative scenario was studied in multi-armed bandits ( Wu et al. , 2016 ) , contextual linear bandits ( Kazerouni et al. , 2016 ; Garcelon et al. , 2020b ) , contextual combinatorial bandits ( Zhang et al. , 2019 ) and tabular MDPs ( Garcelon et al. , 2020a ) . All these works focused on providing an upper-bound to the regret of a conservative algorithm . Other problems that have been considered in conservative exploration are combinatorial semi-bandit with exchangeable actions ( Katariya et al. , 2019 ) and contextual combinatorial cascading bandits ( Wang et al. , 2021 ) . Du et al . ( 2020 ) have recently considered conservative exploration with sample-path constraint . Our work is also related to safe bandits/RL ( Amani et al. , 2019 ; Pacchiano et al. , 2021 ; Amani et al. , 2021 ) and constrained RL ( Altman , 1999 ; Efroni et al. , 2020 ; Ding et al. , 2020 ; 2021 ) . The setting of safe bandits/RL is different from conservative bandits/RL . Specifically , the safety constraint requires that the expected cost at each stage is below a certain threshold . This constraint is stage-wise , and is independent of the history . On the contrary , the conservative constraint requires that the total reward is not too small . For the constrained MDP , the goal is to maximize the expected reward value subject to a constraint on the expected utility value ( value function with respect to another reward function ) . In conservative RL , however , the agnet aims to maximize the expected reward value subject to the constaint that the ( same ) reward value is not significantly worse that of the baseline policy .

Summary: This paper proposes a reduction-based framework for a large class of reinforcement learning algorithms, including bandits, linear bandits, tabular MDP and linear MDP. The authors notably propose a generic lower bound that holds for all the studied class of algorithms. The lower bound is built on the regret decomposition between the regret of the conservative baseline during the time when the budget is not reached, and the regret of a non-conservative algorithm that learns on the baseline. While the obtained lower in the bandit setting is not as tight as the one obtained in (Wu et al, 2016), this lower bound holds for a larger class of algorithms. Then, two generic algorithms are proposed for handling conservative reinforcement learning. Budget-Exploration consists in learning the non-conservative algorithm on the conservative baseline during a fixed period of time and then to play the non-conservative algorithm. The regret upper bound of Budget-Exploration matches the lower bound, but the knowledge of the baseline gap is need. The second algorithm LCBCE does not necessitate that the baseline gap be known. The idea of the algorithm is to compute an upper bound of the budget thanks to the lower bounds of rewards obtained by the non-conservative algorithm run on the baseline. The regret upper bound of LCBCE still matches the proposed lower bound.