A Simple Reward-free Approach to Constrained Reinforcement Learning

In constrained reinforcement learning ( RL ) , a learning agent seeks to not only optimize the overall reward but also satisfy the additional safety , diversity , or budget constraints . Consequently , existing constrained RL solutions require several new algorithmic ingredients that are notably different from standard RL . On the other hand , reward-free RL is independently developed in the unconstrained literature , which learns the transition dynamics without using the reward information , and thus naturally capable of addressing RL with multiple objectives under the common dynamics . This paper bridges reward-free RL and constrained RL . Particularly , we propose a simple meta-algorithm such that given any reward-free RL oracle , the approachability and constrained RL problems can be directly solved with negligible overheads in sample complexity . Utilizing the existing reward-free RL solvers , our framework provides sharp sample complexity results for constrained RL in the tabular MDP setting , matching the best existing results up to a factor of horizon dependence ; our framework directly extends to a setting of tabular two-player Markov games , and gives a new result for constrained RL with linear function approximation . 1 INTRODUCTION . In a wide range of modern reinforcement learning ( RL ) applications , it is not sufficient for the learning agents to only maximize a scalar reward . More importantly , they must satisfy various constraints . For instance , such constraints can be the physical limit of power consumption or torque in motors for robotics tasks ( Tessler et al. , 2019 ) ; the budget for computation and the frequency of actions for real-time strategy games ( Vinyals et al. , 2019 ) ; and the requirement for safety , fuel efficiency and human comfort for autonomous drive ( Le et al. , 2019 ) . In addition , constraints are also crucial in tasks such as dynamic pricing with limited supply ( Besbes & Zeevi , 2009 ; Babaioff et al. , 2015 ) , scheduling of resources on a computer cluster ( Mao et al. , 2016 ) , imitation learning ( Syed & Schapire , 2007 ; Ziebart et al. , 2008 ; Sun et al. , 2019 ) , as well as RL with fairness ( Jabbari et al. , 2017 ) . These huge demand in practice gives rise to a subfield—constrained RL , which focuses on designing efficient algorithms to find near-optimal policies for RL problems under linear or general convex constraints . Most constrained RL works directly combine the existing techniques such as value iteration and optimism from unconstrained literature , with new techniques specifically designed to deal with linear constraints ( Efroni et al. , 2020 ; Ding et al. , 2021 ; Qiu et al. , 2020 ) or general convex constraints ( Brantley et al. , 2020 ; Yu et al. , 2021 ) . The end product is a single new complex algorithm which is tasked to solve all the challenges of learning dynamics , exploration , planning as well as constraints satisfaction simultaneously . Thus , these algorithms need to be re-analyzed from scratch , and it is highly nontrivial to translate the progress in the unconstrained RL to the constrained setting . On the other hand , reward-free RL—proposed in Jin et al . ( 2020a ) —is a framework for the unconstrained setting , which learns the transition dynamics without using the reward . The framework has two phases : in the exploration phase , the agent first collects trajectories from a Markov decision process ( MDP ) and learns the dynamics without a pre-specified reward function . After exploration , the agent is tasked with computing near-optimal policies under the MDP for a collection of given reward functions . This framework is particularly suitable when there are multiple reward functions of interest , and has been developed recently to attack various settings including tabular MDPs ( Jin et al. , 2020a ; Zhang et al. , 2020 ) , linear MDPs ( Wang et al. , 2020 ; Zanette et al. , 2020 ) , and tabular Markov games ( Liu et al. , 2020 ) . Contribution . In this paper , we propose a simple approach to solve constrained RL problems by bridging the reward-free RL literature and constrained RL literature . Our approach isolates the challenges of constraint satisfaction , and leaves the remaining RL challenges such as learning dynamics and exploration to reward-free RL . This allows us to design a new algorithm which purely focuses on addressing the constraints . Formally , we design a meta-algorithm for RL problems with general convex constraints . Our meta-algorithm takes a reward-free RL solver , and can be used to directly solve the approachability problem , as well as the constrained MDP problems using very small amount of samples in addition to what is required for reward-free RL . Our framework enables direct translation of any progress in reward-free RL to constrained RL . Leveraging recent advances in reward-free RL , our meta-algorithm directly implies sample-efficient guarantees of constrained RL in the settings of tabular MDP , linear MDP , as well as tabular two-player Markov games . In particular , • Tabular setting : Our work achieves sample complexity of Õ ( min { d , S } H4SA/ 2 ) for all three tasks of reward-free RL for Vector-valued MDPs ( VMDP ) , approachability , and RL with general convex constraints . Here d is the dimension of VMDP or the number of constraints , S , A are the number of states and actions , H is the horizon , and is the error tolerance . It matches the best existing results up to a factor of H . • Linear setting : Our work provides new sample complexity of Õ ( d3linH6/ 2 ) for all three tasks above for linear MDPs . To our best knowledge , this result is the first sample-efficient result for approachability and also constrained RL with general convex constraints in the linear function approximation setting . • Two-player setting : Our work extends to the setting of tabular two-player vector-valued Markov games and achieves low regret of α ( T ) = O ( /2 + √ H2ι/T ) at the cost of this O ( ) bias in regret as well as additional samples for preprocessing . 1.1 RELATED WORK . In this section , we review the related works on three tasks studied in this paper—reward-free RL , approachability , and constrained RL . Reward-free RL . Reward-free exploration has been formalized by Jin et al . ( 2020a ) for the tabular setting . Furthermore , Jin et al . ( 2020a ) proposed an algorithm which has sample complexity Õ ( poly ( H ) S2A/ 2 ) outputting -optimal policy for arbitrary number of reward functions . More 1The presented sample complexities are all under the L2 normalization conditions as studied in this paper . We comment that the results of ( Wu et al. , 2020 ; Brantley et al. , 2020 ; Yu et al. , 2021 ) are originally presented under L1/L∞ normalization conditions . While the results in Wu et al . ( 2020 ) can be directly adapted to our setting as stated in the table , the other two results Brantley et al . ( 2020 ) ; Yu et al . ( 2021 ) will be no better than the displayed results after adaptation . recently , Zhang et al . ( 2020 ) ; Liu et al . ( 2020 ) propose algorithm VI-Zero with sharp sample complexity of Õ ( poly ( H ) log ( N ) SA/ 2 ) capable of handling N fixed reward functions . Wang et al . ( 2020 ) ; Zanette et al . ( 2020 ) further provide reward-free learning results in the setting of linear function approximation , in particular , Wang et al . ( 2020 ) guarantees to find the near-optimal policies for an arbitrary number of ( linear ) reward functions within a sample complexity of Õ ( poly ( H ) d3lin/ 2 ) . All results mentioned above are for scalar-valued MDPs . For the vector-valued MDPs ( VMDPs ) , very recent work of Wu et al . ( 2020 ) designs a reward-free algorithm with sample complexity guarantee Õ ( poly ( H ) min { d , S } SA/ 2 ) in the tabular setting . Compared to Wu et al . ( 2020 ) , our reward-free algorithms for VMDP is adapted from the VI-Zero algorithm presented in Liu et al . ( 2020 ) ; While achieving the same sample complexity , it allows arbitrary planning algorithms in the planning phase . Approachability and Constrained RL Approachability and Constrained RL are two related tasks involving constraints . Inspired by Blackwell approachability ( Blackwell et al. , 1956 ) , recent work of Miryoosefi et al . ( 2019 ) introduces approachability task for VMDPs . However , the proposed algorithm does not have polynomial sample complexity guarantees . More recently , Yu et al . ( 2021 ) gave a new algorithm for approachability for both VMDPs and vector-valued Markov games ( VMGs ) . Yu et al . ( 2021 ) provides regret bounds for the proposed algorithm resulting in sample complexity guarantees of Õ ( poly ( H ) min { d , S } SA/ 2 ) for approachability in VMDPs and Õ ( poly ( H ) min { d , S } SAB/ 2 ) for approachability in VMGs . Sample-efficient exploration in constrained reinforcement learning has been recently studied in a recent line of work by Brantley et al . ( 2020 ) ; Qiu et al . ( 2020 ) ; Efroni et al . ( 2020 ) ; Ding et al . ( 2021 ) ; Singh et al . ( 2020 ) . All these works are also limited to linear constraints except Brantley et al . ( 2020 ) which extends their approach to general convex constraints achieving sample complexity of Õ ( poly ( H ) d2S2A/ 2 ) . However , Brantley et al . ( 2020 ) requires solving a large-scale convex optimization sub-problem . The best result for constrained RL with general convex constraints can be achieved by the approachability-based algorithm in Yu et al . ( 2021 ) obtaining sample complexity of Õ ( poly ( H ) min { d , S } SA/ 2 ) . Technically , our meta-algorithm is based on the Fenchel ’ s duality , which is similar to Yu et al . ( 2021 ) . In contrast , Yu et al . ( 2021 ) does not use reward-free RL , and is thus different from our results in terms of algorithmic approaches . Consequently , Yu et al . ( 2021 ) does not reveal the deep connections between reward-free RL and constrained RL , which is one of the main contribution of this paper . In addition , Yu et al . ( 2021 ) does not address the function approximation setting . Finally , we note that among all results mentioned above , only Ding et al . ( 2021 ) has considered models beyond tabular setting in the context of constrained RL . The model studied in Ding et al . ( 2021 ) is known as linear mixture MDPs which is different and incomparable to the linear MDP models considered in this paper . We further comment that Ding et al . ( 2021 ) can only handle linear constraints for CMDP , while our results is capable of solving CMDPs with general convex constraints . 2 PRELIMINARIES AND PROBLEM SETUP . We consider an episodic vector-valued Markov decision process ( VMDP ) specified by a tuple M = ( S , A , H , P , r ) , where S is the state space , A is the action space , H is the length of each episode , P = { Ph } Hh=1 is the collection of unknown transition probabilities with Ph ( s′ | s , a ) equal to the probability of transiting to s′ after taking action a in state s at the hth step , and r = { rh : S × A → B ( 1 ) } Hh=1 is a collection of unknown d-dimensional return functions , where B ( r ) is the d-dimensional Euclidean ball of radius r centered at the origin . Interaction protocol . In each episode , agent starts at a fixed initial state s1 . Then , at each step h ∈ [ H ] , the agent observes the current state sh , takes action ah , receives stochastic sample of the return vector rh ( sh , ah ) , and it causes the environment to transit to sh+1 ∼ Ph ( · | sh , ah ) . We assume that stochastic samples of the return function are also in B ( 1 ) , almost surely . Policy and value function . A policy π of an agent is a collection of H functions { πh : S → ∆ ( A ) } Hh=1 that map states to distribution over actions . The agent following policy π , picks action ah ∼ πh ( sh ) at the hth step . We denote Vπh : S → B ( H ) as the value func- tion at step h for policy π , defined as Vπh ( s ) : = Eπ [ ∑H h′=h rh′ ( sh′ , ah′ ) | sh = s ] . Similarly , we denote Qπh : S × A → B ( H ) as the Q-value function at step h for policy π , where Qπh ( s , a ) : = Eπ [ ∑H h′=h rh′ ( sh′ , ah′ ) | sh = s , ah = a ] . Scalarized MDP . For a VMDP M and θ ∈ B ( 1 ) , we define scalar-valued MDP Mθ = ( S , A , H , P , rθ ) , where rθ = { 〈θ , rh〉 : S ×A → [ −1 , 1 ] } Hh=1 . We denote V πh ( · ; θ ) : S → [ −H , H ] as the scalarized value function at step h for policy π , defined as V πh ( s ; θ ) : = Eπ [ ∑H h′=h〈θ , rh′ ( sh′ , ah′ ) 〉 | sh = s ] = 〈θ , Vπh ( s ) 〉 . Similarly , we denote Qπh ( · ; θ ) : S ×A → [ −H , H ] as the scalarized Q-value function at step h for policy π , where Qπh ( s , a ; θ ) : = Eπ [ ∑H h′=h〈θ , rh′ ( sh′ , ah′ ) 〉 | sh = s , ah = a ] = 〈θ , Qπh ( s , a ) 〉 . For a fixed θ ∈ Rd , there exists an optimal policy π ? θ , maximizing value for all states ( Puterman , 2014 ) ; i.e. , V π ? θ h ( s ; θ ) = supπ V π h ( s ; θ ) for all s ∈ S and h ∈ [ H ] . We abbreviate V π ? θ ( · ; θ ) and Qπ ? θ ( · ; θ ) as V ? ( · ; θ ) and Q ? ( · ; θ ) respectively .

Summary: The paper applies the techniques from reward-free RL literature to the constrained RL setting. They propose a meta-algorithm that takes a reward-free RL solver and uses it to solve the approachability and constrained-RL problems with convex constraints. The proposed approach comes with an overhead factor that is logarithmic in the number of samples.   They apply this approach to provide sharp analysis on three different settings: tabular VMDPs, linear VMDPs and two-player VMGs.