Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs

√ K ) while allowing an Õ ( √ K ) constraint violation in K episodes . A critical question that arises is whether it is possible to keep the constraint violation even smaller . We show that when a strictly safe policy is known , then one can confine the system to zero constraint violation with arbitrarily high probability while keeping the reward regret of order Õ ( √ K ) . The algorithm which does so employs the principle of optimistic pessimism in the face of uncertainty to achieve safe exploration . When no strictly safe policy is known , though one is known to exist , then it is possible to restrict the system to bounded constraint violation with arbitrarily high probability . This is shown to be realized by a primal-dual algorithm with an optimistic primal estimate and a pessimistic dual update . 1 Introduction . Reinforcement learning ( RL ) addresses the problem of learning an optimal control policy that maximizes the expected cumulative reward while interacting with an unknown environment [ 25 ] . Standard RL algorithms typically focus only on maximizing a single objective . However , in many real-world applications , the control policy learned by an RL algorithm has to additionally satisfy stringent safety constraints [ 9 , 4 ] . For example , an autonomous vehicle may need to reach its destination in the minimum possible time without violating safety constraints such as crossing the middle of the road . The Constrained Markov Decision Process ( CMDP ) [ 2 , 12 ] formalism , where one seeks to maximize a reward while satisfying safety constraints , is a standard approach for modeling the necessary safety criteria of a control problem via constraints on cumulative costs . Several policy-gradient-based algorithms have been proposed to solve CMDPs . Lagrangian-based methods [ 26 , 24 , 21 , 16 ] formulate the CMDP problem as a saddle-point problem and optimize it via primal-dual methods , while Constrained Policy Optimization [ 1 , 29 ] ( inspired by the trust region policy optimization [ 23 ] ) computes new dual variables from scratch at each update to maintain constraints during learning . Although these algorithms provide ways to learn an optimal policy , performance guarantees about reward regret , safety violation or sample complexity are rare . One class of RL algorithms for which performance guarantees are available follow the principle of Optimism in the Face of Uncertainty ( OFU ) [ 8 , 10 , 22 ] , and provide an Õ ( √ K ) guarantee ∗The first two authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for the reward regret , where K is the number of episodes . However , these algorithms also have Õ ( √ K ) safety violations . Such significant violation of the safety constraints during learning may be unacceptable in many safety-critical real-world applications such as the control of autonomous vehicles or power systems . These applications demand a class of safe RL algorithms that can provably guarantee safety during learning . With this goal in mind , we aim to answer the following open theoretical question in this paper : Can we design safe RL algorithms that can achieve an Õ ( √ K ) regret with respect to the performance objective , while guaranteeing zero or bounded safety constraint violation with arbitrarily high probability ? We answer the above question affirmatively by proposing two algorithms and establishing their stringent safety performance during learning . Our focus is on the tabular episodic constrained RL setting ( unknown transition probabilities , rewards , and costs ) . The key idea behind both algorithms is a concept used earlier for safe exploration in constrained bandits [ 20 , 17 ] , which we call “ Optimistic Pessimism in the Face of Uncertainty ( OPFU ) ” here . The optimistic aspect incentivizes the algorithm for using exploration policies that can visit new state-action pairs , while the pessimistic aspect disincentivizes the algorithm from using exploration policies that can violate safety constraints . By carefully balancing optimism and pessimism , the proposed algorithms guarantee zero or bounded safety constraint violation during learning while achieving an Õ ( √ K ) regret with respect to the reward objective . The two algorithms address two different classes of the safe learning problem : whether a strictly safe policy is known a priori or not . The resulting exploration strategies are very different in the two cases . 1 . OptPess-LP Algorithm : This algorithm assumes the prior knowledge of a strictly safe policy . It ensures zero safety constraint violation during learning with high probability and utilizes the linear programming ( LP ) approach for solving a CMDP problem . The algorithm achieves a reward regret of Õ ( H 3 τ−c0 √ |S|3|A|K ) with respect to the performance objective , where H is the number of steps per episode , τ is the given constraint on safety violation , c0 is the known safety constraint value of a strictly safe policy π0 , and |S| and |A| are the number of states and actions respectively . 2 . OptPess-PrimalDual Algorithm : This algorithm addresses the case where no strictly safe policy , but a feasible strictly safe cost is known . By allowing a bounded ( in K ) safety cost , it opens up space for exploration . The OptPess-PrimalDual algorithm avoids linear programming and its attendant complexity and exploits the primal-dual approach for solving a CMDP problem . The proposed approach improves the computational tractability , while 2This table is presented for K ≥ poly ( |S| , |A| , H ) , with polynomial terms independent of K omitted . 3Efroni et al . [ 10 ] use N , the maximum number of non-zero transition probabilities across the entire state-action space , in their regret and constraint violation analysis . For consistency , we use |S|2|A| to boundN . 4Zheng et al . [ 30 ] assumes a known transition kernel and analyzes regret in the long-term average setting . 5The detailed constraint violation is O ( C′′H +H2 √ |S|3|A|C′′ log ( C′′/δ′ ) ) , which is independent of K. Here , δ′ = δ/ ( 16|S|2|A|H ) and C′′ = O ( H 4|S|3|A| ( τ−c0 ) 2 log H4|S|3|A| ( τ−c0 ) 2δ′ ) . ensuring a bounded safety constraint violation during learning and a reward regret of Õ ( H 3 τ−c0 √ |S|3|A|K ) with respect to the objective . Compared with the other methods listed in Table 1 , though the proposed algorithms have an additional H/ ( τ − c0 ) or √ |S|/ ( τ − c0 ) factor in the regret bounds , they are able to reduce the constraint violation to zero or constant with high probability . This improvement in safety can be extremely important for many mission-critical applications . 1.1 Related work . The problem of learning an optimal control policy that satisfies safety constraints has been studied both in the RL setting and the multi-armed bandits setting . Constrained RL : Several policy-gradient algorithms have seen success in practice [ 26 , 24 , 21 , 16 , 1 , 29 ] . Also of interest are works which utilize Gaussian processes to model the transition probabilities and value functions [ 5 , 27 , 15 , 7 ] . Several algorithms with provable guarantees are closely related to our work . Zheng et al . [ 30 ] considered the constrained RL problem in an infinite horizon setting with unknown reward and cost functions . The approach is similar to the UCRL2 algorithm [ 13 ] and achieves a sub-linear reward regret Õ ( T 34 ) while satisfying constraints with high probability during learning . In contrast to this work , we consider the setting of unknown transition probabilities . Efroni et al [ 10 ] focused on the episodic setting of unknown non-stationary transitions over a finite horizon , attaining both a reward regret and a constraint violation of Õ ( H2 √ |S|3|A|K ) . Ding et al . [ 8 ] studied an episodic setting with linear function approximation ( suitable for large state space cases ) , and proposed algorithms that can achieve Õ ( dH3 √ K ) for both the regret and the constraint violation ( where d is the dimension of the feature mapping ) . The regret analysis in [ 8 ] can be easily extended to the tabular case , yielding Õ ( H3 √ |S|2|A|K ) . Constrained Multi-Armed Bandits : Multi-armed bandit problems are special cases of MDPs , with both the number of states as well as the episode length being one . Linear bandits with constraints ( satisfied with high probability ) have been investigated in different settings . One setting , referred to as conservative bandits [ 28 , 14 , 11 ] , requires the cumulative reward to remain above a fixed percentage of the cumulative reward of a given baseline policy . Another setting is where each arm is associated with two unknown distributions ( similar to our setting ) , generating reward and cost signals respectively [ 3 , 20 , 17 , 18 ] . 2 Problem formulation . A finite-horizon constrained non-stationary MDP model is defined as a tuple M = ( S , A , H , P , r , c , τ , µ ) , where S is the state space , A is the action space , H is the number of steps in each episode , r : S ×A → [ 0 , 1 ] is the unknown reward function of interest , c : S ×A → [ 0 , 1 ] is the unknown safety cost function used to model the constraint violation , τ ∈ ( 0 , H ] is the given constant used to define the safety constraint , and µ is the known initial distribution of the state . P· ( ·|s , a ) ∈ ∆HS , ∀s ∈ S , ∀a ∈ A , where ∆S is the |S|-dimensional probability simplex , and Ph ( s ′|s , a ) is the unknown transition probability that the next state is s′ when action a is taken for state s at step h. Some further notation is necessary to define the problem . A ( randomized Markov ) policy is defined by a map π : S × [ H ] → ∆A , with πh ( a|s ) being the probability of taking action a in state s at time step h. With St and At representing the state and the action at time t respectively , let V πh ( s ; g , P ) : = EP , π [ H∑ t=h g ( St , At ) |Sh = s ] , ∀s ∈ S denote the expected cumulative value with respect to a function g : S × A → R+ under P for a policy π over a time interval [ h , h+ 1 , . . . , H ] . With slightly abuse of notation , we use V π1 ( µ ; g , P ) to denote ES1∼µ [ V π1 ( S1 ; g , P ) ] . In the formulation below there are a total of K episodes with H steps each . Each episode k ∈ [ K ] begins with an initial probability distribution µ for S1 . Then , the agent determines a randomized Markov policy πk for that episode based on the information gathered from the previous episodes , and executes it . At time step h during the execution of the k-th episode , after taking action Akh at state Skh , the agent receives a noisy reward and cost of R k h ( S k h , A k h ) = rh ( S k h , A k h ) + ξ k h ( S k h , A k h ; r ) and Ckh ( S k h , A k h ) = ch ( S k h , A k h ) + ξ k h ( S k h , A k h ; c ) , respectively . Assumption 2.1 ( Sub-Gaussian noise ) . For all h ∈ [ H ] , k ∈ [ K ] , the reward and cost noise random variables are conditionally independent zero-mean 1/2-sub-Gaussian , i.e. , E [ ξkh|Fk−1 ] = 0 , E [ exp ( λξkh ) |Fk−1 ] ≤ exp ( λ2/4 ) , ∀λ ∈ R. Here Fk is the σ-algebra generated by the random variables up to episode k. Let π∗ denote the optimal policy of the following CMDP model : max π V π1 ( µ ; r , P ) s.t . V π 1 ( µ ; c , P ) ≤ τ . ( 1 ) A policy π is said to be strictly safe if V π1 ( µ ; c , P ) < τ . Assumption 2.2 . There exists a strictly safe policy π0 with V π 0 1 ( µ ; c , P ) = c 0 < τ . There are two important cases of the safe learning problem . Zero constraint violation case : The agent has prior knowledge of a strictly safe policy π0 and its safety cost value c0 : = V π 0 1 ( µ ; c , P ) . The agent wishes to attain a sublinear ( in K ) cumulative regret , Reg ( K ; r ) : = K∑ k=1 ( V π ∗ 1 ( µ ; r , P ) − V π k 1 ( µ ; r , P ) ) , ( 2 ) while incurring zero constraint violation with at least a specified high probability ( 1− δ ) , i.e. , P ( V π k 1 ( µ ; c , P ) ≤ τ , ∀k ∈ [ K ] ) ≥ 1− δ. Bounded constraint violation case : The agent knows that there exists a strictly safe policy with a known safety cost value c0 , but does not know any strictly safe policy . It aims to achieve a cumulative regret ( 2 ) that grows sublinearly with K , while ensuring that the regret of constraint violation , Reg ( K ; c ) : = ( K∑ k=1 ( V π k 1 ( µ ; c , P ) − τ ) ) + ( where ( a ) + : = max { a , 0 } ) , satisfies supK Reg ( K ; c ) < +∞ with at least a specified high probability ( 1− δ ) . Remarks . For the zero constraint violation case , the assumption of knowing c0 can be relaxed , as shown in Appendix E .

Summary: This paper presents two novel reinforcement learning algorithms for CMDPS, one which has no constraint violations with high probability, but requires a known safe policy, and another which bounds the probability of constraint regret, but only requires the existence of a safe policy. The first algorithm picks pessimistically safe policies if they obtain a high enough value, but chooses a best reward policy otherwise. The second algorithm iteratively chooses the highest value policy subject to a dynamically tightening constraint. Both algorithms have a higher regret than previous approaches, but are provably safe  in the respective metrics