Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs

√ K ) while allowing an Õ ( √ K ) constraint violation in K episodes . A critical question that arises is whether it is possible to keep the constraint violation even smaller . We show that when a strictly safe policy is known , then one can confine the system to zero constraint violation with arbitrarily high probability while keeping the reward regret of order Õ ( √ K ) . The algorithm which does so employs the principle of optimistic pessimism in the face of uncertainty to achieve safe exploration . When no strictly safe policy is known , though one is known to exist , then it is possible to restrict the system to bounded constraint violation with arbitrarily high probability . This is shown to be realized by a primal-dual algorithm with an optimistic primal estimate and a pessimistic dual update . 1 Introduction . Reinforcement learning ( RL ) addresses the problem of learning an optimal control policy that maximizes the expected cumulative reward while interacting with an unknown environment [ 25 ] . Standard RL algorithms typically focus only on maximizing a single objective . However , in many real-world applications , the control policy learned by an RL algorithm has to additionally satisfy stringent safety constraints [ 9 , 4 ] . For example , an autonomous vehicle may need to reach its destination in the minimum possible time without violating safety constraints such as crossing the middle of the road . The Constrained Markov Decision Process ( CMDP ) [ 2 , 12 ] formalism , where one seeks to maximize a reward while satisfying safety constraints , is a standard approach for modeling the necessary safety criteria of a control problem via constraints on cumulative costs . Several policy-gradient-based algorithms have been proposed to solve CMDPs . Lagrangian-based methods [ 26 , 24 , 21 , 16 ] formulate the CMDP problem as a saddle-point problem and optimize it via primal-dual methods , while Constrained Policy Optimization [ 1 , 29 ] ( inspired by the trust region policy optimization [ 23 ] ) computes new dual variables from scratch at each update to maintain constraints during learning . Although these algorithms provide ways to learn an optimal policy , performance guarantees about reward regret , safety violation or sample complexity are rare . One class of RL algorithms for which performance guarantees are available follow the principle of Optimism in the Face of Uncertainty ( OFU ) [ 8 , 10 , 22 ] , and provide an Õ ( √ K ) guarantee ∗The first two authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for the reward regret , where K is the number of episodes . However , these algorithms also have Õ ( √ K ) safety violations . Such significant violation of the safety constraints during learning may be unacceptable in many safety-critical real-world applications such as the control of autonomous vehicles or power systems . These applications demand a class of safe RL algorithms that can provably guarantee safety during learning . With this goal in mind , we aim to answer the following open theoretical question in this paper : Can we design safe RL algorithms that can achieve an Õ ( √ K ) regret with respect to the performance objective , while guaranteeing zero or bounded safety constraint violation with arbitrarily high probability ? We answer the above question affirmatively by proposing two algorithms and establishing their stringent safety performance during learning . Our focus is on the tabular episodic constrained RL setting ( unknown transition probabilities , rewards , and costs ) . The key idea behind both algorithms is a concept used earlier for safe exploration in constrained bandits [ 20 , 17 ] , which we call “ Optimistic Pessimism in the Face of Uncertainty ( OPFU ) ” here . The optimistic aspect incentivizes the algorithm for using exploration policies that can visit new state-action pairs , while the pessimistic aspect disincentivizes the algorithm from using exploration policies that can violate safety constraints . By carefully balancing optimism and pessimism , the proposed algorithms guarantee zero or bounded safety constraint violation during learning while achieving an Õ ( √ K ) regret with respect to the reward objective . The two algorithms address two different classes of the safe learning problem : whether a strictly safe policy is known a priori or not . The resulting exploration strategies are very different in the two cases . 1 . OptPess-LP Algorithm : This algorithm assumes the prior knowledge of a strictly safe policy . It ensures zero safety constraint violation during learning with high probability and utilizes the linear programming ( LP ) approach for solving a CMDP problem . The algorithm achieves a reward regret of Õ ( H 3 τ−c0 √ |S|3|A|K ) with respect to the performance objective , where H is the number of steps per episode , τ is the given constraint on safety violation , c0 is the known safety constraint value of a strictly safe policy π0 , and |S| and |A| are the number of states and actions respectively . 2 . OptPess-PrimalDual Algorithm : This algorithm addresses the case where no strictly safe policy , but a feasible strictly safe cost is known . By allowing a bounded ( in K ) safety cost , it opens up space for exploration . The OptPess-PrimalDual algorithm avoids linear programming and its attendant complexity and exploits the primal-dual approach for solving a CMDP problem . The proposed approach improves the computational tractability , while 2This table is presented for K ≥ poly ( |S| , |A| , H ) , with polynomial terms independent of K omitted . 3Efroni et al . [ 10 ] use N , the maximum number of non-zero transition probabilities across the entire state-action space , in their regret and constraint violation analysis . For consistency , we use |S|2|A| to boundN . 4Zheng et al . [ 30 ] assumes a known transition kernel and analyzes regret in the long-term average setting . 5The detailed constraint violation is O ( C′′H +H2 √ |S|3|A|C′′ log ( C′′/δ′ ) ) , which is independent of K. Here , δ′ = δ/ ( 16|S|2|A|H ) and C′′ = O ( H 4|S|3|A| ( τ−c0 ) 2 log H4|S|3|A| ( τ−c0 ) 2δ′ ) . ensuring a bounded safety constraint violation during learning and a reward regret of Õ ( H 3 τ−c0 √ |S|3|A|K ) with respect to the objective . Compared with the other methods listed in Table 1 , though the proposed algorithms have an additional H/ ( τ − c0 ) or √ |S|/ ( τ − c0 ) factor in the regret bounds , they are able to reduce the constraint violation to zero or constant with high probability . This improvement in safety can be extremely important for many mission-critical applications . 1.1 Related work . The problem of learning an optimal control policy that satisfies safety constraints has been studied both in the RL setting and the multi-armed bandits setting . Constrained RL : Several policy-gradient algorithms have seen success in practice [ 26 , 24 , 21 , 16 , 1 , 29 ] . Also of interest are works which utilize Gaussian processes to model the transition probabilities and value functions [ 5 , 27 , 15 , 7 ] . Several algorithms with provable guarantees are closely related to our work . Zheng et al . [ 30 ] considered the constrained RL problem in an infinite horizon setting with unknown reward and cost functions . The approach is similar to the UCRL2 algorithm [ 13 ] and achieves a sub-linear reward regret Õ ( T 34 ) while satisfying constraints with high probability during learning . In contrast to this work , we consider the setting of unknown transition probabilities . Efroni et al [ 10 ] focused on the episodic setting of unknown non-stationary transitions over a finite horizon , attaining both a reward regret and a constraint violation of Õ ( H2 √ |S|3|A|K ) . Ding et al . [ 8 ] studied an episodic setting with linear function approximation ( suitable for large state space cases ) , and proposed algorithms that can achieve Õ ( dH3 √ K ) for both the regret and the constraint violation ( where d is the dimension of the feature mapping ) . The regret analysis in [ 8 ] can be easily extended to the tabular case , yielding Õ ( H3 √ |S|2|A|K ) . Constrained Multi-Armed Bandits : Multi-armed bandit problems are special cases of MDPs , with both the number of states as well as the episode length being one . Linear bandits with constraints ( satisfied with high probability ) have been investigated in different settings . One setting , referred to as conservative bandits [ 28 , 14 , 11 ] , requires the cumulative reward to remain above a fixed percentage of the cumulative reward of a given baseline policy . Another setting is where each arm is associated with two unknown distributions ( similar to our setting ) , generating reward and cost signals respectively [ 3 , 20 , 17 , 18 ] . 2 Problem formulation . A finite-horizon constrained non-stationary MDP model is defined as a tuple M = ( S , A , H , P , r , c , τ , µ ) , where S is the state space , A is the action space , H is the number of steps in each episode , r : S ×A → [ 0 , 1 ] is the unknown reward function of interest , c : S ×A → [ 0 , 1 ] is the unknown safety cost function used to model the constraint violation , τ ∈ ( 0 , H ] is the given constant used to define the safety constraint , and µ is the known initial distribution of the state . P· ( ·|s , a ) ∈ ∆HS , ∀s ∈ S , ∀a ∈ A , where ∆S is the |S|-dimensional probability simplex , and Ph ( s ′|s , a ) is the unknown transition probability that the next state is s′ when action a is taken for state s at step h. Some further notation is necessary to define the problem . A ( randomized Markov ) policy is defined by a map π : S × [ H ] → ∆A , with πh ( a|s ) being the probability of taking action a in state s at time step h. With St and At representing the state and the action at time t respectively , let V πh ( s ; g , P ) : = EP , π [ H∑ t=h g ( St , At ) |Sh = s ] , ∀s ∈ S denote the expected cumulative value with respect to a function g : S × A → R+ under P for a policy π over a time interval [ h , h+ 1 , . . . , H ] . With slightly abuse of notation , we use V π1 ( µ ; g , P ) to denote ES1∼µ [ V π1 ( S1 ; g , P ) ] . In the formulation below there are a total of K episodes with H steps each . Each episode k ∈ [ K ] begins with an initial probability distribution µ for S1 . Then , the agent determines a randomized Markov policy πk for that episode based on the information gathered from the previous episodes , and executes it . At time step h during the execution of the k-th episode , after taking action Akh at state Skh , the agent receives a noisy reward and cost of R k h ( S k h , A k h ) = rh ( S k h , A k h ) + ξ k h ( S k h , A k h ; r ) and Ckh ( S k h , A k h ) = ch ( S k h , A k h ) + ξ k h ( S k h , A k h ; c ) , respectively . Assumption 2.1 ( Sub-Gaussian noise ) . For all h ∈ [ H ] , k ∈ [ K ] , the reward and cost noise random variables are conditionally independent zero-mean 1/2-sub-Gaussian , i.e. , E [ ξkh|Fk−1 ] = 0 , E [ exp ( λξkh ) |Fk−1 ] ≤ exp ( λ2/4 ) , ∀λ ∈ R. Here Fk is the σ-algebra generated by the random variables up to episode k. Let π∗ denote the optimal policy of the following CMDP model : max π V π1 ( µ ; r , P ) s.t . V π 1 ( µ ; c , P ) ≤ τ . ( 1 ) A policy π is said to be strictly safe if V π1 ( µ ; c , P ) < τ . Assumption 2.2 . There exists a strictly safe policy π0 with V π 0 1 ( µ ; c , P ) = c 0 < τ . There are two important cases of the safe learning problem . Zero constraint violation case : The agent has prior knowledge of a strictly safe policy π0 and its safety cost value c0 : = V π 0 1 ( µ ; c , P ) . The agent wishes to attain a sublinear ( in K ) cumulative regret , Reg ( K ; r ) : = K∑ k=1 ( V π ∗ 1 ( µ ; r , P ) − V π k 1 ( µ ; r , P ) ) , ( 2 ) while incurring zero constraint violation with at least a specified high probability ( 1− δ ) , i.e. , P ( V π k 1 ( µ ; c , P ) ≤ τ , ∀k ∈ [ K ] ) ≥ 1− δ. Bounded constraint violation case : The agent knows that there exists a strictly safe policy with a known safety cost value c0 , but does not know any strictly safe policy . It aims to achieve a cumulative regret ( 2 ) that grows sublinearly with K , while ensuring that the regret of constraint violation , Reg ( K ; c ) : = ( K∑ k=1 ( V π k 1 ( µ ; c , P ) − τ ) ) + ( where ( a ) + : = max { a , 0 } ) , satisfies supK Reg ( K ; c ) < +∞ with at least a specified high probability ( 1− δ ) . Remarks . For the zero constraint violation case , the assumption of knowing c0 can be relaxed , as shown in Appendix E .

Summary: The paper studies the problem of safety in learning, within the framework of tabular episodic Constrained MDPs (unknown transition, rewards and costs). The authors propose a method based on the principle of optimistic pessimism in face of uncertainty that achieves a regret of $\mathcal{O}(\sqrt{K})$ for the rewards ($K$ being the number of episodes) that is comparable to the existing methods. The main advantage of the proposed approach is that it achieves a smaller constraint violation: particularly zero constraint violation when an initial safe policy is given, and bounded constraint violation when no safe policy is known (but it is known that one such policy exists). The known policy case is based on the LP formulation of solving the pessimistic CMDP, and the unknown policy case uses a  primal-dual approach.