Formal Language Constrained Markov Decision Processes

1 INTRODUCTION . The ability to impose safety constraints on an agent is key to the deployment of reinforcement learning ( RL ) systems in real-world environments ( Amodei et al. , 2016 ) . Controllers that are derived mathematically typically rely on a full a priori analysis of agent behavior remaining within a predefined envelope of safety in order to guarantee safe operation ( Aréchiga & Krogh , 2014 ) . This approach restricts controllers to pre-defined , analytical operational limits , but allows for verification of safety properties ( Huth & Kwiatkowska , 1997 ) and satisfaction of software contracts ( Helm et al. , 1990 ) , which enables their use as a component in larger systems . By contrast , RL controllers are free to learn control trajectories that better suit their tasks and goals ; however , understanding and verifying their safety properties is challenging . A particular hazard of learning an RL controller is the requirement of exploration in an unknown environment . It is desirable not only to obey constraints in the final policy , but also throughout the exploration and learning process ( Ray et al. , 2019 ) . The goal of safe operation as an optimization objective is formalized by the constrained Markov decision process ( CMDP ) ( Altman , 1999 ) , which adds to a Markov decision process ( MDP ) a cost signal similar to the reward signal , and poses a constrained optimization problem in which discounted reward is maximized while the total cost must remain below a pre-specified limit per constraint . We use this framework and propose specifying CMDP constraints in formal languages to add useful structure based on expert knowledge , e.g. , building sensitivity to proximity into constraints on object collision or converting a non-Markovian constraint into a Markovian one ( De Giacomo et al. , 2020 ) . A significant advantage of specifying constraints with formal languages is that they already form a well-developed basis for components of safety-critical systems ( Huth & Kwiatkowska , 1997 ; Clarke et al. , 2001 ; Kwiatkowska et al. , 2002 ; Baier et al. , 2003 ) and safety properties specified in formal languages can be verified a priori ( Kupferman et al. , 2000 ; Bouajjani et al. , 1997 ) . Moreover , the recognition problem for many classes of formal languages imposes modest computational requirements , making them suitable for efficient runtime verification ( Chen & Roşu , 2007 ) . This allows for low-overhead incorporation of potentially complex constraints into RL training and deployment . We propose ( 1 ) a method for posing formal language constraints defined over MDP trajectories as CMDP cost functions ; ( 2 ) augmenting MDP state with constraint automaton state to more explicitly encourage learning of joint MDP/constraint dynamics ; ( 3 ) a method for learning a dense cost function given a sparse cost function from joint MDP/constraint dynamics ; and ( 4 ) a method based on constraint structure to dynamically modify the set of available actions to guarantee the prevention of constraint violations . We validate our methods over a variety of RL algorithms with standard constraints in Safety Gym and hand-built constraints in MuJoCo and Atari environments . The remainder of this work is organized as follows . Section 2 presents related work in CMDPs , using expert advice in RL and safety , as well as formal languages in similar settings . Section 3 describes our definition of a formal language-based cost function , as well as how it ’ s employed in state augmentation , cost shaping , and action shaping . Section 4 details our experimental setup and results and finally , discussion of limitations and future work are located in Section 5 . 2 RELATED WORK . Safety and CMDP Framework The CMDP framework doesn ’ t prescribe the exact form of constraints or how to satisfy the constrained optimization problem . Chow et al . ( 2017 ) propose conditional value-at-risk of accumulated cost and chance constraints as the values to be constrained and use a Lagrangian formulation to derive a Bellman optimality condition . Dalal et al . ( 2018 ) use a different constraint for each MDP state and a safety layer that analytically solves a linearized action correction formulation per state . Similarly , Pham et al . ( 2018 ) introduce a layer that corrects the output of a policy to respect constraints on the dynamic of a robotic arm . Teacher Advice A subset of work in safe exploration uses expert advice with potential-based reward shaping mechanisms ( Ng et al. , 1999 ) . Wiewiora et al . ( 2003 ) introduce a general method for incorporating arbitrary advice into the reward structure . Saunders et al . ( 2017 ) use a human in the loop to learn an effective RL agent while minimizing cost accumulated over training . Camacho et al . ( 2017a ; b ) use DFAs with static reward shaping attached to states to express non-Markovian rewards . We generalise this work with a learned shaping function in the case of dense soft constraints , and by generalising from reward shaping to other CMDP learning mechanisms . Similar to teacher advice is shielding ( Jansen et al. , 2018 ; Alshiekh et al. , 2018 ) , in which an agent ’ s actions are filtered through a shield that blocks actions that would introduce an unsafe state ( similar to hard constraints ; section 3 ) , but typically requires MDP states to be enumerable and few enough that a shield can be constructed efficiently . Formal Languages Formal languages and automata have been used before in RL for task specification or as task abstractions ( options ) in hierarchical reinforcement learning ( Icarte et al. , 2018b ; Li et al. , 2017 ; Wen et al. , 2017 ; Mousavi et al. , 2014 ) . In some cases , these automata were derived from Linear Temporal Logic ( LTL ) formulae , in others LTL or other formal language formulae have been directly used to specify tasks ( Icarte et al. , 2018a ) . Littman et al . ( 2017 ) defines a modified LTL designed for use in reinforcement learning . In robotics , LTL is used for task learning ( Li et al. , 2017 ) , sometimes in conjunction with teacher demonstrations ( Li et al. , 2018 ) . Zhu et al . ( 2019 ) and Fulton & Platzer ( 2019 ) both study the use of formal languages for safe RL , though each makes assumptions about a prior knowledge of the environment dynamics . Hasanbeig et al . ( 2018 ) and Hasanbeig et al . ( 2020 ) learn a product MDP with a safety constraint specified in a formal language , but require knowledge of MDP structure and don ’ t scale to large MDPs . Hahn et al . ( 2019 ) uses an omega-regular language to specify an objective and constructs a product MDP to ensure the objective is reached almost surely , but requires large reward shaping which prevents learning when used for a constraint rather than objective . 3 FORMAL LANGUAGE CONSTRAINED MDPS . The constrained Markov decision process ( CMDP ) ( Altman , 1999 ) extends the Markov decision process ( Sutton & Barto , 2018 ) to incorporate constraints . The difference is an additional set of cost functions ci : S ⇥ A ⇥ S ! R and set of cost limits di 2 R. Then , the constrained optimization problem is argmax⇡ Jr ( ⇡ ) s.t . Jci ( ⇡ ) di , i = 1 , . . . , k where Jr ( ⇡ ) is a return-based objective , e.g. , finite horizon discounted return defined Jr ( ⇡ ) = E⌧⇠⇡ [ P t2⌧ trt ] and Jci is a cost-based constraint function defined similarly , replacing rt with ci , t . We propose formal language constrained MDPs ( FLCMDPs ) as a subset of CMDPs in which each constraint Ci ⇢ ( S⇥A⇥S ) ⇤ is defined by a set of prohibited trajectories . ( Subscript i is suppressed from this point without loss of generality ) . Because C is defined by a formal language , it can be recognized efficiently by an automaton , which we use to construct the cost function . We define three functions for interacting with the constraint automaton : a translation function TC : ( S⇥A⇥S ) ! ⌃C that converts MDP transitions into a symbol in the recognizer ’ s input language , a recognizer function DC : Qc⇥⌃C ! QC that steps the recognizer automaton using its current state and the input symbol and returns its next state , and finally a cost assignment GC : QC ! R that assigns a real-valued cost to each recognizer state . The composition of these three functions forms a CMDP cost function defined c = GC DC TC : ( S ⇥ A ⇥ S ) ⇥ Qc ! R , which requires constraint state Qc to be tracked along with the MDP state , S. The interaction of these functions with the underlying MDP framework is illustrated in Figure 1 ( a ) , where the constraint uses the MDP state and action at time t to calculate the cost signal at time t and , if action shaping is being employed as discussed below , influence the action at time t+ 1 . We note that this construction does not require knowledge of the MDP transition probabilities and further , its complexity is independent of the size of the MDP , which allows it to scale to large , challenging environments . Translation Function TC The translation function accepts as input the MDP state and action at each time step , and outputs a token in the discrete , finite language of the associated recognizer . This allows the recognizer automaton to be defined in a small , discrete language , rather than over unwieldy and potentially non-discrete MDP transitions . Further , freedom in choice of input language allows for flexible design of the constraint automaton to encode the desired inductive bias , and thus more meaningful structured states . Recognizer Function DC Each constraint is instantiated with a finite automaton recognizer that decides whether a trajectory is in the constraint set . The only necessary assumption about the recognizer is that it defines some meaningful state that may be used for learning the constraint dynamics . Our implementation uses a deterministic finite automaton ( DFA ) as the recognizer for each constraint , defined as ( Q , ⌃ , , q0 , F ) , where Q is the set of the DFA ’ s states , ⌃ is the alphabet over which the constraint is defined , : Q⇥ ⌃ ! Q is the transition function , q0 2 Q is the start state , and F ⇢ Q is the set of accepting states that represent constraint violations . The DFA is set to its initial state at the start of each episode and is advanced at each time step with the token output by the translation layer . Although our experiments use DFAs as a relatively simple recognizer , the framework can be easily modified to work with automata that encode richer formal languages like pushdown automata or hybrid automata . Constraint State Augmentation In order to more efficiently learn constraint dynamics , the MDP state st is augmented with a one-hot representation of the recognizer state qt . To preserve the Markov property of the underlying MDP , state augmentation should contain sufficient information about the recognizer state and , if it is stateful , the translation function . To enhance performance , the one-hot state is embedded to dlog2 ( |Q| ) e dimensions before being input into any network and the embedding is learned with gradients backpropagated through the full network . Cost Assignment Function The cost assignment function GC assigns a real-valued cost to each state of the recognizer . This cost can be used in optimization to enforce the constraint with a Lagrangian-derived objective penalty , or via reward shaping , which updates the reward function to rt ct , where is a scaling coeffficient . Cost assignments are frequently sparse , where GC is only non-zero at accepting states that recognize a constraint violation . This poses a learning problem for optimization-based methods that use reward shaping or an objective penalty to solve the CMDP . A goal of constrained RL is to minimize accumulated constraint violations over training but , to ensure that the frequency of violations is small , the optimization penalty can be large relative to the reward signal . This can lead to a situation in which an unnecessarily conservative policy is adopted early in training , slowing exploration . We next propose a method for learning a dense cost function that takes advantage of the structure of the constraint automaton to more quickly learn constraint dynamics and avoid unnecessarily conservative behavior . Learned Dense Cost The goal of learning a dense cost is not to change the optimality or nearoptimality of a policy with respect to the constrained learning problem . Thus , we use the form of potential-based shaping : F ( st 1 , at , st ) = ( st ) ( st 1 ) , where is a potential function ( see Ng et al . ( 1999 ) for details ) . This is added as a shaping term to the sparse cost to get the dense cost G0C ( qt 1 , qt ) = GC ( qt ) + ( ( qt ) ( qt 1 ) ) , where scales the added dense cost relative to the sparse cost , and is a function of the recognizer state rather than the MDP state , which requires st 2 and at 2 as additional inputs to calculate qt 1 . Generally , if the value of increases as the automaton state is nearer to a violation , then the added shaping terms add cost for moving nearer to a constraint violation and refund cost for backing away from a potential violation . In our experiments , the potential ⇡ ( qt ) is defined using tv ( qt ) , which is a random variable defined as the number of steps between visiting recognizer state qt and an accepting recognizer state . This variable ’ s distribution is based on ⇡ and the MDP ’ s transition function . Its value is small if a violation is expected to occur soon after reaching qt and vice-versa . We then define the potential function as ⇡ ( qt ) = 1 2 ( E⇡ [ Tv ( qt ) ] /tbaselinev ) , which ensures that its value is always in [ 0 , 1 ] and rises exponentially as the expected time to a violation becomes smaller . If the expected time to next violation is much larger than the provided baseline , tbaselinev , then the potential value becomes small , as shaping is unnecessary in safe states . The expected value of Tv ( qt ) over trajectories may be calculated as a Monte Carlo estimate from rollouts by counting the number of steps between each visit to qt and the next constraint violation . Each Tv is tracked with an exponential moving average and is updated between episodes to ensure that it ’ s stationary in each rollout . We set tbaselinev to be the ratio of estimated or exact length of an episode and the constraint limit di , but find empirically that the the method is resilient to the exact choice . Hard Constraints and Action Shaping When safety constraints are strict , i.e. , when the limit on the number of constraint violations d is zero , the set of available actions is reduced to ensure a violation can not occur . If a constraint isn ’ t fully state-dependent ( i.e. , there is always an action choice that avoids violation ) , then action shaping can guarantee that a constraint is never violated . Otherwise , knowledge of which actions lead to future violating trajectories requires knowledge of the underlying MDP dynamics , which is possible by learning a model that converts state constraints into state-conditional action constraints as in Dalal et al . ( 2018 ) . Our implementation of hard constraints initially allows the agent to freely choose its action , but before finalizing that choice , simulates stepping the DFA with the resulting token from the translation function and , if that lookahead step would move it into a violating state , it switches to the next best choice until a non-violating action is found . For the constraints in our experiments , it is always possible to choose a non-violating action . A known safe fallback policy can be employed in the case when an episode can not be terminated . Action shaping can be applied during training or deployment , as opposed to reward shaping , which is only applied during training . We experiment with applying action shaping only during training , only during evaluation , or in both training and evaluation .

Summary: The paper proposes a constrained reinforcement learning (RL) formulation relying on constraints written in a formal language. The proposed formulation is based on constrained Markov decision processes where the constraint is represented as a deterministic finite automaton that rejects any trajectory violating the constraint. The proposed solution relies on transforming the automaton's sparse binary cost into an approximate dense cost and augmenting that with the reward objective. The paper presents a series of results from simulations in Safety Gym, MuJoCo, and Atari environments.