Formal Language Constrained Markov Decision Processes

1 INTRODUCTION . The ability to impose safety constraints on an agent is key to the deployment of reinforcement learning ( RL ) systems in real-world environments ( Amodei et al. , 2016 ) . Controllers that are derived mathematically typically rely on a full a priori analysis of agent behavior remaining within a predefined envelope of safety in order to guarantee safe operation ( Aréchiga & Krogh , 2014 ) . This approach restricts controllers to pre-defined , analytical operational limits , but allows for verification of safety properties ( Huth & Kwiatkowska , 1997 ) and satisfaction of software contracts ( Helm et al. , 1990 ) , which enables their use as a component in larger systems . By contrast , RL controllers are free to learn control trajectories that better suit their tasks and goals ; however , understanding and verifying their safety properties is challenging . A particular hazard of learning an RL controller is the requirement of exploration in an unknown environment . It is desirable not only to obey constraints in the final policy , but also throughout the exploration and learning process ( Ray et al. , 2019 ) . The goal of safe operation as an optimization objective is formalized by the constrained Markov decision process ( CMDP ) ( Altman , 1999 ) , which adds to a Markov decision process ( MDP ) a cost signal similar to the reward signal , and poses a constrained optimization problem in which discounted reward is maximized while the total cost must remain below a pre-specified limit per constraint . We use this framework and propose specifying CMDP constraints in formal languages to add useful structure based on expert knowledge , e.g. , building sensitivity to proximity into constraints on object collision or converting a non-Markovian constraint into a Markovian one ( De Giacomo et al. , 2020 ) . A significant advantage of specifying constraints with formal languages is that they already form a well-developed basis for components of safety-critical systems ( Huth & Kwiatkowska , 1997 ; Clarke et al. , 2001 ; Kwiatkowska et al. , 2002 ; Baier et al. , 2003 ) and safety properties specified in formal languages can be verified a priori ( Kupferman et al. , 2000 ; Bouajjani et al. , 1997 ) . Moreover , the recognition problem for many classes of formal languages imposes modest computational requirements , making them suitable for efficient runtime verification ( Chen & Roşu , 2007 ) . This allows for low-overhead incorporation of potentially complex constraints into RL training and deployment . We propose ( 1 ) a method for posing formal language constraints defined over MDP trajectories as CMDP cost functions ; ( 2 ) augmenting MDP state with constraint automaton state to more explicitly encourage learning of joint MDP/constraint dynamics ; ( 3 ) a method for learning a dense cost function given a sparse cost function from joint MDP/constraint dynamics ; and ( 4 ) a method based on constraint structure to dynamically modify the set of available actions to guarantee the prevention of constraint violations . We validate our methods over a variety of RL algorithms with standard constraints in Safety Gym and hand-built constraints in MuJoCo and Atari environments . The remainder of this work is organized as follows . Section 2 presents related work in CMDPs , using expert advice in RL and safety , as well as formal languages in similar settings . Section 3 describes our definition of a formal language-based cost function , as well as how it ’ s employed in state augmentation , cost shaping , and action shaping . Section 4 details our experimental setup and results and finally , discussion of limitations and future work are located in Section 5 . 2 RELATED WORK . Safety and CMDP Framework The CMDP framework doesn ’ t prescribe the exact form of constraints or how to satisfy the constrained optimization problem . Chow et al . ( 2017 ) propose conditional value-at-risk of accumulated cost and chance constraints as the values to be constrained and use a Lagrangian formulation to derive a Bellman optimality condition . Dalal et al . ( 2018 ) use a different constraint for each MDP state and a safety layer that analytically solves a linearized action correction formulation per state . Similarly , Pham et al . ( 2018 ) introduce a layer that corrects the output of a policy to respect constraints on the dynamic of a robotic arm . Teacher Advice A subset of work in safe exploration uses expert advice with potential-based reward shaping mechanisms ( Ng et al. , 1999 ) . Wiewiora et al . ( 2003 ) introduce a general method for incorporating arbitrary advice into the reward structure . Saunders et al . ( 2017 ) use a human in the loop to learn an effective RL agent while minimizing cost accumulated over training . Camacho et al . ( 2017a ; b ) use DFAs with static reward shaping attached to states to express non-Markovian rewards . We generalise this work with a learned shaping function in the case of dense soft constraints , and by generalising from reward shaping to other CMDP learning mechanisms . Similar to teacher advice is shielding ( Jansen et al. , 2018 ; Alshiekh et al. , 2018 ) , in which an agent ’ s actions are filtered through a shield that blocks actions that would introduce an unsafe state ( similar to hard constraints ; section 3 ) , but typically requires MDP states to be enumerable and few enough that a shield can be constructed efficiently . Formal Languages Formal languages and automata have been used before in RL for task specification or as task abstractions ( options ) in hierarchical reinforcement learning ( Icarte et al. , 2018b ; Li et al. , 2017 ; Wen et al. , 2017 ; Mousavi et al. , 2014 ) . In some cases , these automata were derived from Linear Temporal Logic ( LTL ) formulae , in others LTL or other formal language formulae have been directly used to specify tasks ( Icarte et al. , 2018a ) . Littman et al . ( 2017 ) defines a modified LTL designed for use in reinforcement learning . In robotics , LTL is used for task learning ( Li et al. , 2017 ) , sometimes in conjunction with teacher demonstrations ( Li et al. , 2018 ) . Zhu et al . ( 2019 ) and Fulton & Platzer ( 2019 ) both study the use of formal languages for safe RL , though each makes assumptions about a prior knowledge of the environment dynamics . Hasanbeig et al . ( 2018 ) and Hasanbeig et al . ( 2020 ) learn a product MDP with a safety constraint specified in a formal language , but require knowledge of MDP structure and don ’ t scale to large MDPs . Hahn et al . ( 2019 ) uses an omega-regular language to specify an objective and constructs a product MDP to ensure the objective is reached almost surely , but requires large reward shaping which prevents learning when used for a constraint rather than objective . 3 FORMAL LANGUAGE CONSTRAINED MDPS . The constrained Markov decision process ( CMDP ) ( Altman , 1999 ) extends the Markov decision process ( Sutton & Barto , 2018 ) to incorporate constraints . The difference is an additional set of cost functions ci : S ⇥ A ⇥ S ! R and set of cost limits di 2 R. Then , the constrained optimization problem is argmax⇡ Jr ( ⇡ ) s.t . Jci ( ⇡ ) di , i = 1 , . . . , k where Jr ( ⇡ ) is a return-based objective , e.g. , finite horizon discounted return defined Jr ( ⇡ ) = E⌧⇠⇡ [ P t2⌧ trt ] and Jci is a cost-based constraint function defined similarly , replacing rt with ci , t . We propose formal language constrained MDPs ( FLCMDPs ) as a subset of CMDPs in which each constraint Ci ⇢ ( S⇥A⇥S ) ⇤ is defined by a set of prohibited trajectories . ( Subscript i is suppressed from this point without loss of generality ) . Because C is defined by a formal language , it can be recognized efficiently by an automaton , which we use to construct the cost function . We define three functions for interacting with the constraint automaton : a translation function TC : ( S⇥A⇥S ) ! ⌃C that converts MDP transitions into a symbol in the recognizer ’ s input language , a recognizer function DC : Qc⇥⌃C ! QC that steps the recognizer automaton using its current state and the input symbol and returns its next state , and finally a cost assignment GC : QC ! R that assigns a real-valued cost to each recognizer state . The composition of these three functions forms a CMDP cost function defined c = GC DC TC : ( S ⇥ A ⇥ S ) ⇥ Qc ! R , which requires constraint state Qc to be tracked along with the MDP state , S. The interaction of these functions with the underlying MDP framework is illustrated in Figure 1 ( a ) , where the constraint uses the MDP state and action at time t to calculate the cost signal at time t and , if action shaping is being employed as discussed below , influence the action at time t+ 1 . We note that this construction does not require knowledge of the MDP transition probabilities and further , its complexity is independent of the size of the MDP , which allows it to scale to large , challenging environments . Translation Function TC The translation function accepts as input the MDP state and action at each time step , and outputs a token in the discrete , finite language of the associated recognizer . This allows the recognizer automaton to be defined in a small , discrete language , rather than over unwieldy and potentially non-discrete MDP transitions . Further , freedom in choice of input language allows for flexible design of the constraint automaton to encode the desired inductive bias , and thus more meaningful structured states . Recognizer Function DC Each constraint is instantiated with a finite automaton recognizer that decides whether a trajectory is in the constraint set . The only necessary assumption about the recognizer is that it defines some meaningful state that may be used for learning the constraint dynamics . Our implementation uses a deterministic finite automaton ( DFA ) as the recognizer for each constraint , defined as ( Q , ⌃ , , q0 , F ) , where Q is the set of the DFA ’ s states , ⌃ is the alphabet over which the constraint is defined , : Q⇥ ⌃ ! Q is the transition function , q0 2 Q is the start state , and F ⇢ Q is the set of accepting states that represent constraint violations . The DFA is set to its initial state at the start of each episode and is advanced at each time step with the token output by the translation layer . Although our experiments use DFAs as a relatively simple recognizer , the framework can be easily modified to work with automata that encode richer formal languages like pushdown automata or hybrid automata . Constraint State Augmentation In order to more efficiently learn constraint dynamics , the MDP state st is augmented with a one-hot representation of the recognizer state qt . To preserve the Markov property of the underlying MDP , state augmentation should contain sufficient information about the recognizer state and , if it is stateful , the translation function . To enhance performance , the one-hot state is embedded to dlog2 ( |Q| ) e dimensions before being input into any network and the embedding is learned with gradients backpropagated through the full network . Cost Assignment Function The cost assignment function GC assigns a real-valued cost to each state of the recognizer . This cost can be used in optimization to enforce the constraint with a Lagrangian-derived objective penalty , or via reward shaping , which updates the reward function to rt ct , where is a scaling coeffficient . Cost assignments are frequently sparse , where GC is only non-zero at accepting states that recognize a constraint violation . This poses a learning problem for optimization-based methods that use reward shaping or an objective penalty to solve the CMDP . A goal of constrained RL is to minimize accumulated constraint violations over training but , to ensure that the frequency of violations is small , the optimization penalty can be large relative to the reward signal . This can lead to a situation in which an unnecessarily conservative policy is adopted early in training , slowing exploration . We next propose a method for learning a dense cost function that takes advantage of the structure of the constraint automaton to more quickly learn constraint dynamics and avoid unnecessarily conservative behavior . Learned Dense Cost The goal of learning a dense cost is not to change the optimality or nearoptimality of a policy with respect to the constrained learning problem . Thus , we use the form of potential-based shaping : F ( st 1 , at , st ) = ( st ) ( st 1 ) , where is a potential function ( see Ng et al . ( 1999 ) for details ) . This is added as a shaping term to the sparse cost to get the dense cost G0C ( qt 1 , qt ) = GC ( qt ) + ( ( qt ) ( qt 1 ) ) , where scales the added dense cost relative to the sparse cost , and is a function of the recognizer state rather than the MDP state , which requires st 2 and at 2 as additional inputs to calculate qt 1 . Generally , if the value of increases as the automaton state is nearer to a violation , then the added shaping terms add cost for moving nearer to a constraint violation and refund cost for backing away from a potential violation . In our experiments , the potential ⇡ ( qt ) is defined using tv ( qt ) , which is a random variable defined as the number of steps between visiting recognizer state qt and an accepting recognizer state . This variable ’ s distribution is based on ⇡ and the MDP ’ s transition function . Its value is small if a violation is expected to occur soon after reaching qt and vice-versa . We then define the potential function as ⇡ ( qt ) = 1 2 ( E⇡ [ Tv ( qt ) ] /tbaselinev ) , which ensures that its value is always in [ 0 , 1 ] and rises exponentially as the expected time to a violation becomes smaller . If the expected time to next violation is much larger than the provided baseline , tbaselinev , then the potential value becomes small , as shaping is unnecessary in safe states . The expected value of Tv ( qt ) over trajectories may be calculated as a Monte Carlo estimate from rollouts by counting the number of steps between each visit to qt and the next constraint violation . Each Tv is tracked with an exponential moving average and is updated between episodes to ensure that it ’ s stationary in each rollout . We set tbaselinev to be the ratio of estimated or exact length of an episode and the constraint limit di , but find empirically that the the method is resilient to the exact choice . Hard Constraints and Action Shaping When safety constraints are strict , i.e. , when the limit on the number of constraint violations d is zero , the set of available actions is reduced to ensure a violation can not occur . If a constraint isn ’ t fully state-dependent ( i.e. , there is always an action choice that avoids violation ) , then action shaping can guarantee that a constraint is never violated . Otherwise , knowledge of which actions lead to future violating trajectories requires knowledge of the underlying MDP dynamics , which is possible by learning a model that converts state constraints into state-conditional action constraints as in Dalal et al . ( 2018 ) . Our implementation of hard constraints initially allows the agent to freely choose its action , but before finalizing that choice , simulates stepping the DFA with the resulting token from the translation function and , if that lookahead step would move it into a violating state , it switches to the next best choice until a non-violating action is found . For the constraints in our experiments , it is always possible to choose a non-violating action . A known safe fallback policy can be employed in the case when an episode can not be terminated . Action shaping can be applied during training or deployment , as opposed to reward shaping , which is only applied during training . We experiment with applying action shaping only during training , only during evaluation , or in both training and evaluation .

Summary: The paper builds on the constrained MDP framework (Altman, 1999), by considering the special-case where the cost functions are defined in terms of states from a parser of a formal language. In the experiments the work uses deterministic finite automata (DFA) but in principle other more expressive classes could be used. Using a formal language to specify constraints may simplify model checking (although this is left to future work).