Constraining Action Sequences with Formal Languages for Deep Reinforcement Learning

We study the problem of deep reinforcement learning where the agent ’ s action sequences are constrained , e.g. , prohibition of dithering or overactuating action sequences that might damage a robot , drone , or other physical device . Our model focuses on constraints that can be described by automata such as DFAs or PDAs . We then propose multiple approaches to augment the state descriptions of the Markov decision process ( MDP ) with summaries of recent action histories . We empirically evaluate these methods applying DQN to three Atari games , training with reward shaping . We found that our approaches are effective in significantly reducing , and even eliminating , constraint violations while maintaining high reward . We also observed that the total reward achieved by an agent can be highly sensitive to how much the constraints encourage or discourage exploration of potentially effective actions during training , and , in addition to helping ensure safe policies , the use of constraints can enhance exploration during training . 1 INTRODUCTION . Deep reinforcement learning ( DRL ) shows great potential in learning decision-making agents with application to many areas , including safety-critical ones such as credential identification , autonomous vehicles , finance , and healthcare . Problematically , the inner workings of DRL models are very difficult to understand , control , or make verifiable claims about . This leads to the problem of how to ensure safe behavior in a learned agent . We propose a method using reward shaping ( Ng et al. , 1999 ; Pecka and Svoboda , 2014 ) and MDP state augmentation to bias the training process away from unsafe behaviors that have been defined in formal languages . We describe a general framework for incorporating constraints in deep reinforcement learning ( DRL ) that are expressed as formal languages over an agent ’ s action space . We explore several strategies for augmenting the state of the agent so as to maximize rewards subject to constraint satisfaction . To illustrate , consider the no-1D-dithering constraint in Figure 1 defined over an Atari Breakout agent with four actions : fire ( f ) , no move ( n ) , move left ( ` ) , and move right ( r ) . The figure ’ s minimized deterministic finite state automaton ( DFA ) accepts strings ending in ( ` r ) 2 and judges them to be constraint violations . Negative rewards can be applied whenever a violation is detected to drive DRL to avoid that behavior , but the effectiveness of such reward shaping will depend on the ability of the agent to correlate the negative rewards with features of the state . The features that are relevant to formal language constraints are inherently history-based and they may not be present in the original agent state . A straightforward state augmentation might add a finite action history . For the example property , a history of 3 or more actions would permit the DRL agent to “ see ” the structure of violating suffixes , but such an approach would not be effective for constraints that restrict sequences like ( ` f∗ r ) 2 , since there may be arbitrarily many f actions in a violating suffix . Generally speaking , formal languages provide a rich set of representations that can be exploited for augmenting agent state . For a DFA , a natural approach is to monitor the action history recording the current DFA state and adding it to the agent ’ s input state of the MDP . Referring to Figure 1 , if the current DFA state were q3 , then when an agent takes the r-action , DRL has the chance to relate the detected violation with the agent ’ s augmented input state . While we restrict our attention here to constraints expressed as regular languages defined over agent actions , the approach can be extended to richer formal languages , e.g. , pushdown automata where the augmented state includes automata and stack embeddings , and to settings where the MDP state conditions the constraint . Our contributions include : ( 1 ) a model for constraining RL with respect to action sequences , represented by automata such as DFAs ; ( 2 ) methods that augment the MDP state descriptions with representations of action sequences in order to learn to avoid constraint violations while maximizing discounted reward ; and ( 3 ) experimental results comparing our approaches ( trained with reward shaping ) to a baseline method trained without knowledge of the constraints . We found that our approaches are effective in significantly reducing , and even eliminating , constraint violations while maintaining high reward . We also observed that the total reward achieved by an agent can be highly sensitive to how much the constraints encourage or discourage exploration of potentially effective actions during training . For example , training to avoid violating the no-1D-dithering constraint in the game Breakout significantly negatively impacts the reward the agent receives in game play . In contrast , imposing a generalization of this constraint ( no-2D-dithering ) when learning Seaquest can significantly increase reward during game play . This is because in Breakout , a dithering action is in fact a useful tactic ( increasing the effective size of the paddle ) whereas in Seaquest , dithering is ineffective , so the constraint during training encourages exploration of alternate actions . We feel that , in addition to helping ensure safe policies , the use of constraints can enhance exploration during training , particularly when training time is too limited to explore effectively . The rest of our paper is as follows . Section 2 gives related work . Section 3 presents our constrained RL model , and Section 4 describes our approaches . We present our experimental results in Section 5 and conclude in Section 6 . 2 RELATED WORK . Safety in reinforcement learning is an active research area . The notion of safety during exploration was proposed by Geibel and Wysotzki ( 2005 ) . It is concerned with the visiting of undesirable states . Some safe exploration methods require a priori knowledge of varying degrees . Garcia and Fernandez ( 2012 ) utilize a known backup policy to improve on a predefined safe baseline policy while avoiding catastrophic states . Turchetta et al . ( 2016 ) expand an initial set of known safe states on regularity assumptions about safety of neighboring states . Others eliminate the need for a priori knowledge . Moldovan and Abbeel ( 2012 ) emphasize the impossibility of visiting other states from a catastrophic state and consider safety in terms of ergodicity . They optimize δ-safe policies which take the agent to start of exploration with probability at least δ. Krakovna et al . ( 2018 ) address ergodicity in dynamic environments and in presence of irreversible actions . Others use a learned a state safety function and a backup policy to revert to a safe state ( Hans et al. , 2008 ) or a backup sequence that brings the agent to close proximity of a state discovered to be safe ( Mannucci et al. , 2018 ) . Lipton et al . ( 2016 ) propose a model that remembers previously visited catastrophic states by use of a buffer and a trained safety predictor model . Jansen et al . ( 2018 ) use constraint specifications given in probabilistic computation tree logic ( PCTL ) to assess safety of possible actions for each state . In contrast , our model is neither trained to avoid states nor predicts safe actions given a state . Rather , our model uses state descriptions augmented with information about action history using constraints . Numerous policy gradient methods that aim at zero-constraint violation during exploration have been proposed . Achiam et al . ( 2017 ) proposed a constrained policy optimization method that builds on Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015 ) . The policy is kept feasible by using a recovery policy that decreases the constraint value and projects the policy to a feasible set . Chow et al . ( 2015 ) also propose a policy optimization method where the objective is a tradeoff between return and risk . Lyapunov functions have been used in assessing safety of regions ( Berkenkamp et al. , 2017 ) and guaranteeing global safety of agent ’ s policy . Contrary to the policy optimization works cited so far that consider state-action pair constraints , Dalal et al . ( 2018 ) consider constraints that are state-based . Their approach approximates a change in the constraint value of a state based on the previous state and action . In contrast , our constraints are sequence-based and implemented using DFAs instead of scalar-valued functions . In an agent versus environment game setting , Wen et al . ( 2015 ) use linear temporal logic formulas to define permissive strategies . Bou-Ammar et al . ( 2015 ) incorporated physical constraints to a safe on-line lifelong multi-task policy gradient learner operating in an adversarial framework . Lee et al . ( 2018 ) infer constraints from the actions of a demonstrator on the assumption that states visited by a demonstrator should be safe . Recently , Koller et al . ( 2018 ) proposed a Gaussian process ( GP ) -based RL that can make multi-step ahead safety predictions guarantee safe trajectories that satisfy system constraints . Aoude et al . ( 2013 ) propose a real-time GP-based path planning framework that can predict dynamic obstacles . Safe exploration techniques for reinforcement learning have been applied to practical problems . A robotic arm was trained using TRPO and a QP solver ( Pham et al. , 2017 ) and by teacher demonstrations where safety is maintained by updating a library of dangerous actions ( Martínez et al. , 2015 ) . Zhou and Li ( 2018 ) incorporate safety specifications expressed in a PCTL formula to the learning phase and use counterexamples to guide policy search in case of PCTL violations in learning from teacher demonstrations . Orseau and Armstrong ( 2016 ) consider the cases where an agent might learn to avoid ( or seek ) human intervention , thus limiting exploration and showing that Q-learning is safely interruptible , i.e. , an agent can ’ t learn to avoid ( or seek ) interruptions . Mhamdi et al . ( 2017 ) address the same problem for multi-agent systems in both the joint-action and independent learners ’ frameworks . Cizelj et al . ( 2011 ) and Cizelj and Belta ( 2012 ) consider safe vehicle driving in an environment with static and dynamic adversaries using PCTL formulas which describes mission objectives . Held et al . ( 2017 ) address the problem of simultaneously maximizing the expected return and managing expected damage for robot operation by adjusting the unsafety rate of a policy and the upper limit of physical ( state-based ) constraints that can incur damage , such as torque . In contrast to state-based constraints , Shen et al . ( 2018 ) , similar to our model , address action-based constraints in an interactive e-learning setting . However , their approach doesn ’ t directly address constraining action sequences . Rather , they constrain the total number of certain kinds of actions taken . Mazumder et al . ( 2018 ) use observed state-action pairs to predict the permissibility of actions given a state . However , their predictor doesn ’ t handle action sequences . State space augmentation was used by Oh et al . ( 2016 ) to predict future frames who applied an attention mechanism to the raw observations ( pixel values ) and augmented this with past state information retrieved from memory . Dosovitskiy and Koltun ( 2016 ) augmented high-dimensional state information with lower-dimensional “ measurement information ” relevant to the agent ’ s state . In contrast , our model uses direct or encoded information about action history for state augmentation .

Summary: This paper presents an approach for biasing an agent to avoid particular action sequences. These action sequence constraints are defined with a deterministic finite state automaton (DFA). The agent is given an additional shaping reward that penalizes it for violating these constraints. To make this an easier learning problem for the agent, its state is augmented with additional information: either an action history, the state of the DFA, or an embedding of the DFA state. The authors show that these approaches do reduce these action constraint violations over not doing anything about them.