Constraining Action Sequences with Formal Languages for Deep Reinforcement Learning

We study the problem of deep reinforcement learning where the agent ’ s action sequences are constrained , e.g. , prohibition of dithering or overactuating action sequences that might damage a robot , drone , or other physical device . Our model focuses on constraints that can be described by automata such as DFAs or PDAs . We then propose multiple approaches to augment the state descriptions of the Markov decision process ( MDP ) with summaries of recent action histories . We empirically evaluate these methods applying DQN to three Atari games , training with reward shaping . We found that our approaches are effective in significantly reducing , and even eliminating , constraint violations while maintaining high reward . We also observed that the total reward achieved by an agent can be highly sensitive to how much the constraints encourage or discourage exploration of potentially effective actions during training , and , in addition to helping ensure safe policies , the use of constraints can enhance exploration during training . 1 INTRODUCTION . Deep reinforcement learning ( DRL ) shows great potential in learning decision-making agents with application to many areas , including safety-critical ones such as credential identification , autonomous vehicles , finance , and healthcare . Problematically , the inner workings of DRL models are very difficult to understand , control , or make verifiable claims about . This leads to the problem of how to ensure safe behavior in a learned agent . We propose a method using reward shaping ( Ng et al. , 1999 ; Pecka and Svoboda , 2014 ) and MDP state augmentation to bias the training process away from unsafe behaviors that have been defined in formal languages . We describe a general framework for incorporating constraints in deep reinforcement learning ( DRL ) that are expressed as formal languages over an agent ’ s action space . We explore several strategies for augmenting the state of the agent so as to maximize rewards subject to constraint satisfaction . To illustrate , consider the no-1D-dithering constraint in Figure 1 defined over an Atari Breakout agent with four actions : fire ( f ) , no move ( n ) , move left ( ` ) , and move right ( r ) . The figure ’ s minimized deterministic finite state automaton ( DFA ) accepts strings ending in ( ` r ) 2 and judges them to be constraint violations . Negative rewards can be applied whenever a violation is detected to drive DRL to avoid that behavior , but the effectiveness of such reward shaping will depend on the ability of the agent to correlate the negative rewards with features of the state . The features that are relevant to formal language constraints are inherently history-based and they may not be present in the original agent state . A straightforward state augmentation might add a finite action history . For the example property , a history of 3 or more actions would permit the DRL agent to “ see ” the structure of violating suffixes , but such an approach would not be effective for constraints that restrict sequences like ( ` f∗ r ) 2 , since there may be arbitrarily many f actions in a violating suffix . Generally speaking , formal languages provide a rich set of representations that can be exploited for augmenting agent state . For a DFA , a natural approach is to monitor the action history recording the current DFA state and adding it to the agent ’ s input state of the MDP . Referring to Figure 1 , if the current DFA state were q3 , then when an agent takes the r-action , DRL has the chance to relate the detected violation with the agent ’ s augmented input state . While we restrict our attention here to constraints expressed as regular languages defined over agent actions , the approach can be extended to richer formal languages , e.g. , pushdown automata where the augmented state includes automata and stack embeddings , and to settings where the MDP state conditions the constraint . Our contributions include : ( 1 ) a model for constraining RL with respect to action sequences , represented by automata such as DFAs ; ( 2 ) methods that augment the MDP state descriptions with representations of action sequences in order to learn to avoid constraint violations while maximizing discounted reward ; and ( 3 ) experimental results comparing our approaches ( trained with reward shaping ) to a baseline method trained without knowledge of the constraints . We found that our approaches are effective in significantly reducing , and even eliminating , constraint violations while maintaining high reward . We also observed that the total reward achieved by an agent can be highly sensitive to how much the constraints encourage or discourage exploration of potentially effective actions during training . For example , training to avoid violating the no-1D-dithering constraint in the game Breakout significantly negatively impacts the reward the agent receives in game play . In contrast , imposing a generalization of this constraint ( no-2D-dithering ) when learning Seaquest can significantly increase reward during game play . This is because in Breakout , a dithering action is in fact a useful tactic ( increasing the effective size of the paddle ) whereas in Seaquest , dithering is ineffective , so the constraint during training encourages exploration of alternate actions . We feel that , in addition to helping ensure safe policies , the use of constraints can enhance exploration during training , particularly when training time is too limited to explore effectively . The rest of our paper is as follows . Section 2 gives related work . Section 3 presents our constrained RL model , and Section 4 describes our approaches . We present our experimental results in Section 5 and conclude in Section 6 . 2 RELATED WORK . Safety in reinforcement learning is an active research area . The notion of safety during exploration was proposed by Geibel and Wysotzki ( 2005 ) . It is concerned with the visiting of undesirable states . Some safe exploration methods require a priori knowledge of varying degrees . Garcia and Fernandez ( 2012 ) utilize a known backup policy to improve on a predefined safe baseline policy while avoiding catastrophic states . Turchetta et al . ( 2016 ) expand an initial set of known safe states on regularity assumptions about safety of neighboring states . Others eliminate the need for a priori knowledge . Moldovan and Abbeel ( 2012 ) emphasize the impossibility of visiting other states from a catastrophic state and consider safety in terms of ergodicity . They optimize δ-safe policies which take the agent to start of exploration with probability at least δ. Krakovna et al . ( 2018 ) address ergodicity in dynamic environments and in presence of irreversible actions . Others use a learned a state safety function and a backup policy to revert to a safe state ( Hans et al. , 2008 ) or a backup sequence that brings the agent to close proximity of a state discovered to be safe ( Mannucci et al. , 2018 ) . Lipton et al . ( 2016 ) propose a model that remembers previously visited catastrophic states by use of a buffer and a trained safety predictor model . Jansen et al . ( 2018 ) use constraint specifications given in probabilistic computation tree logic ( PCTL ) to assess safety of possible actions for each state . In contrast , our model is neither trained to avoid states nor predicts safe actions given a state . Rather , our model uses state descriptions augmented with information about action history using constraints . Numerous policy gradient methods that aim at zero-constraint violation during exploration have been proposed . Achiam et al . ( 2017 ) proposed a constrained policy optimization method that builds on Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015 ) . The policy is kept feasible by using a recovery policy that decreases the constraint value and projects the policy to a feasible set . Chow et al . ( 2015 ) also propose a policy optimization method where the objective is a tradeoff between return and risk . Lyapunov functions have been used in assessing safety of regions ( Berkenkamp et al. , 2017 ) and guaranteeing global safety of agent ’ s policy . Contrary to the policy optimization works cited so far that consider state-action pair constraints , Dalal et al . ( 2018 ) consider constraints that are state-based . Their approach approximates a change in the constraint value of a state based on the previous state and action . In contrast , our constraints are sequence-based and implemented using DFAs instead of scalar-valued functions . In an agent versus environment game setting , Wen et al . ( 2015 ) use linear temporal logic formulas to define permissive strategies . Bou-Ammar et al . ( 2015 ) incorporated physical constraints to a safe on-line lifelong multi-task policy gradient learner operating in an adversarial framework . Lee et al . ( 2018 ) infer constraints from the actions of a demonstrator on the assumption that states visited by a demonstrator should be safe . Recently , Koller et al . ( 2018 ) proposed a Gaussian process ( GP ) -based RL that can make multi-step ahead safety predictions guarantee safe trajectories that satisfy system constraints . Aoude et al . ( 2013 ) propose a real-time GP-based path planning framework that can predict dynamic obstacles . Safe exploration techniques for reinforcement learning have been applied to practical problems . A robotic arm was trained using TRPO and a QP solver ( Pham et al. , 2017 ) and by teacher demonstrations where safety is maintained by updating a library of dangerous actions ( Martínez et al. , 2015 ) . Zhou and Li ( 2018 ) incorporate safety specifications expressed in a PCTL formula to the learning phase and use counterexamples to guide policy search in case of PCTL violations in learning from teacher demonstrations . Orseau and Armstrong ( 2016 ) consider the cases where an agent might learn to avoid ( or seek ) human intervention , thus limiting exploration and showing that Q-learning is safely interruptible , i.e. , an agent can ’ t learn to avoid ( or seek ) interruptions . Mhamdi et al . ( 2017 ) address the same problem for multi-agent systems in both the joint-action and independent learners ’ frameworks . Cizelj et al . ( 2011 ) and Cizelj and Belta ( 2012 ) consider safe vehicle driving in an environment with static and dynamic adversaries using PCTL formulas which describes mission objectives . Held et al . ( 2017 ) address the problem of simultaneously maximizing the expected return and managing expected damage for robot operation by adjusting the unsafety rate of a policy and the upper limit of physical ( state-based ) constraints that can incur damage , such as torque . In contrast to state-based constraints , Shen et al . ( 2018 ) , similar to our model , address action-based constraints in an interactive e-learning setting . However , their approach doesn ’ t directly address constraining action sequences . Rather , they constrain the total number of certain kinds of actions taken . Mazumder et al . ( 2018 ) use observed state-action pairs to predict the permissibility of actions given a state . However , their predictor doesn ’ t handle action sequences . State space augmentation was used by Oh et al . ( 2016 ) to predict future frames who applied an attention mechanism to the raw observations ( pixel values ) and augmented this with past state information retrieved from memory . Dosovitskiy and Koltun ( 2016 ) augmented high-dimensional state information with lower-dimensional “ measurement information ” relevant to the agent ’ s state . In contrast , our model uses direct or encoded information about action history for state augmentation .

Summary: This paper presents an DFA-based approach to constrain certain behavior of RL agents, where "behavior" is defined by a sequence of actions. This approach assumes that the developer has knowledge of what are good/bad behavior for a specific task and that the behavior can be checked by hand-coded DFAs or PDAs. During training, whenever such behavior is detected, the agent is given a negative reward, and the RL state is augmented with the DFA state. The authors experimented with different state augmentation methods (e.g. one-hot encoding, learned embedding) on 3 Atari tasks.