The Logical Options Framework

1 INTRODUCTION . To operate in the real world , intelligent agents must be able to make long-term plans by reasoning over symbolic abstractions while also maintaining the ability to react to low-level stimuli in their environment ( Zhang & Sridharan , 2020 ) . Many environments obey rules that can be represented as logical formulae ; e.g. , the rules a car follows while driving , or a recipe a chef follows to cook a dish . Traditional motion and path planning techniques struggle to formulate plans over these kinds of long-horizon tasks , but hierarchical approaches such as hierarchical reinforcement learning ( HRL ) can solve lengthy tasks by planning over both the high-level rules and the low-level environment . However , solving these problems involves trade-offs among multiple desirable properties , which we identify as satisfaction , optimality , and composability ( described below ) . Most of today ’ s algorithms sacrifice at least one of these objectives . For example , Reward Machines from Icarte et al . ( 2018 ) is satisfying and optimal , but not composable ; the options framework ( Sutton et al. , 1999 ) is composable and hierarchically optimal , but can not satisfy specifications . We introduce a new approach called the Logical Options Framework , which builds upon the options framework and aims to combine symbolic reasoning and low-level control to achieve satisfaction , optimality , and composability with as few compromises as possible . Furthermore , we show that our framework is compatible with a large variety of domains and planning algorithms , from discrete domains and value iteration to continuous domains and proximal policy optimization ( PPO ) . Satisfaction : An agent operating in an environment governed by rules must be able to satisfy the specified rules . Satisfaction is a concept from formal logic , in which the input to a logical formula causes the formula to evaluate to True . Logical formulae can encapsulate rules and tasks like the ones described in Fig . 1 , such as “ pick up the groceries ” and “ do not drive into a lake ” . In this paper , we state conditions under which our method is guaranteed to learn satisfying policies . Optimality : Optimality requires that the agent maximize its expected cumulative reward for each episode . In general , satisfaction can be achieved by rewarding the agent for satisfying the rules of the environment . In hierarchical planning there are several types of optimality , including hierarchical optimality ( optimal with respect to the hierarchy ) and optimality ( optimal with respect to everything ) . We prove in this paper that our method is hierarchically optimal and , under certain conditions , optimal . Composability : Our method also has the property of composability – once it has learned to satisfy a task , the learned model can be rearranged to satisfy a large variety of related tasks . More specifically , the rules of an environment can be factored into liveness and safety properties , which we discuss in Sec . 3 . The learned model can be adapted to satisfy any appropriate new liveness property . A shortcoming of many RL models is that they are not composable – trained to solve one specific task , they are incapable of handling even small variations in the task structure . However , the real world is a dynamic and unpredictable place , so the ability to automatically reason over as-yet-unseen tasks and rules is a crucial element of intelligence . The illustrations in Fig . 1 give an overview of our work . The environment is a world with a grocery store , your ( hypothetical ) kid , your house , and some lakes , and in which you , the agent , are driving a car . The propositions are divided into “ subgoals ” , representing events that can be achieved , such as going grocery shopping ; “ safety ” propositions , representing events that must be avoided ( driving into a lake ) ; and “ event ” propositions , corresponding to events that you have no control over ( receiving a phone call ) ( Fig . 1b ) . In this environment , you have to follow rules ( Fig . 1a ) . These rules can be converted into a logical formula , and from there into a finite state automaton ( FSA ) ( Fig . 1b ) . The Logical Options Framework learns an option for each subgoal ( illustrated by the arrows in Fig . 1c ) , and a metapolicy for choosing amongst the options to reach the goal state of the FSA . After learning , the options can be recombined to fulfill other tasks . 1.1 CONTRIBUTIONS . We introduce the Logical Options Framework ( LOF ) , which makes four contributions to the hierarchical reinforcement learning literature : 1 . The definition of a hierarchical semi-Markov Decision Process ( SMDP ) that is the product of a logical FSA and a low-level environment MDP . 2 . A planning algorithm for learning options and metapolicies for the SMDP that allows for the options to be composed to solve new tasks with only 10-50 retraining steps . 3 . Conditions and proofs for achieving satisfaction and optimality . 4 . Experiments on a discrete domain and a continuous domain on four tasks demonstrating satisfaction , optimality , and composability . 2 BACKGROUND . Linear Temporal Logic : We use linear temporal logic ( LTL ) to formally specify rules ( Clarke et al. , 2001 ) . LTL formulae are used only indirectly in LOF , as they are converted into automata that the algorithm uses directly . We chose to use LTL to represent rules because LTL corresponds closely to natural language and has proven to be a more natural way of expressing tasks and rules for engineers than designing FSAs by hand ( Kansou , 2019 ) . Formulae φ have the syntax grammar φ : = p | ¬φ | φ1 ∨ φ2 | © φ | φ1 U φ2 where p is a proposition ( a boolean-valued truth statement that can correspond to objects or events in the world ) , ¬ is negation , ∨ is disjunction , © is “ next ” , and U is “ until ” . The derived rules are conjunction ( ∧ ) , implication ( =⇒ ) , equivalence ( ↔ ) , “ eventually ” ( ♦φ ≡ TrueU φ ) and “ always ” ( φ ≡ ¬♦¬φ ) ( Baier & Katoen , 2008 ) . φ1 U φ2 means that φ1 is true until φ2 is true , ♦φ means that there is a time where φ is true and φ means that φ is always true . The Options Framework : The options framework is a framework for defining and solving semiMarkov Decision Processes ( SMDPs ) with a type of macro-action or subpolicy called an option ( Sutton et al. , 1999 ) . The inclusion of options in an MDP problem turns it into an SMDP problem , because actions are dependent not just on the previous state but also on the identity of the currently active option , which could have been initiated many time steps before the current time . An option o is a variable-length sequence of actions defined as o = ( I , π , β , Ro ( s ) , To ( s′|s ) ) . I ⊆ S is the initiation set of the option . π : S×A → [ 0 , 1 ] is the policy the option follows while the option is active . β : S → [ 0 , 1 ] is the termination condition . Ro ( s ) is the reward model of the option . To ( s ′|s ) is the transition model . A major challenge in option learning is that , in general , the number of time steps before the option terminates , k , is a random variable . With this in mind , Ro ( s ) is defined as the expected cumulative reward of option o given that the option is initiated in state s at time t and ends after k time steps . Letting rt be the reward received by the agent at t time steps from the beginning of the option , Ro ( s ) = E [ r1 + γr2 + . . . γ k−1rk ] ( 1 ) To ( s ′|s ) is the combined probability p ( s′ , k ) that option owill terminate at state s′ after k time steps : To ( s ′|s ) = ∞∑ k=1 p ( s′ , k ) γk ( 2 ) In the next section , we describe how Eqs . 1 and 2 can be simplified in the context of LOF . 3 LOGICAL OPTIONS FRAMEWORK . Here is a brief overview of how we will present our formulation of LOF : 1 . The LTL formula is decomposed into liveness and safety properties . The liveness property defines the task specification and the safety property defines the costs for violating rules . 2 . The propositions of the formula are divided into three types : subgoals , safety propositions , and event propositions . Subgoals are used to define tasks , and each subgoal is associated with its own option , whose goal is to achieve that subgoal . Safety propositions are used to define rules . Event propositions serve as control flow variables that affect the task . 3 . We define an SMDP that is the product of a low-level MDP and a high-level logical FSA . 4 . We describe how the logical options can be defined and learned . 5 . We present an algorithm for finding the hierarchically optimal policy on the SMDP . 6 . We state conditions under which satisfaction of the LTL specification is guaranteed , and we prove that the planning algorithm converges to an optimal policy by showing that the hierarchically optimal SMDP policy is the same as the optimal MDP policy . The Logic Formula : LTL formulae can be translated into Büchi automata using automatic translation tools such as SPOT ( Duret-Lutz et al. , 2016 ) . All Büchi automata can be decomposed into liveness and safety properties ( Alpern & Schneider , 1987 ) . To simplify the formulation , we assume that the LTL formula itself can be divided into liveness and safety formulae , φ = φliveness∧φsafety . For the case where the LTL formula can not be factored into independent formulae , please see Appendix A . The liveness property describes “ things that must happen ” to satisfy the LTL formula . It is a task specification , and it is used in planning to determine which subgoals the agent must achieve . The safety property describes “ things that can never happen ” and is used to define costs for violating the rules . In LOF , the liveness property must be written using a finite-trace subset of LTL called syntactically co-safe LTL ( Bhatia et al. , 2010 ) , in which the ( “ always ” ) operator is not allowed and © , U , and ♦ are only used in positive normal form . This way , the liveness property can be satisfied by finite-length sequences of propositions , and the property can be represented as an FSA . Propositions : Propositions are boolean-valued truth statements corresponding to goals , objects , and events in the environment . We distinguish between three types of propositions : subgoals PG , safety propositions PS , and event propositions PE . Subgoal propositions are propositions that must be achieved in order to satisfy the liveness property . They are associated with goals such as “ the agent is at the grocery store ” . They only appear in φliveness . Each subgoal may only be associated with one state . Note that in general , it may be impossible to avoid having subgoals appear in φsafety . Appendix A describes how to deal with this scenario . Safety propositions are propositions that the agent must avoid – for example , driving into a lake . They only appear in φsafety . Event propositions have a set value that affects the task specification – for example , whether or not a phone call is received . They may occur in φliveness , and , with some extensions that are described in Appendix A , in φsafety . Although in the fully observable setting , event propositions are somewhat trivial , in the partially observable setting , where the value of the event proposition is revealed to the agent at a random point in time , they are very useful . Our optimality guarantees only apply in the fully observable setting ; however , LOF ’ s properties of satisfaction and composability still apply in the partially observable setting . The goal state of the liveness property must be reachable from every other state using only subgoals . This means that no matter what the values of the event propositions are , it is always possible for the agent to satisfy the liveness property . Proposition labeling functions relate states to the set of propositions that are true at that state : TPG : S → 2PG , TPS : S → 2PS ; for event propositions , a function identifies the set of true propositions , TPE : 2 PE → { 0 , 1 } . Hierarchical SMDP : LOF works by defining a hierarchical semi-Markov Decision Process ( SMDP ) , learning the options , and then planning over the options . The high-level part of the hierarchy is defined by an FSA specified using LTL . The low level is an environment MDP . We assume that the high-level LTL specification φ can be decomposed into a liveness property φliveness and a safety property φsafety . The set of propositions P is the union of the sets of subgoals PG , safety propositions PS , and event propositions PE . We assume that the liveness property can be translated into an FSA T = ( F , P , TF , RF , f0 , fg ) . F is the set of automaton states ; P is the set of propositions ; TF is the transition function relating the current state and proposition to the next state , TF : F × P × F → [ 0 , 1 ] . In practice , TF is deterministic despite our use of probabilistic notation . We assume that there is a single initial state f0 and final state fg , and that the goal state fg is reachable from every state f ∈ F using only subgoals . There is also a reward function that assigns a reward to every state , RF : F → R. In our experiments , we assume that the safety property takes the form ∧ ps∈PS ¬ps . This simple safety property implies that every safety proposition is not allowed , and that the safety propositions have associated costs , RS : 2PS → R. φsafety is not limited to this simple case ; the general case is covered in Appendix A . There is a low-level environment MDP E = ( S , A , RE , TE , γ ) . S is the state space and A is the action space . They can be discrete or continuous . RE : S × A → R is a low-level reward function that characterizes , for example , distance or actuation costs . RE is a combination of the safety reward function RS and RE , e.g . RE ( s , a ) = RE ( s , a ) + RS ( TPS ( s ) ) . The transition function of the environment is TE : S ×A× S → [ 0 , 1 ] . From these parts we define a hierarchical SMDPM = ( S×F , A , P , O , TE×TP×TF , RSMDP , γ ) . The hierarchical state space contains two elements : low-level states S and FSA states F . The action space isA . The set of propositions is P . The set of options ( one option associated with each subgoal in PG ) is O . The transition function consists of the low-level environment transitions TE and the FSA transitions TF . TP = TPG × TPS × TPE . We classify TP , relating states to propositions , as a transition function because it helps to determine when FSA transitions occur . The transitions are applied in the order TE , TP , TF . The reward function RSMDP ( f , s , o ) = RF ( f ) Ro ( s ) , so RF ( f ) is a weighting on the option rewards . Lastly , the SMDP has the same discount factor γ as E . Planning is done on the SMDP in two steps : first , the options O are learned over E using an Algorithm 1 Learning and Planning with Logical Options 1 : procedure LEARNING-AND-PLANNING-WITH-LOGICAL-OPTIONS 2 : Given : Propositions P partitioned into subgoals PG , safety props PS , and event props PE Logical FSA T = ( F , PG × PE , TF , RF , f0 , fg ) derived from φliveness Low-level MDP E = ( S , A , RE , TE , γ ) , where RE ( s , a ) = RE ( s , a ) +RS ( TPS ( s ) ) combines the environment and safety rewards Proposition labeling functions TPG : S → 2PG , TPS : S → 2PS , and TPE : 2 PE → { 0 , 1 } 3 : To learn : 4 : Set of options O , one for each subgoal proposition p ∈ PG 5 : Metapolicy µ ( f , s , o ) along with Q ( f , s , o ) and V ( f , s ) 6 : Learn logical options : 7 : For every p in PG , learn an option for achieving p , op = ( Iop , πop , βop , Rop , Top ) 8 : Iop = S 9 : βop = { 1 if p ∈ TPG ( s ) 0 otherwise 10 : πop = optimal policy on E with rollouts terminating when p ∈ TPG ( s ) 11 : Top ( s ′|s ) = { Eγk if p ∈ TPG ( s′ ) , where k is number of time steps to reach p 0 otherwise 12 : Rop ( s ) = E [ RE ( s , a1 ) + γRE ( s1 , a2 ) + · · ·+ γk−1RE ( sk−1 , ak ) ] 13 : Find a metapolicy µ over the options : 14 : Initialize Q : F × S ×O → R and V : F × S → R to 0 15 : For ( k , f , s ) ∈ [ 1 , . . . , n ] ×F × S : 16 : For o ∈ O : 17 : Qk ( f , s , o ) ← RF ( f ) Ro ( s ) + ∑ f ′∈F ∑ p̄e∈2PE ∑ s′∈S TF ( f ′|f , TPG ( s′ ) , p̄e ) 18 : TPE ( p̄e ) To ( s ′|s ) Vk−1 ( f ′ , s′ ) 19 : Vk ( f , s ) ← max o∈O Qk ( f , s , o ) 20 : µ ( f , s , o ) = arg max o∈O Q ( f , s , o ) 21 : return Options O , metapolicy µ ( f , s , o ) and Q- and value functions Q ( f , s , o ) , V ( f , s ) 22 : end procedure appropriate policy-learning algorithm such as PPO or Reward Machines . Next , a metapolicy over the task specification T is found using the learned options and the reward function RSMDP . Logical Options : The first step of Alg . 1 is to learn the logical options . We associate every subgoal p with an option op = ( Iop , πop , βop , Rop , Top ) . These terms are defined starting at Alg . 1 line 6 . Every op has a policy πop whose goal is to reach the state sp where p is true . Options are learned by training on the environment MDP E and terminating only when sp is reached . As we discuss in Sec . 3.1 , under certain conditions the optimal option policy is guaranteed to always terminate at the subgoal . This allows us to simplify the transition model of Eq . 2 to the form in Alg . 1 line 11 . In the experiments , we further simplify this expression by setting γ = 1 . Logical Value Iteration : After finding the logical options , the next step is to find a policy for FSA T over the options , as described in Alg . 1 line 13 . A value function and Q-function are found for the SMDP using the Bellman update equations : Qk ( f , s , o ) ← RF ( f ) Ro ( s ) + ∑ f ′∈F ∑ p̄e∈2PE ∑ s′∈S TF ( f ′|f , TPG ( s′ ) , p̄e ) TPE ( p̄e ) To ( s ′|s ) Vk−1 ( f ′ , s′ ) ( 3 ) Vk ( f , s ) ← max o∈O Qk ( f , s , o ) ( 4 ) Eq . 3 differs from the generic equations for SMDP value iteration in that the transition function has two extra components , ∑ f ′∈F TF ( f ′|f , TP ( s′ ) , p̄e ) and ∑ p̄e∈2PE TPE ( p̄e ) . The equations are derived from Araki et al . ( 2019 ) and the fact that , on every step in the environment , three transitions are applied : the option transition To , the event proposition “ transition ” TPE , and the FSA transition TF . Note that Ro ( s ) and To ( s′|s ) compress the consequences of choosing an option o at a state s from a multi-step trajectory into two real-valued numbers , allowing for more efficient planning .

Summary: This paper is on a new RL framework that leverages logical reasoning to improve the learning performance of RL agents. In particular, the knowledge is encoded using LTL, and includes both safety knowledge (used for reward function definition) and liveness knowledge (used for constructing FSA). THe developed framework has been evaluated using tasks in both discrete and continuous domains, where the RL agent was realized using Q-learning and PPO respectively. The framework was compared with baselines including another LTL-based RL methods (Reward Machines). Results show that LOF performed better than the baseline methods in learning rate and policy quality (in most cases).