The Logical Options Framework

1 INTRODUCTION . To operate in the real world , intelligent agents must be able to make long-term plans by reasoning over symbolic abstractions while also maintaining the ability to react to low-level stimuli in their environment ( Zhang & Sridharan , 2020 ) . Many environments obey rules that can be represented as logical formulae ; e.g. , the rules a car follows while driving , or a recipe a chef follows to cook a dish . Traditional motion and path planning techniques struggle to formulate plans over these kinds of long-horizon tasks , but hierarchical approaches such as hierarchical reinforcement learning ( HRL ) can solve lengthy tasks by planning over both the high-level rules and the low-level environment . However , solving these problems involves trade-offs among multiple desirable properties , which we identify as satisfaction , optimality , and composability ( described below ) . Most of today ’ s algorithms sacrifice at least one of these objectives . For example , Reward Machines from Icarte et al . ( 2018 ) is satisfying and optimal , but not composable ; the options framework ( Sutton et al. , 1999 ) is composable and hierarchically optimal , but can not satisfy specifications . We introduce a new approach called the Logical Options Framework , which builds upon the options framework and aims to combine symbolic reasoning and low-level control to achieve satisfaction , optimality , and composability with as few compromises as possible . Furthermore , we show that our framework is compatible with a large variety of domains and planning algorithms , from discrete domains and value iteration to continuous domains and proximal policy optimization ( PPO ) . Satisfaction : An agent operating in an environment governed by rules must be able to satisfy the specified rules . Satisfaction is a concept from formal logic , in which the input to a logical formula causes the formula to evaluate to True . Logical formulae can encapsulate rules and tasks like the ones described in Fig . 1 , such as “ pick up the groceries ” and “ do not drive into a lake ” . In this paper , we state conditions under which our method is guaranteed to learn satisfying policies . Optimality : Optimality requires that the agent maximize its expected cumulative reward for each episode . In general , satisfaction can be achieved by rewarding the agent for satisfying the rules of the environment . In hierarchical planning there are several types of optimality , including hierarchical optimality ( optimal with respect to the hierarchy ) and optimality ( optimal with respect to everything ) . We prove in this paper that our method is hierarchically optimal and , under certain conditions , optimal . Composability : Our method also has the property of composability – once it has learned to satisfy a task , the learned model can be rearranged to satisfy a large variety of related tasks . More specifically , the rules of an environment can be factored into liveness and safety properties , which we discuss in Sec . 3 . The learned model can be adapted to satisfy any appropriate new liveness property . A shortcoming of many RL models is that they are not composable – trained to solve one specific task , they are incapable of handling even small variations in the task structure . However , the real world is a dynamic and unpredictable place , so the ability to automatically reason over as-yet-unseen tasks and rules is a crucial element of intelligence . The illustrations in Fig . 1 give an overview of our work . The environment is a world with a grocery store , your ( hypothetical ) kid , your house , and some lakes , and in which you , the agent , are driving a car . The propositions are divided into “ subgoals ” , representing events that can be achieved , such as going grocery shopping ; “ safety ” propositions , representing events that must be avoided ( driving into a lake ) ; and “ event ” propositions , corresponding to events that you have no control over ( receiving a phone call ) ( Fig . 1b ) . In this environment , you have to follow rules ( Fig . 1a ) . These rules can be converted into a logical formula , and from there into a finite state automaton ( FSA ) ( Fig . 1b ) . The Logical Options Framework learns an option for each subgoal ( illustrated by the arrows in Fig . 1c ) , and a metapolicy for choosing amongst the options to reach the goal state of the FSA . After learning , the options can be recombined to fulfill other tasks . 1.1 CONTRIBUTIONS . We introduce the Logical Options Framework ( LOF ) , which makes four contributions to the hierarchical reinforcement learning literature : 1 . The definition of a hierarchical semi-Markov Decision Process ( SMDP ) that is the product of a logical FSA and a low-level environment MDP . 2 . A planning algorithm for learning options and metapolicies for the SMDP that allows for the options to be composed to solve new tasks with only 10-50 retraining steps . 3 . Conditions and proofs for achieving satisfaction and optimality . 4 . Experiments on a discrete domain and a continuous domain on four tasks demonstrating satisfaction , optimality , and composability . 2 BACKGROUND . Linear Temporal Logic : We use linear temporal logic ( LTL ) to formally specify rules ( Clarke et al. , 2001 ) . LTL formulae are used only indirectly in LOF , as they are converted into automata that the algorithm uses directly . We chose to use LTL to represent rules because LTL corresponds closely to natural language and has proven to be a more natural way of expressing tasks and rules for engineers than designing FSAs by hand ( Kansou , 2019 ) . Formulae φ have the syntax grammar φ : = p | ¬φ | φ1 ∨ φ2 | © φ | φ1 U φ2 where p is a proposition ( a boolean-valued truth statement that can correspond to objects or events in the world ) , ¬ is negation , ∨ is disjunction , © is “ next ” , and U is “ until ” . The derived rules are conjunction ( ∧ ) , implication ( =⇒ ) , equivalence ( ↔ ) , “ eventually ” ( ♦φ ≡ TrueU φ ) and “ always ” ( φ ≡ ¬♦¬φ ) ( Baier & Katoen , 2008 ) . φ1 U φ2 means that φ1 is true until φ2 is true , ♦φ means that there is a time where φ is true and φ means that φ is always true . The Options Framework : The options framework is a framework for defining and solving semiMarkov Decision Processes ( SMDPs ) with a type of macro-action or subpolicy called an option ( Sutton et al. , 1999 ) . The inclusion of options in an MDP problem turns it into an SMDP problem , because actions are dependent not just on the previous state but also on the identity of the currently active option , which could have been initiated many time steps before the current time . An option o is a variable-length sequence of actions defined as o = ( I , π , β , Ro ( s ) , To ( s′|s ) ) . I ⊆ S is the initiation set of the option . π : S×A → [ 0 , 1 ] is the policy the option follows while the option is active . β : S → [ 0 , 1 ] is the termination condition . Ro ( s ) is the reward model of the option . To ( s ′|s ) is the transition model . A major challenge in option learning is that , in general , the number of time steps before the option terminates , k , is a random variable . With this in mind , Ro ( s ) is defined as the expected cumulative reward of option o given that the option is initiated in state s at time t and ends after k time steps . Letting rt be the reward received by the agent at t time steps from the beginning of the option , Ro ( s ) = E [ r1 + γr2 + . . . γ k−1rk ] ( 1 ) To ( s ′|s ) is the combined probability p ( s′ , k ) that option owill terminate at state s′ after k time steps : To ( s ′|s ) = ∞∑ k=1 p ( s′ , k ) γk ( 2 ) In the next section , we describe how Eqs . 1 and 2 can be simplified in the context of LOF . 3 LOGICAL OPTIONS FRAMEWORK . Here is a brief overview of how we will present our formulation of LOF : 1 . The LTL formula is decomposed into liveness and safety properties . The liveness property defines the task specification and the safety property defines the costs for violating rules . 2 . The propositions of the formula are divided into three types : subgoals , safety propositions , and event propositions . Subgoals are used to define tasks , and each subgoal is associated with its own option , whose goal is to achieve that subgoal . Safety propositions are used to define rules . Event propositions serve as control flow variables that affect the task . 3 . We define an SMDP that is the product of a low-level MDP and a high-level logical FSA . 4 . We describe how the logical options can be defined and learned . 5 . We present an algorithm for finding the hierarchically optimal policy on the SMDP . 6 . We state conditions under which satisfaction of the LTL specification is guaranteed , and we prove that the planning algorithm converges to an optimal policy by showing that the hierarchically optimal SMDP policy is the same as the optimal MDP policy . The Logic Formula : LTL formulae can be translated into Büchi automata using automatic translation tools such as SPOT ( Duret-Lutz et al. , 2016 ) . All Büchi automata can be decomposed into liveness and safety properties ( Alpern & Schneider , 1987 ) . To simplify the formulation , we assume that the LTL formula itself can be divided into liveness and safety formulae , φ = φliveness∧φsafety . For the case where the LTL formula can not be factored into independent formulae , please see Appendix A . The liveness property describes “ things that must happen ” to satisfy the LTL formula . It is a task specification , and it is used in planning to determine which subgoals the agent must achieve . The safety property describes “ things that can never happen ” and is used to define costs for violating the rules . In LOF , the liveness property must be written using a finite-trace subset of LTL called syntactically co-safe LTL ( Bhatia et al. , 2010 ) , in which the ( “ always ” ) operator is not allowed and © , U , and ♦ are only used in positive normal form . This way , the liveness property can be satisfied by finite-length sequences of propositions , and the property can be represented as an FSA . Propositions : Propositions are boolean-valued truth statements corresponding to goals , objects , and events in the environment . We distinguish between three types of propositions : subgoals PG , safety propositions PS , and event propositions PE . Subgoal propositions are propositions that must be achieved in order to satisfy the liveness property . They are associated with goals such as “ the agent is at the grocery store ” . They only appear in φliveness . Each subgoal may only be associated with one state . Note that in general , it may be impossible to avoid having subgoals appear in φsafety . Appendix A describes how to deal with this scenario . Safety propositions are propositions that the agent must avoid – for example , driving into a lake . They only appear in φsafety . Event propositions have a set value that affects the task specification – for example , whether or not a phone call is received . They may occur in φliveness , and , with some extensions that are described in Appendix A , in φsafety . Although in the fully observable setting , event propositions are somewhat trivial , in the partially observable setting , where the value of the event proposition is revealed to the agent at a random point in time , they are very useful . Our optimality guarantees only apply in the fully observable setting ; however , LOF ’ s properties of satisfaction and composability still apply in the partially observable setting . The goal state of the liveness property must be reachable from every other state using only subgoals . This means that no matter what the values of the event propositions are , it is always possible for the agent to satisfy the liveness property . Proposition labeling functions relate states to the set of propositions that are true at that state : TPG : S → 2PG , TPS : S → 2PS ; for event propositions , a function identifies the set of true propositions , TPE : 2 PE → { 0 , 1 } . Hierarchical SMDP : LOF works by defining a hierarchical semi-Markov Decision Process ( SMDP ) , learning the options , and then planning over the options . The high-level part of the hierarchy is defined by an FSA specified using LTL . The low level is an environment MDP . We assume that the high-level LTL specification φ can be decomposed into a liveness property φliveness and a safety property φsafety . The set of propositions P is the union of the sets of subgoals PG , safety propositions PS , and event propositions PE . We assume that the liveness property can be translated into an FSA T = ( F , P , TF , RF , f0 , fg ) . F is the set of automaton states ; P is the set of propositions ; TF is the transition function relating the current state and proposition to the next state , TF : F × P × F → [ 0 , 1 ] . In practice , TF is deterministic despite our use of probabilistic notation . We assume that there is a single initial state f0 and final state fg , and that the goal state fg is reachable from every state f ∈ F using only subgoals . There is also a reward function that assigns a reward to every state , RF : F → R. In our experiments , we assume that the safety property takes the form ∧ ps∈PS ¬ps . This simple safety property implies that every safety proposition is not allowed , and that the safety propositions have associated costs , RS : 2PS → R. φsafety is not limited to this simple case ; the general case is covered in Appendix A . There is a low-level environment MDP E = ( S , A , RE , TE , γ ) . S is the state space and A is the action space . They can be discrete or continuous . RE : S × A → R is a low-level reward function that characterizes , for example , distance or actuation costs . RE is a combination of the safety reward function RS and RE , e.g . RE ( s , a ) = RE ( s , a ) + RS ( TPS ( s ) ) . The transition function of the environment is TE : S ×A× S → [ 0 , 1 ] . From these parts we define a hierarchical SMDPM = ( S×F , A , P , O , TE×TP×TF , RSMDP , γ ) . The hierarchical state space contains two elements : low-level states S and FSA states F . The action space isA . The set of propositions is P . The set of options ( one option associated with each subgoal in PG ) is O . The transition function consists of the low-level environment transitions TE and the FSA transitions TF . TP = TPG × TPS × TPE . We classify TP , relating states to propositions , as a transition function because it helps to determine when FSA transitions occur . The transitions are applied in the order TE , TP , TF . The reward function RSMDP ( f , s , o ) = RF ( f ) Ro ( s ) , so RF ( f ) is a weighting on the option rewards . Lastly , the SMDP has the same discount factor γ as E . Planning is done on the SMDP in two steps : first , the options O are learned over E using an Algorithm 1 Learning and Planning with Logical Options 1 : procedure LEARNING-AND-PLANNING-WITH-LOGICAL-OPTIONS 2 : Given : Propositions P partitioned into subgoals PG , safety props PS , and event props PE Logical FSA T = ( F , PG × PE , TF , RF , f0 , fg ) derived from φliveness Low-level MDP E = ( S , A , RE , TE , γ ) , where RE ( s , a ) = RE ( s , a ) +RS ( TPS ( s ) ) combines the environment and safety rewards Proposition labeling functions TPG : S → 2PG , TPS : S → 2PS , and TPE : 2 PE → { 0 , 1 } 3 : To learn : 4 : Set of options O , one for each subgoal proposition p ∈ PG 5 : Metapolicy µ ( f , s , o ) along with Q ( f , s , o ) and V ( f , s ) 6 : Learn logical options : 7 : For every p in PG , learn an option for achieving p , op = ( Iop , πop , βop , Rop , Top ) 8 : Iop = S 9 : βop = { 1 if p ∈ TPG ( s ) 0 otherwise 10 : πop = optimal policy on E with rollouts terminating when p ∈ TPG ( s ) 11 : Top ( s ′|s ) = { Eγk if p ∈ TPG ( s′ ) , where k is number of time steps to reach p 0 otherwise 12 : Rop ( s ) = E [ RE ( s , a1 ) + γRE ( s1 , a2 ) + · · ·+ γk−1RE ( sk−1 , ak ) ] 13 : Find a metapolicy µ over the options : 14 : Initialize Q : F × S ×O → R and V : F × S → R to 0 15 : For ( k , f , s ) ∈ [ 1 , . . . , n ] ×F × S : 16 : For o ∈ O : 17 : Qk ( f , s , o ) ← RF ( f ) Ro ( s ) + ∑ f ′∈F ∑ p̄e∈2PE ∑ s′∈S TF ( f ′|f , TPG ( s′ ) , p̄e ) 18 : TPE ( p̄e ) To ( s ′|s ) Vk−1 ( f ′ , s′ ) 19 : Vk ( f , s ) ← max o∈O Qk ( f , s , o ) 20 : µ ( f , s , o ) = arg max o∈O Q ( f , s , o ) 21 : return Options O , metapolicy µ ( f , s , o ) and Q- and value functions Q ( f , s , o ) , V ( f , s ) 22 : end procedure appropriate policy-learning algorithm such as PPO or Reward Machines . Next , a metapolicy over the task specification T is found using the learned options and the reward function RSMDP . Logical Options : The first step of Alg . 1 is to learn the logical options . We associate every subgoal p with an option op = ( Iop , πop , βop , Rop , Top ) . These terms are defined starting at Alg . 1 line 6 . Every op has a policy πop whose goal is to reach the state sp where p is true . Options are learned by training on the environment MDP E and terminating only when sp is reached . As we discuss in Sec . 3.1 , under certain conditions the optimal option policy is guaranteed to always terminate at the subgoal . This allows us to simplify the transition model of Eq . 2 to the form in Alg . 1 line 11 . In the experiments , we further simplify this expression by setting γ = 1 . Logical Value Iteration : After finding the logical options , the next step is to find a policy for FSA T over the options , as described in Alg . 1 line 13 . A value function and Q-function are found for the SMDP using the Bellman update equations : Qk ( f , s , o ) ← RF ( f ) Ro ( s ) + ∑ f ′∈F ∑ p̄e∈2PE ∑ s′∈S TF ( f ′|f , TPG ( s′ ) , p̄e ) TPE ( p̄e ) To ( s ′|s ) Vk−1 ( f ′ , s′ ) ( 3 ) Vk ( f , s ) ← max o∈O Qk ( f , s , o ) ( 4 ) Eq . 3 differs from the generic equations for SMDP value iteration in that the transition function has two extra components , ∑ f ′∈F TF ( f ′|f , TP ( s′ ) , p̄e ) and ∑ p̄e∈2PE TPE ( p̄e ) . The equations are derived from Araki et al . ( 2019 ) and the fact that , on every step in the environment , three transitions are applied : the option transition To , the event proposition “ transition ” TPE , and the FSA transition TF . Note that Ro ( s ) and To ( s′|s ) compress the consequences of choosing an option o at a state s from a multi-step trajectory into two real-valued numbers , allowing for more efficient planning .

Summary: The authors propose the Logical Options Framework (LOF) --- a framework for reasoning over high-level plans and learning low-level control policies. This framework uses Linear Temporal Logic (LTL) to specify properties (high-level tasks) in terms of propositions. The authors propose a framework in which a separate sub-task policy is learnt to accomplish each such sub-task proposition. These low-level control policies may be reused, without further training, to accomplish new high-level tasks by performing value-iteration in the proposed Hierarchical SMDP. Experimental results demonstrate the method’s effectiveness for several tasks and experimental domains.