Automata Guided Skill Composition

1 INTRODUCTION . Policies learned using reinforcement learning aim to maximize the given reward function and are often difficult to transfer to other problem domains . Skill composition is the process of constructing new skills out of existing ones ( policies ) with little to no additional learning . In stochastic optimal control , this idea has been adopted by Todorov ( 2009 ) and Da Silva et al . ( 2009 ) to construct provably optimal control laws based on linearly solvable Markov decision processes . Temporal logic ( TL ) is a formal language commonly used in software and digital circuit verification ( Baier & Katoen , 2008 ) as well as formal synthesis ( Belta et al. , 2017 ) . It allows for convenient expression of complex behaviors and causal relationships . TL has been used by ( Tabuada & Pappas , 2004 ) , ( Fainekos et al. , 2006 ) , ( Fainekos et al. , 2005 ) to synthesize provably correct control policies . Aksaray et al . ( 2016 ) have also combined TL with Q-learning to learn satisfiable policies in discrete state and action spaces . We make the distinction between skill composition and multi-task learning/meta-learning where the latter often requires a predefined set of tasks/task distributions to learn and generalize from , whereas the focus of the former is to construct new policies from a library of already learned policies that achieve new tasks ( often some combination of the constituent tasks ) with little to no additional constraints on task distribution at learning time . In this work , we focus on skill composition with policies learned using automata guided reinforcement learning ( Li et al. , 2018 ) . We adopt the syntactically co-safe truncated linear temporal logic ( scTLTL ) as the task specification language . Compared to most heuristic reward structures used in the RL literature , formal specification language has the advantage of semantic rigor and interpretability . In our framework , skill composition is accomplished by taking the product of finite state automata ( FSA ) . Instead of interpolating/extrapolating among learned skills/latent features , our method is based on graph manipulation of the FSA . Therefore , the outcome is much more transparent . Compared with previous work on skill composition , we impose no constraints on the policy representation or the problem class . We validate our framework in simulation ( discrete state and action spaces ) and experimentally on a Baxter robot ( continuous state and action spaces ) . 2 RELATED WORK . Recent efforts in skill composition have mainly adopted the approach of combining value functions learned using different rewards . Peng et al . ( 2018 ) constructs a composite policy by combining the value functions of individual policies using the Boltzmann distribution . With a similar goal , Zhu et al . ( 2017 ) achieves task space transfer using deep successor representations ( Kulkarni et al. , 2016 ) . However , it is required that the reward function be represented as a linear combination of state-action features . Haarnoja et al . ( 2018 ) have showed that when using energy-based models ( Haarnoja et al. , 2017 ) , an approximately optimal composite policy can result from taking the average of the Q-functions of existing policies . The resulting composite policy achieves the −AND− task composition i.e . the composite policy maximizes the average reward of individual tasks . van Niekerk et al . ( 2018 ) have taken this idea a step further and showed that by combining individual Q-functions using the log-sum-exponential function , the −OR− task composition ( the composite policy maximizes the ( soft ) maximum of the reward of constituent tasks ) can be achieved optimally . We build on the results of ( van Niekerk et al. , 2018 ) and show that incorporating temporal logic allows us to compose tasks of greater logical complexity with higher interpretability . Our composite policy is optimal in both −AND− and −OR− task compositions . 3 PRELIMINARIES . 3.1 ENTROPY-REGULARIZED REINFORCEMENT LEARNING AND Q-COMPOSITION . We start with the definition of a Markov Decision Process . Definition 1 . An MDP is defined as a tupleM = 〈S , A , p ( ·|· , · ) , r ( · , · , · ) 〉 , where S ⊆ IRn is the state space ; A ⊆ IRm is the action space ( S andA can also be discrete sets ) ; p : S×A×S → [ 0 , 1 ] is the transition function with p ( s′|s , a ) being the conditional probability density of taking action a ∈ A at state s ∈ S and ending up in state s′ ∈ S ; r : S×A×S → IR is the reward function with r ( s , a , s′ ) being the reward obtained by executing action a at state s and transitioning to s′ . In entropy-regularized reinforcement learning ( Schulman et al. , 2017 ) , the goal is to maximize the following objective J ( π ) = T−1∑ t=0 Eπ [ rt + αH ( π ( ·|st ) ) ] , ( 1 ) where π : S×A→ [ 0 , 1 ] is a stochastic policy . Eπ is the expectation following π. H ( π ( ·|st ) ) is the entropy of π. α is the temperature parameter . In the limit α→ 0 , Equation ( 1 ) becomes the standard RL objective . The soft Q-learning algorithm introduced by ( Haarnoja et al. , 2017 ) optimizes the above objective and finds a policy represented by an energy-based model π ? ( at|st ) ∝ exp ( −E ( st , at ) ) ( 2 ) where E ( st , at ) is an energy function that can be represented by a function approximator . Let rentt = rt + αH ( π ( ·|st ) ) , the state-action value function ( Q-function ) following π is defined as Qπ ( s , a ) = Eπ [ ∑T−1 t=0 r ent t |s0 = s , a0 = a ] . Suppose we have a set of n tasks indexed by i , i ∈ { 0 , ... , n } , each task is defined by an MDP Mi that differs only in their reward function ri . Let Q π ? i α be the optimal entropy-regularized Q-function . Authors of ( van Niekerk et al. , 2018 ) provide the following results Theorem 1 . Define vectors r = [ r1 , ... , rn ] , Qπ ? α = [ Q π ? 1 α , ... , Q π ? n α ] . Given a set of nonnegative weights w with ||w|| = 1 , the optimal Q-function for a new task defined by r = α log ( || exp ( r/α ) ||w ) is given by Qπ ? α = α log ( || exp ( Qπ ? α /α ) ||w ) , ( 3 ) where || · ||w is the weighted 1-norm . The authors proceed to provide the following corollary Corollary 1. maxQπ ? α ↑ Qπ ? 0 as α → 0 , where Qπ ? 0 is the optimal Q-function for the objective J ( π ) = ∑T−1 t=0 Eπ [ rt ] . Corollary 1 states that in the low temperature limit , the maximum of the optimal entropy-regularized Q-functions approaches the standard optimal Q-function 3.2 SCTLTL AND FINITE STATE AUTOMATA . We consider tasks specified with syntactically co-safe Truncated Linear Temporal Logic ( scTLTL ) which is derived from truncated linear temporal logic ( TLTL ) ( Li et al. , 2018 ) . The syntax of scTLTL is defined as φ : = > | f ( s ) < c | ¬φ | φ ∧ ψ | ♦φ | φU ψ | φ T ψ | © φ ( 4 ) where > is the True Boolean constant . s ∈ S is a MDP state in Definition 1. f ( s ) < c is a predicate over the MDP states where c ∈ IR . ¬ ( negation/not ) , ∧ ( conjunction/and ) are Boolean connectives . ♦ ( eventually ) , U ( until ) , T ( then ) , © ( next ) , are temporal operators.⇒ ( implication ) and and ∨ ( disjunction/or ) can be derived from the above operators . We denote st ∈ S to be the MDP state at time t , and st : t+k to be a sequence of states ( state trajectory ) from time t to t+ k , i.e. , st : t+k = stst+1 ... st+k . The Boolean semantics of scTLTL is defined as : st : t+k |= f ( s ) < c ⇔ f ( st ) < c , st : t+k |= ¬φ ⇔ ¬ ( st : t+k |= φ ) , st : t+k |= φ⇒ ψ ⇔ ( st : t+k |= φ ) ⇒ ( st : t+k |= ψ ) , st : t+k |= φ ∧ ψ ⇔ ( st : t+k |= φ ) ∧ ( st : t+k |= ψ ) , st : t+k |= φ ∨ ψ ⇔ ( st : t+k |= φ ) ∨ ( st : t+k |= ψ ) , st : t+k |=©φ ⇔ ( st+1 : t+k |= φ ) ∧ ( k > 0 ) , st : t+k |= ♦φ ⇔ ∃t′ ∈ [ t , t+ k ) st′ : t+k |= φ , st : t+k |= φ U ψ ⇔ ∃t′ ∈ [ t , t+ k ) s.t . st′ : t+k |= ψ ∧ ( ∀t′′ ∈ [ t , t′ ) st′′ : t′ |= φ ) , st : t+k |= φ T ψ ⇔ ∃t′ ∈ [ t , t+ k ) s.t . st′ : t+k |= ψ ∧ ( ∃t′′ ∈ [ t , t′ ) st′′ : t′ |= φ ) . A trajectory s0 : T is said to satisfy formula φ if s0 : T |= φ . The quantitative semantics ( also referred to as robustness ) is defined recursively as ρ ( st : t+k , > ) = ρmax , ρ ( st : t+k , f ( st ) < c ) = c− f ( st ) , ρ ( st : t+k , ¬φ ) = − ρ ( st : t+k , φ ) , ρ ( st : t+k , φ ⇒ ψ ) = max ( −ρ ( st : t+k , φ ) , ρ ( st : t+k , ψ ) ) ρ ( st : t+k , φ1 ∧ φ2 ) = min ( ρ ( st : t+k , φ1 ) , ρ ( st : t+k , φ2 ) ) , ρ ( st : t+k , φ1 ∨ φ2 ) = max ( ρ ( st : t+k , φ1 ) , ρ ( st : t+k , φ2 ) ) , ρ ( st : t+k , ©φ ) = ρ ( st+1 : t+k , φ ) ( k > 0 ) , ρ ( st : t+k , ♦φ ) = max t′∈ [ t , t+k ) ( ρ ( st′ : t+k , φ ) ) , ρ ( st : t+k , φ U ψ ) = max t′∈ [ t , t+k ) ( min ( ρ ( st′ : t+k , ψ ) , min t′′∈ [ t , t′ ) ρ ( st′′ : t′ , φ ) ) ) , ρ ( st : t+k , φ T ψ ) = max t′∈ [ t , t+k ) ( min ( ρ ( st′ : t+k , ψ ) , max t′′∈ [ t , t′ ) ρ ( st′′ : t′ , φ ) ) ) , where ρmax represents the maximum robustness value . A robustness of greater than zero implies that st : t+k satisfies φ and vice versa ( ρ ( st : t+k , φ ) > 0 ⇒ st : t+k |= φ and ρ ( st : t+k , φ ) < 0 ⇒ st : t+k 6|= φ ) . The robustness is used as a measure of the level of satisfaction of a trajectory s0 : T with respect to a scTLTL formula φ . Definition 2 . An FSA corresponding to a scTLTL formula φ. is defined as a tuple Aφ = 〈Qφ , Ψφ , qφ,0 , pφ ( ·|· ) , Fφ〉 , where Qφ is a set of automaton states ; Ψφ is the input alphabet ( a set of first order logic formula ) ; qφ,0 ∈ Qφ is the initial state ; pφ : Qφ × Qφ → [ 0 , 1 ] is a conditional probability defined as pφ ( qφ , j |qφ , i ) = { 1 ψqφ , i , qφ , j is true 0 otherwise . or pφ ( qφ , j |qφ , i , s ) = { 1 ρ ( s , ψqφ , i , qφ , j ) > 0 0 otherwise . ( 5 )

Summary: This paper mainly focuses on combining RL tasks with linear temporal logic formulas and proposed a method that helps to construct policy from learned subtasks. This method provides a structured solution for reusing learned skills (with scTLTL formulas), and can also help when new skills need to be involved in original tasks. The topic of the composition of skills is interesting. However, the joining of LTL and RL has been developed previously. The main contribution of this work is limited to the application of the previous techniques.