Year: 2018

Abstract:
Skills learned through reinforcement learning often generalize poorly across tasks and re-training is necessary when presented with a new task. We present a framework that combines techniques in formal methods with reinforcement learning (RL) that allows for the construction of new skills from existing ones with no additional exploration necessary. Our method also allows for convenient specification of complex temporal dependent tasks using logical expressions. We provide theoretical results for our composition technique and evaluate on a simple grid world simulation as well as a robotic manipulation task.

heading: 1 INTRODUCTION
text: Policies learned using reinforcement learning aim to maximize the given reward function and are often difficult to transfer to other problem domains. Skill composition is the process of constructing new skills out of existing ones (policies) with little to no additional learning. In stochastic optimal control, this idea has been adopted by Todorov (2009) and Da Silva et al. (2009) to construct provably optimal control laws based on linearly solvable Markov decision processes.
Temporal logic (TL) is a formal language commonly used in software and digital circuit verification (Baier & Katoen, 2008) as well as formal synthesis (Belta et al., 2017). It allows for convenient expression of complex behaviors and causal relationships. TL has been used by (Tabuada & Pappas, 2004), (Fainekos et al., 2006), (Fainekos et al., 2005) to synthesize provably correct control policies. Aksaray et al. (2016) have also combined TL with Q-learning to learn satisfiable policies in discrete state and action spaces.
We make the distinction between skill composition and multi-task learning/meta-learning where the latter often requires a predefined set of tasks/task distributions to learn and generalize from, whereas the focus of the former is to construct new policies from a library of already learned policies that achieve new tasks (often some combination of the constituent tasks) with little to no additional constraints on task distribution at learning time. In this work, we focus on skill composition with policies learned using automata guided reinforcement learning (Li et al., 2018). We adopt the syntactically co-safe truncated linear temporal logic (scTLTL) as the task specification language. Compared to most heuristic reward structures used in the RL literature, formal specification language has the advantage of semantic rigor and interpretability.
In our framework, skill composition is accomplished by taking the product of finite state automata (FSA). Instead of interpolating/extrapolating among learned skills/latent features, our method is based on graph manipulation of the FSA. Therefore, the outcome is much more transparent. Compared with previous work on skill composition, we impose no constraints on the policy representation or the problem class. We validate our framework in simulation (discrete state and action spaces) and experimentally on a Baxter robot (continuous state and action spaces).

heading: 2 RELATED WORK
text: Recent efforts in skill composition have mainly adopted the approach of combining value functions learned using different rewards. Peng et al. (2018) constructs a composite policy by combining the value functions of individual policies using the Boltzmann distribution. With a similar goal, Zhu et al. (2017) achieves task space transfer using deep successor representations (Kulkarni et al., 2016). However, it is required that the reward function be represented as a linear combination of state-action features.
Haarnoja et al. (2018) have showed that when using energy-based models (Haarnoja et al., 2017), an approximately optimal composite policy can result from taking the average of the Q-functions of existing policies. The resulting composite policy achieves the −AND− task composition i.e. the composite policy maximizes the average reward of individual tasks.
van Niekerk et al. (2018) have taken this idea a step further and showed that by combining individual Q-functions using the log-sum-exponential function, the −OR− task composition (the composite policy maximizes the (soft) maximum of the reward of constituent tasks) can be achieved optimally.
We build on the results of (van Niekerk et al., 2018) and show that incorporating temporal logic allows us to compose tasks of greater logical complexity with higher interpretability. Our composite policy is optimal in both −AND− and −OR− task compositions.

heading: 3 PRELIMINARIES

heading: 3.1 ENTROPY-REGULARIZED REINFORCEMENT LEARNING AND Q-COMPOSITION
text: We start with the definition of a Markov Decision Process. Definition 1. An MDP is defined as a tupleM = 〈S,A, p(·|·, ·), r(·, ·, ·)〉, where S ⊆ IRn is the state space ;A ⊆ IRm is the action space (S andA can also be discrete sets); p : S×A×S → [0, 1] is the transition function with p(s′|s, a) being the conditional probability density of taking action a ∈ A at state s ∈ S and ending up in state s′ ∈ S; r : S×A×S → IR is the reward function with r(s, a, s′) being the reward obtained by executing action a at state s and transitioning to s′.
In entropy-regularized reinforcement learning (Schulman et al., 2017), the goal is to maximize the following objective
J(π) = T−1∑ t=0 Eπ[rt + αH(π(·|st))], (1)
where π : S×A→ [0, 1] is a stochastic policy. Eπ is the expectation following π. H(π(·|st)) is the entropy of π. α is the temperature parameter. In the limit α→ 0, Equation (1) becomes the standard RL objective. The soft Q-learning algorithm introduced by (Haarnoja et al., 2017) optimizes the above objective and finds a policy represented by an energy-based model
π?(at|st) ∝ exp(−E(st, at)) (2)
where E(st, at) is an energy function that can be represented by a function approximator. Let rentt = rt + αH(π(·|st)), the state-action value function (Q-function) following π is defined as Qπ(s, a) = Eπ[ ∑T−1 t=0 r ent t |s0 = s, a0 = a]. Suppose we have a set of n tasks indexed by i, i ∈ {0, ..., n}, each task is defined by an MDP Mi that differs only in their reward function ri. Let Q π?i α be the optimal entropy-regularized Q-function. Authors of (van Niekerk et al., 2018) provide the following results
Theorem 1. Define vectors r = [r1, ..., rn], Qπ ? α = [Q π?1 α , ..., Q π?n α ]. Given a set of nonnegative weights w with ||w|| = 1, the optimal Q-function for a new task defined by r = α log(|| exp(r/α)||w) is given by
Qπ ? α = α log(|| exp(Qπ ? α /α)||w), (3)
where || · ||w is the weighted 1-norm.
The authors proceed to provide the following corollary Corollary 1. maxQπ ?
α ↑ Qπ ? 0 as α → 0, where Qπ ?
0 is the optimal Q-function for the objective J(π) = ∑T−1 t=0 Eπ[rt].
Corollary 1 states that in the low temperature limit, the maximum of the optimal entropy-regularized Q-functions approaches the standard optimal Q-function

heading: 3.2 SCTLTL AND FINITE STATE AUTOMATA
text: We consider tasks specified with syntactically co-safe Truncated Linear Temporal Logic (scTLTL) which is derived from truncated linear temporal logic(TLTL) (Li et al., 2018). The syntax of scTLTL is defined as
φ := > | f(s) < c | ¬φ | φ ∧ ψ | ♦φ | φU ψ | φ T ψ | © φ (4)
where > is the True Boolean constant. s ∈ S is a MDP state in Definition 1. f(s) < c is a predicate over the MDP states where c ∈ IR. ¬ (negation/not), ∧ (conjunction/and) are Boolean connectives. ♦ (eventually), U (until), T (then),© (next), are temporal operators.⇒ (implication) and and ∨ (disjunction/or) can be derived from the above operators. We denote st ∈ S to be the MDP state at time t, and st:t+k to be a sequence of states (state trajectory) from time t to t+ k, i.e., st:t+k = stst+1...st+k. The Boolean semantics of scTLTL is defined as:
st:t+k |= f(s) < c ⇔ f(st) < c, st:t+k |= ¬φ ⇔ ¬(st:t+k |= φ), st:t+k |= φ⇒ ψ ⇔ (st:t+k |= φ)⇒ (st:t+k |= ψ), st:t+k |= φ ∧ ψ ⇔ (st:t+k |= φ) ∧ (st:t+k |= ψ), st:t+k |= φ ∨ ψ ⇔ (st:t+k |= φ) ∨ (st:t+k |= ψ), st:t+k |=©φ ⇔ (st+1:t+k |= φ) ∧ (k > 0), st:t+k |= ♦φ ⇔ ∃t′ ∈ [t, t+ k) st′:t+k |= φ, st:t+k |= φ U ψ ⇔ ∃t′ ∈ [t, t+ k) s.t. st′:t+k |= ψ ∧ (∀t′′ ∈ [t, t′) st′′:t′ |= φ), st:t+k |= φ T ψ ⇔ ∃t′ ∈ [t, t+ k) s.t. st′:t+k |= ψ
∧ (∃t′′ ∈ [t, t′) st′′:t′ |= φ). A trajectory s0:T is said to satisfy formula φ if s0:T |= φ. The quantitative semantics (also referred to as robustness) is defined recursively as
ρ(st:t+k,>) = ρmax, ρ(st:t+k, f(st) < c) = c− f(st), ρ(st:t+k,¬φ) = − ρ(st:t+k, φ), ρ(st:t+k, φ ⇒ ψ) = max(−ρ(st:t+k, φ), ρ(st:t+k, ψ)) ρ(st:t+k, φ1 ∧ φ2) = min(ρ(st:t+k, φ1), ρ(st:t+k, φ2)), ρ(st:t+k, φ1 ∨ φ2) = max(ρ(st:t+k, φ1), ρ(st:t+k, φ2)), ρ(st:t+k,©φ) = ρ(st+1:t+k, φ) (k > 0), ρ(st:t+k,♦φ) = max
t′∈[t,t+k) (ρ(st′:t+k, φ)),
ρ(st:t+k, φ U ψ) = max t′∈[t,t+k) (min(ρ(st′:t+k, ψ),
min t′′∈[t,t′) ρ(st′′:t′ , φ))),
ρ(st:t+k, φ T ψ) = max t′∈[t,t+k) (min(ρ(st′:t+k, ψ),
max t′′∈[t,t′) ρ(st′′:t′ , φ))),
where ρmax represents the maximum robustness value. A robustness of greater than zero implies that st:t+k satisfies φ and vice versa (ρ(st:t+k, φ) > 0 ⇒ st:t+k |= φ and ρ(st:t+k, φ) < 0 ⇒ st:t+k 6|= φ). The robustness is used as a measure of the level of satisfaction of a trajectory s0:T with respect to a scTLTL formula φ.
Definition 2. An FSA corresponding to a scTLTL formula φ. is defined as a tuple Aφ = 〈Qφ,Ψφ, qφ,0, pφ(·|·),Fφ〉, where Qφ is a set of automaton states; Ψφ is the input alphabet (a set of first order logic formula); qφ,0 ∈ Qφ is the initial state; pφ : Qφ × Qφ → [0, 1] is a conditional probability defined as
pφ(qφ,j |qφ,i) = {
1 ψqφ,i,qφ,j is true 0 otherwise.
or pφ(qφ,j |qφ,i, s) = { 1 ρ(s, ψqφ,i,qφ,j ) > 0
0 otherwise.
(5)

heading: Fφ is a set of final automaton states.
text: Here qφ,i is the ith automaton state of Aφ. ψqφ,i,qφ,j ∈ Ψφ is the predicate guarding the transition from qφ,i to qφ,j . Because ψqφ,i,qφ,j is a predicate without temporal operators, the robustness ρ(st:t+k, ψqφ,i,qφ,j ) is only evaluated at st. Therefore, we use the shorthand ρ(st, ψqφ,i,qφ,j ) = ρ(st:t+k, ψqφ,i,qφ,j ). The translation from a TLTL formula to a FSA can be done automatically with available packages like Lomap (Vasile, 2017).

heading: 3.3 FSA AUGMENTED MDP
text: The FSA Augmented MDP is defined as follows
Definition 3. (Li et al., 2018) An FSA augmented MDP corresponding to scTLTL formula φ (constructed from FSA 〈Qφ,Ψφ, qφ,0, pφ(·|·),Fφ〉 and MDP 〈S,A, p(·|·, ·), r(·, ·, ·)〉) is defined as Mφ = 〈S̃, A, p̃(·|·, ·), r̃(·, ·),Fφ〉 where S̃ ⊆ S × Qφ, p̃(s̃′|s̃, a) is the probability of transitioning to s̃′ given s̃ and a,
p̃(s̃′|s̃, a) = p ( (s′, q′φ)|(s, qφ), a ) = { p(s′|s, a) pφ(qφ′|qφ, s) = 1 0 otherwise.
(6)
pφ is defined in Equation (5). r̃ : S̃ × S̃ → IR is the FSA augmented reward function, defined by
r̃(s̃, s̃′) = ρ(s′, D qφ φ ), (7)
where Dqφφ = ∨ q′φ∈Ωqφ
ψqφ,q′φ represents the disjunction of all predicates guarding the transitions that originate from qφ (Ωqφ is the set of automata states that are connected with q through outgoing edges).
A policy π?φ is said to satisfy φ if
π?φ = arg max πφ Eπφ [1(ρ(s0:T , φ) > 0)]. (8)
where 1(ρ(s0:T , φ) > 0) is an indicator function with value 1 if ρ(s0:T , φ) > 0 and 0 otherwise. As is mentioned in the original paper, there can be multiple policies that meet the requirement of Equation (8), therefore, a discount factor is used to find a maximally satisfying policy - one that leads to satisfaction in the least amount of time.
The FSA augmented MDPMφ establishes a connection between the TL specification and the standard reinforcement learning problem. A policy learned using Mφ has implicit knowledge of the FSA through the automaton state qφ ∈ Qφ. We will take advantage of this characteristic in our skill composition framework.

heading: 4 PROBLEM FORMULATION
text: Problem 1. Given two scTLTL formula φ1 and φ2 and their optimal Q-functions Q?φ1 and Q ? φ2 , obtain the optimal policy π?φ∧ that satisfies φ∧ = φ1 ∧ φ2 and π ? φ∨ that satisfies φ∨ = φ1 ∨ φ2.
Here Q?φ1 and Q ? φ2 can be the optimal Q-functions for the entropy-regularized MDP or the standard MDP. Problem 1 defines the problem of skill composition: given two policies each satisfying a scTLTL specification, construct the policy that satisfies the conjunction (−AND−)/disjunction (−OR−) of the given specifications. Solving this problem is useful when we want to break a complex task into simple and manageable components, learn a policy that satisfies each component and ”stitch” all the components together so that the original task is satisfied. It can also be the case that as the scope of the task grows with time, the original task specification is amended with new items. Instead of having to re-learn the task from scratch, we can learn only policies that satisfies the new items and combine them with the old policy.

heading: 5 AUTOMATA GUIDED SKILL COMPOSITION
text: In this section, we provide a solution for Problem 1 by constructing the FSA of φ∧ from that of φ1 and φ2 and using φ∧ to synthesize the policy for the combined skill. We start with the following definition.
Definition 4. Given Aφ1 = 〈Qφ1 ,Ψφ1 , qφ1,0, pφ1 ,Fφ1〉 and Aφ2 = 〈Qφ2 ,Ψφ2 , qφ2,0, pφ2 ,Fφ2〉 corresponding to formulas φ1 and φ2, the FSA of φ∧ = φ1∧φ2 is the product automaton ofAφ1 and Aφ1 , i.e. Aφ∧ = φ1 ∧ φ2 = Aφ1 ×Aφ2 = 〈Qφ∧ ,Ψφ∧ , qφ∧,0, pφ∧ ,Fφ∧〉 where Qφ∧ ⊆ Qφ1 ×Qφ2 is the set of product automaton states, qφ∧,0 = (qφ1,0, qφ2,0) is the product initial state, Fφ∧ ⊆ Fφ1 ∩Fφ2 are the final accepting states. Following Definition 2, for states qφ∧ = (qφ1 , qφ2) ∈ Qφ∧ and q′φ∧ = (q ′ φ1 , q′φ2) ∈ Qφ∧ , the transition probability pφ∧ is defined as
pφ∧(q ′ φ∧ |qφ∧) =
{ 1 pφ1(q ′ φ1 |qφ1)pφ2(q′φ2 |qφ2) = 1
0 otherwise. (9)
Example 1. Figure 1 illustrates the FSA of Aφ1and Aφ2 and their product automaton Aφ∧ . Here φ1 = ♦r∧♦g which entails that both r and g needs to be true at least once (order does not matter), and φ2 = ♦b. The resultant product corresponds to the formula φ = ♦r ∧ ♦g ∧ ♦b.
We provide the following theorem on automata guided skill composition
Theorem 2. Let Qπ ? φ = [Q π?1 φ1 , ..., Q π?n φn ] be a vector with entries Qπ ? i φi being the optimal Q-function for the FSA augmented MDPMφi . The optimal Q-function forMφ∧ where φ∧ = ∧ i φi is Q π?∧ φ∧ = max(Qπ ?
φ ).
Proof. For qφ∧ = (qφ1 , qφ2) ∈ Qφ∧ , let Ψqφ∧ , Ψqφ1 and Ψqφ2 denote the set of predicates guarding the edges originating from qφ∧ , qφ1 and qφ2 respectively. Equation (9) entails that a transition at qφ∧ in the product automaton Aφ∧ exists only if corresponding transitions at qφ1 , qφ2 exist in Aφ1and Aφ2 respectively. Therefore, ψqφ∧ ,q′φ∧ = ψqφ1 ,q′φ1 ∧ ψqφ2 ,q′φ2 , for ψqφ∧ ,q′φ∧ ∈ Ψqφ∧ , ψqφ1 ,q′φ1 ∈ Ψqφ1 , ψqφ2 ,q′φ2 ∈ Ψqφ2 (here q ′ φi is a state such that pφi(q ′ φi |qφi) = 1). Therefore, we have
D qφ∧ φ∧
= ∨
q′φ1 ,q′φ2
(ψqφ1 ,q′φ1 ∧ ψqφ2 ,q′φ2 ) (10)
where q′φ1 , q ′ φ2 don’t equal to qφ1 , qφ2 at the same time (to avoid self looping edges). Using the fact thatψqφi ,qφi = ¬ ∨ q′φi 6=qφi ψqφi ,q ′ φi and repeatedly applying the distributive laws (∆∧Ω1)∨(∆∧Ω2) = ∆ ∧ (Ω1 ∨ Ω2) and (∆ ∨ Ω1) ∧ (∆ ∨ Ω2) = ∆ ∨ (Ω1 ∧ Ω2) to D qφ∧ φ∧ , we arrive at
D qφ∧ φ∧ = ( ∨ q′φ1 6=qφ1 ψqφ1 ,q′φ1 ) ∨ ( ∨ q′φ2 6=qφ2 ψqφ2 ,q′φ2 ) = D qφ1 φ1 ∨Dqφ2φ2 . (11)
Let r̃φ∧ , r̃φ1 , r̃φ2 and s̃φ∧ , s̃φ1 , s̃φ2 be the reward functions and states for FSA augmented MDP Mφ∧ ,Mφ1 ,Mφ2 respectively. sφ∧ , sφ1 , sφ2 are the states for the corresponding MDPs. Plugging Equation (11) into Equation (7) and using the robustness definition for disjunction results in
r̃φ∧(s̃φ∧ , s̃ ′ φ∧) = ρ(s ′ φ∧ , D qφ∧ φ∧ )
= ρ(s′φ∧ , D qφ1 φ1 ∨Dqφ2φ2 ) = max ( ρ(s′φ1 , D qφ1 φ1 ), ρ(s′φ2 , D qφ2 φ2 ) )
= max ( r̃φ1(s̃φ1 , s̃ ′ φ1), r̃φ2(s̃φ2 , s̃ ′ φ2) ) .
(12)
Looking at Theorem 1, the log-sum-exp of the composite reward r = α log(|| exp(r/α)||w) is in fact an approximation of the maximum function. In the low temperature limit we have r → max(r) as α→ 0. Applying Corollary 1 results in Theorem 2.
Having obtained the optimal Q-function, a policy can be constructed by taking the greedy step with respective to the Q-function in the discrete action case. For the case of continuous action space where the policy is represented by a function approximator, the policy update procedure in actorcritic methods can be used to extract a policy from the Q-function.
In our framework, −AND− and −OR− task compositions follow the same procedure (Theorem 2). The only difference is the termination condition. For−AND− task, the final stateFφ∧ = ⋂ Fφi in Definition 4 needs to be reached (i.e. all the constituent FSAs are required to reach terminal state, as in state qφ∧,f in Figure 1 ). Whereas for the −OR− task, only Fφ∨ = ⋃ Fφi needs to be reached (one of states qφ∧,2, qφ∧,4, qφ∧,5, qφ∧,6, qφ∧,f in Figure 1 ). A summary of the composition procedure is provided in Algorithm 1.
In Algorithm 1, steps 3 and 4 seeks to obtain the optimal policy and Q-function using any off-policy actor critic RL algorithm. Bφ1,2 are the replay buffers collected while training for each skill. Step
Algorithm 1 Automata Guided Skill Composition 1: Inputs: scTLTL task specification φ1 and φ2, randomly initialized policies πφ1 , πφ2 and action-
value functions Qπφ1φ1 , Q πφ2 φ2
. State and action spaces of the MDP. 2: Construct FSA augmented MDPsMφ1 andMφ2 . using Definition 3 3: π?φ1 , Q π?φ1 φ1 ,Bφ1 ← ActorCritic(Mφ1) . learns the optimal policy and Q-function 4: π?φ2 , Q π?φ2 φ2 ,Bφ2 ← ActorCritic(Mφ2) 5: Q π?φ2∧φ2 φ2∧φ2 = max(Q π?φ1 φ1 , Q π?φ2 φ2
) . construct the optimal composed Q-function using Theorem 2 6: Bφ∧ ← ConstructProductBuffer(Bφ1 ,Bφ2) 7: π?φ2∧φ2 ← ExtractPolicy(Q π?φ2∧φ2 φ2∧φ2 ,Bφ∧) 8: return π?φ2∧φ2
6 constructs the product replay buffer for policy extraction. This step is necessary because each Bφi contains state of form (s, qi), i ∈ {1, 2} whereas the composed policy takes state (s, q1, q2) as input (as in Definition 4). Therefore, we transform each experience ((s, qi), a, (s′, q′i), r) to ((s, qi, qj 6=i), a, (s ′, q′i, q ′ j 6=i), r) where qj 6=i is chosen at random from the automaton states of Aφj and q′j 6=i is calculated from Equation (6). The reward r will not be used in policy extraction as the Q-function will not be updated. Step 7 extracts the optimal composed policy from the optimal composed Q-function (this corresponds to running only the policy update step in the actor critic algorithm).

heading: 6 CASE STUDIES
text: We evaluate the our composition method in two environments. The first is a simple 2D grid world environment that is used as for proof of concept and policy visualization. The second is a robot manipulation environment.

heading: 6.1 GRID WORLD
text: Consider an agent that navigates in a 8 × 10 grid world. Its MDP state space is S : X × Y where x, y ∈ X,Y are its integer coordinates on the grid. The action space is A : [up, down, left, right, stay]. The transition is such that for each action command, the agent follows that command with probability 0.8 or chooses a random action with probability 0.2. We train the agent on two tasks, φ1 = ♦r ∧ ♦g and φ2 = ♦b (same as in Example 1). The regions are defined by the predicates r = (1 < x < 3) ∧ (1 < y < 3) and g = (4 < x < 6) ∧ (4 < y < 6). Because the coordinates are integers, a and b define a point goal rather than regions. φ2 expresses a similar task for b = (1 < x < 3) ∧ (6 < y < 8). Figure 1 shows the FSA for each task. We apply standard tabular Q-learning Watkins (1989) on the FSA augmented MDP of this environment. For all experiments, we use a discount factor of 0.95, learning rate of 0.1, episode horizon of 200 steps, a random exploration policy and a total number of 2000 update steps which is enough to reach convergence (learning curve is not presented here as it is not the focus of this paper).
Figure 2 (a) and (b) show the learned optimal policies extracted by π?φi(x, y, qφi) = arg max
a Q?φi(x, y, qφi , a). We plot π ? φi (x, y, qφi) for each qφi and observe that each represents a
sub-policy whose goal is given by Equation 7.
Figure 2 (c) shows the composed policy of φ∧ = φ1 ∧ φ2 using Theorem 2. It can be observed that the composed policy is able to act optimally in terms maximizing the expected sum of discounted rewards given by Equation (12). Following the composed policy and transitioning the FSA in Figure 1 (b) will in fact satisfy φ∧ (−AND−). As discussed in the previous section, if the −OR− task is desired, following the same composed policy and terminate at any of the states qφ∧,2, qφ∧,4, qφ∧,5, qφ∧,6, qφ∧,f will satisfy φ∨ = φ1 ∨ φ2.

heading: 6.2 ROBOTIC MANIPULATION
text: In this sub-section, we test our method on a more complex manipulation task.

heading: 6.2.1 EXPERIMENT SETUP
text: Figure 4 (a) presents our experiment setup. Our policy controls the 7 degree-of-freedom joint velocities of the right arm of a Baxter robot. In front of the robot are three circular regions (red, green,
blue plates) and it has to learn to traverse in user specified ways. The positions of the plates are tracked by motion capture systems and thus fully observable. In addition, we also track the position of one of the user’s hands (by wearing a glove with trackers attached). Our MDP state space is 22 dimensional that includes 7 joint angles, xyz positions of the three regions (denoted by pr, pg , pb), the user’s hand (ph) and the robot’s end-effector (pee). State and action spaces are continuous in this case.
We define the following predicates
1. ψi = ||pi − pee|| < , i ∈ {r, g, b, h} where is a threshold which we set to be 5 centimeters. ψi constrains the relative distance between the robot’s end-effector and the selected object.
2. ψhand in sight = (xmin < phx < xmax) ∧ (ymin < phy < ymax) ∧ (zmin < phz < zmax). This predicate evaluates to true if the user’s hand appears in the cubic region defined by [xmin, xmax, ymin, ymax, zmin, zmax]. In this experiment, we take this region to be 40 centimeters above the table (length and width the same as the table).
We test our method on the following composition task
1. φtraverse = ♦(ψr ∧ ♦(ψg ∧ ♦ψb)) Description: traverse the three regions in the order of red, green, blue.
2. φinterrupt = (ψhand in sight ⇒ ♦ψh) U ψb Description: before reaching the blue region, if the user’s hand appears in sight, then eventually reach for the user’s hand and the blue region, otherwise just reach for the blue region.
3. φ∧ = φtraverse ∧ φinterrupt Description: conjunction of the first two tasks.
The FSAs for φ1 and φ2 are presented in Figure 3 . The FSA for φ∧ (14 nodes and 72 edges) is not presented here due to space constraints.

heading: 6.2.2 IMPLEMENTATION DETAILS
text: Our policy and Q-function are represented by a neural network (3 layers each with 300, 200, 100 ReLU units). For tasks φtraverse and φinterrupt, the input state space is 23 dimensional (22 continuous dimensional MDP state and 1 discrete dimension for the automaton state). For task φ∧, the state space is 24 dimensional (2 discrete dimensions for automaton states of φtraverse and φinterrupt). We use soft Q-learning (SQL) (Haarnoja et al., 2017) to learn the optimal policies and Q-functions for φtraverse and φinterrupt and Theorem 2 to compose the Q-functions. The policy for task φ∧ is extracted from the composite Q-function by running the same algorithm on already collected experience without updating the Q-function (similar procedure as (Haarnoja et al., 2018)). After each
episode, the joint angles, the FSA state, the position of the plates as well as the position of the hand (represented by the yellow sphere in Figure 4 (a)) are randomly reset (within certain boundaries) to ensure generalization across different task configurations. The robot is controlled at 20 Hz. Each episode is 100 time-steps (about 5 seconds). The episode restarts if the final automaton state qf is reached. During training, we perform 100 policy and Q-function updates every 5 episodes of exploration. All of our training is performed in simulation and for this set of tasks, the policy is able to transfer to the real robot without further fine-tuning.

heading: 6.2.3 RESULTS AND DISCUSSION
text: In Figure 5 (left), we report the discounted return as a function of policy update steps for task φ∧. 5 evaluation episodes are collected after each set policy updates to calculate the performance statistics. As comparison, we learn the same task using SQL with FSA augmented MDP. We can see that our composition method takes less update steps to reach a policy that achieves higher returns with lower variance than the policy obtained from learning. Figure 5 (right) shows the episode length as a function of policy update (upper bound clipped at 100 steps). As mentioned in the previous section, a shorter episode length indicates faster accomplishment of the task. It can be observed that both the composition and learning method result in high variances likely due to the randomized task configuration (some plate/joint/hand configurations make the task easier to accomplish than others). However, the policy obtained from composition achieves a noticeable decrease in the average episode length.
It is important to note that the wall time for learning a policy is significantly longer than that from composition. For robotic tasks with relatively simple policy representations (feed-forward neural networks), learning time is dominated by the time used to collect experiences and the average episode length (recall that we update the policy 100 times with each 5 episodes of exploration). Since skill composition uses already collected experience, obtaining a policy can be much faster. Table 1 shows the mean training time and standard deviation (over 5 random seeds) for each task (tasks φtraverse, φinterrupt and φ∧(learned) are trained for 80K policy updates. φ∧(composed) is trained for 40K policy updates). In general, training time is shorter for tasks with higher episodic success rate and shorter episode length. We also show the task success rate evaluated on the real robot over 20 evaluation trials. Task success is evaluated by calculating the robustness of the trajectories resulting from executing each policy. A robustness of greater than 0 evaluates to success and vice versa. πφ∧ (learned) fails to complete the task even though a convergence is reached during training. This is likely due to the large FSA of φ∧ with complex per-step reward (D q φ in Equation (7)) which makes learning difficult. Figure 4 (b) shows an evaluation run of the composed policy for task φ∧.

heading: 7 CONCLUSION
text: We provide a technique that takes advantage of the product of finite state automata to perform deterministic skill composition. Our method is able to synthesize optimal composite policies for −AND− and −OR− tasks. We provide theoretical results on our method and show its effectiveness on a grid world simulation and a real world robotic task. For future work, we will adapt our
method to the more general case of task-space transfer - given a library of optimal policies (Qfunctions) that each satisfies its own specification, construct a policy that satisfies a specification that’s an arbitrary (temporal) logical combination of the constituent specifications.

Review summaries:
This paper mainly focuses on combining RL tasks with linear temporal logic formulas and proposed a method that helps to construct policy from learned subtasks. This method provides a structured solution for reusing learned skills (with scTLTL formulas), and can also help when new skills need to be involved in original tasks. The topic of the composition of skills is interesting. However, the joining of LTL and RL has been developed previously. The main contribution of this work is limited to the application of the previous techniques.