Title: AUTOMATA GUIDED SKILL COMPOSITION

Abstract: Skills learned through reinforcement learning often generalize poorly across tasks and re-training is necessary when presented with a new task. We present a framework that combines techniques in formal methods with reinforcement learning (RL) that allows for the construction of new skills from existing ones with no additional exploration necessary. Our method also allows for convenient specification of complex temporal dependent tasks using logical expressions. We provide theoretical results for our composition technique and evaluate on a simple grid world simulation as well as a robotic manipulation task.

1. INTRODUCTION:

Policies learned using reinforcement learning aim to maximize the given reward function and are often difficult to transfer to other problem domains. Skill composition is the process of constructing new skills out of existing ones (policies) with little to no additional learning. In stochastic optimal control, this idea has been adopted by Todorov (2009) and Da Silva et al. (2009) to construct provably optimal control laws based on linearly solvable Markov decision processes.

Temporal logic (TL) is a formal language commonly used in software and digital circuit verification (Baier & Katoen, 2008) as well as formal synthesis (Belta et al., 2017) . It allows for convenient expression of complex behaviors and causal relationships. TL has been used by (Tabuada & Pappas, 2004) , (Fainekos et al., 2006) , (Fainekos et al., 2005) to synthesize provably correct control policies. Aksaray et al. (2016) have also combined TL with Q-learning to learn satisfiable policies in discrete state and action spaces.

We make the distinction between skill composition and multi-task learning/meta-learning where the latter often requires a predefined set of tasks/task distributions to learn and generalize from, whereas the focus of the former is to construct new policies from a library of already learned policies that achieve new tasks (often some combination of the constituent tasks) with little to no additional constraints on task distribution at learning time. In this work, we focus on skill composition with policies learned using automata guided reinforcement learning (Li et al., 2018) . We adopt the syntactically co-safe truncated linear temporal logic (scTLTL) as the task specification language. Compared to most heuristic reward structures used in the RL literature, formal specification language has the advantage of semantic rigor and interpretability.

In our framework, skill composition is accomplished by taking the product of finite state automata (FSA). Instead of interpolating/extrapolating among learned skills/latent features, our method is based on graph manipulation of the FSA. Therefore, the outcome is much more transparent. Compared with previous work on skill composition, we impose no constraints on the policy representation or the problem class. We validate our framework in simulation (discrete state and action spaces) and experimentally on a Baxter robot (continuous state and action spaces).

2. RELATED WORK:

Recent efforts in skill composition have mainly adopted the approach of combining value functions learned using different rewards. Peng et al. (2018) constructs a composite policy by combining the value functions of individual policies using the Boltzmann distribution. With a similar goal, Zhu et al. (2017) achieves task space transfer using deep successor representations (Kulkarni et al., 2016) . However, it is required that the reward function be represented as a linear combination of state-action features. Haarnoja et al. (2018) have showed that when using energy-based models (Haarnoja et al., 2017) , an approximately optimal composite policy can result from taking the average of the Q-functions of existing policies. The resulting composite policy achieves the -AN Dtask composition i.e. the composite policy maximizes the average reward of individual tasks. van Niekerk et al. (2018) have taken this idea a step further and showed that by combining individual Q-functions using the log-sum-exponential function, the -ORtask composition (the composite policy maximizes the (soft) maximum of the reward of constituent tasks) can be achieved optimally.

We build on the results of (van Niekerk et al., 2018) and show that incorporating temporal logic allows us to compose tasks of greater logical complexity with higher interpretability. Our composite policy is optimal in both -AN Dand -ORtask compositions.

3. PRELIMINARIES:



3.1. ENTROPY-REGULARIZED REINFORCEMENT LEARNING AND Q-COMPOSITION:

We start with the definition of a Markov Decision Process. Definition 1. An MDP is defined as a tuple M = S, A, p(•|•, •), r(•, •, •) , where S ⊆ IR n is the state space ; A ⊆ IR m is the action space (S and A can also be discrete sets); p : S ×A×S → [0, 1] is the transition function with p(s |s, a) being the conditional probability density of taking action a ∈ A at state s ∈ S and ending up in state s ∈ S; r : S × A × S → IR is the reward function with r(s, a, s ) being the reward obtained by executing action a at state s and transitioning to s .

In entropy-regularized reinforcement learning (Schulman et al., 2017) , the goal is to maximize the following objective

EQUATION

where π : S × A → [0, 1] is a stochastic policy. E π is the expectation following π. H(π(•|s t )) is the entropy of π. α is the temperature parameter. In the limit α → 0, Equation (1) becomes the standard RL objective. The soft Q-learning algorithm introduced by (Haarnoja et al., 2017) 

Q π (s, a) = E π [ T -1 t=0 r ent t |s 0 = s, a 0 = a]

. Suppose we have a set of n tasks indexed by i, i ∈ {0, ..., n}, each task is defined by an MDP M i that differs only in their reward function r i . Let Q π i α be the optimal entropy-regularized Q-function. Authors of (van Niekerk et al., 2018) provide the following results

Theorem 1. Define vectors r = [r 1 , ..., r n ], Q π α = [Q π 1 α , ..., Q π n α ].

Given a set of nonnegative weights w with ||w|| = 1, the optimal Q-function for a new task defined by r = α log(|| exp(r/α)|| w ) is given by

EQUATION

where || • || w is the weighted 1-norm.

The authors proceed to provide the following corollary Corollary 1. max Q π α ↑ Q π 0 as α → 0, where Q π 0 is the optimal Q-function for the objective

J(π) = T -1 t=0 E π [r t ].

Corollary 1 states that in the low temperature limit, the maximum of the optimal entropy-regularized Q-functions approaches the standard optimal Q-function

3.2. SCTLTL AND FINITE STATE AUTOMATA:

We consider tasks specified with syntactically co-safe Truncated Linear Temporal Logic (scTLTL) which is derived from truncated linear temporal logic(TLTL) (Li et al., 2018) . The syntax of scTLTL is defined as

φ := | f (s) < c | ¬φ | φ ∧ ψ | ♦φ | φ U ψ | φ T ψ | φ (4)

where is the True Boolean constant. s ∈ S is a MDP state in Definition 1. f (s) < c is a predicate over the MDP states where c ∈ IR. ¬ (negation/not), ∧ (conjunction/and) are Boolean connectives. ♦ (eventually), U (until), T (then), (next), are temporal operators.⇒ (implication) and and ∨ (disjunction/or) can be derived from the above operators.

We denote s t ∈ S to be the MDP state at time t, and s t:t+k to be a sequence of states (state trajectory) from time t to t + k, i.e., s t:t+k = s t s t+1 ...s t+k . The Boolean semantics of scTLTL is defined as:

s t:t+k |= f (s) < c ⇔ f (s t ) < c, s t:t+k |= ¬φ ⇔ ¬(s t:t+k |= φ), s t:t+k |= φ ⇒ ψ ⇔ (s t:t+k |= φ) ⇒ (s t:t+k |= ψ), s t:t+k |= φ ∧ ψ ⇔ (s t:t+k |= φ) ∧ (s t:t+k |= ψ), s t:t+k |= φ ∨ ψ ⇔ (s t:t+k |= φ) ∨ (s t:t+k |= ψ), s t:t+k |= φ ⇔ (s t+1:t+k |= φ) ∧ (k > 0), s t:t+k |= ♦φ ⇔ ∃t ∈ [t, t + k) s t :t+k |= φ, s t:t+k |= φ U ψ ⇔ ∃t ∈ [t, t + k) s.t. s t :t+k |= ψ ∧ (∀t ∈ [t, t ) s t :t |= φ), s t:t+k |= φ T ψ ⇔ ∃t ∈ [t, t + k) s.t. s t :t+k |= ψ ∧ (∃t ∈ [t, t ) s t :t |= φ).

A trajectory s 0:T is said to satisfy formula φ if s 0:T |= φ.

The quantitative semantics (also referred to as robustness) is defined recursively as

ρ(s t:t+k , ) = ρ max , ρ(s t:t+k , f (s t ) < c) = c -f (s t ), ρ(s t:t+k , ¬φ) = -ρ(s t:t+k , φ), ρ(s t:t+k , φ ⇒ ψ) = max(-ρ(s t:t+k , φ), ρ(s t:t+k , ψ)) ρ(s t:t+k , φ 1 ∧ φ 2 ) = min(ρ(s t:t+k , φ 1 ), ρ(s t:t+k , φ 2 )), ρ(s t:t+k , φ 1 ∨ φ 2 ) = max(ρ(s t:t+k , φ 1 ), ρ(s t:t+k , φ 2 )), ρ(s t:t+k , φ) = ρ(s t+1:t+k , φ) (k > 0), ρ(s t:t+k , ♦φ) = max t ∈[t,t+k) (ρ(s t :t+k , φ)), ρ(s t:t+k , φ U ψ) = max t ∈[t,t+k) (min(ρ(s t :t+k , ψ), min t ∈[t,t ) ρ(s t :t , φ))), ρ(s t:t+k , φ T ψ) = max t ∈[t,t+k) (min(ρ(s t :t+k , ψ), max t ∈[t,t ) ρ(s t :t , φ))),

where ρ max represents the maximum robustness value. A robustness of greater than zero implies that s t:t+k satisfies φ and vice versa (ρ(s t:t+k , φ) > 0 ⇒ s t:t+k |= φ and ρ(s t:t+k , φ) < 0 ⇒ s t:t+k |= φ). The robustness is used as a measure of the level of satisfaction of a trajectory s 0:T with respect to a scTLTL formula φ.

Definition 2. An FSA corresponding to a scTLTL formula φ. is defined as a tuple

EQUATION

)

F φ is a set of final automaton states.

Here q φ,i is the i th automaton state of A φ . ψ q φ,i ,q φ,j ∈ Ψ φ is the predicate guarding the transition from q φ,i to q φ,j . Because ψ q φ,i ,q φ,j is a predicate without temporal operators, the robustness ρ(s t:t+k , ψ q φ,i ,q φ,j ) is only evaluated at s t . Therefore, we use the shorthand ρ(s t , ψ q φ,i ,q φ,j ) = ρ(s t:t+k , ψ q φ,i ,q φ,j ). The translation from a TLTL formula to a FSA can be done automatically with available packages like Lomap (Vasile, 2017) .

3.3. FSA AUGMENTED MDP:

The FSA Augmented MDP is defined as follows Definition 3. (Li et al., 2018) An FSA augmented MDP corresponding to scTLTL formula φ

(constructed from FSA Q φ , Ψ φ , q φ,0 , p φ (•|•), F φ and MDP S, A, p(•|•, •), r(•, •, •) ) is defined as M φ = S, A, p(•|•, •), r(•, •), F φ where S ⊆ S × Q φ , p(s |s, a)

is the probability of transitioning to s given s and a,

EQUATION

)

p φ is defined in Equation ( 5). r : S × S → IR is the FSA augmented reward function, defined by r(s, s ) = ρ(s , D

EQUATION

where D q φ φ = q φ ∈Ωq φ ψ q φ ,q φ represents the disjunction of all predicates guarding the transitions that originate from q φ (Ω q φ is the set of automata states that are connected with q through outgoing edges).

A policy π φ is said to satisfy φ if

EQUATION

)

where 1(ρ(s 0:T , φ) > 0) is an indicator function with value 1 if ρ(s 0:T , φ) > 0 and 0 otherwise. As is mentioned in the original paper, there can be multiple policies that meet the requirement of Equation ( 8), therefore, a discount factor is used to find a maximally satisfying policy -one that leads to satisfaction in the least amount of time.

The FSA augmented MDP M φ establishes a connection between the TL specification and the standard reinforcement learning problem. A policy learned using M φ has implicit knowledge of the FSA through the automaton state q φ ∈ Q φ . We will take advantage of this characteristic in our skill composition framework. 

4. PROBLEM FORMULATION:

Problem 1. Given two scTLTL formula φ 1 and φ 2 and their optimal Q-functions Q φ1 and Q φ2 , obtain the optimal policy π φ∧ that satisfies φ ∧ = φ 1 ∧ φ 2 and π φ∨ that satisfies φ ∨ = φ 1 ∨ φ 2 .

Here Q φ1 and Q φ2 can be the optimal Q-functions for the entropy-regularized MDP or the standard MDP. Problem 1 defines the problem of skill composition: given two policies each satisfying a scTLTL specification, construct the policy that satisfies the conjunction (-AN D-)/disjunction (-OR-) of the given specifications. Solving this problem is useful when we want to break a complex task into simple and manageable components, learn a policy that satisfies each component and "stitch" all the components together so that the original task is satisfied. It can also be the case that as the scope of the task grows with time, the original task specification is amended with new items. Instead of having to re-learn the task from scratch, we can learn only policies that satisfies the new items and combine them with the old policy.

5. AUTOMATA GUIDED SKILL COMPOSITION:

In this section, we provide a solution for Problem 1 by constructing the FSA of φ ∧ from that of φ 1 and φ 2 and using φ ∧ to synthesize the policy for the combined skill. We start with the following definition.

Definition 4. Given A φ1 = Q φ1 , Ψ φ1 , q φ1,0 , p φ1 , F φ1 and A φ2 = Q φ2 , Ψ φ2 , q φ2,0 , p φ2 , F φ2 corresponding to formulas φ 1 and φ 2 , the FSA of φ ∧ = φ 1 ∧φ 2 is the product automaton of A φ1 and A φ1 , i.e.

A φ∧ = φ 1 ∧ φ 2 = A φ1 × A φ2 = Q φ∧ , Ψ φ∧ , q φ∧,0 , p φ∧ , F φ∧ where Q φ∧ ⊆ Q φ1 × Q φ2

is the set of product automaton states, q φ∧,0 = (q φ1,0 , q φ2,0 ) is the product initial state, F φ∧ ⊆ F φ1 ∩ F φ2 are the final accepting states. Following Definition 2, for states q φ∧ = (q φ1 , q φ2 ) ∈ Q φ∧ and q φ∧ = (q φ1 , q φ2 ) ∈ Q φ∧ , the transition probability p φ∧ is defined as

EQUATION

)

Example 1. Figure 1 illustrates the FSA of A φ1 and A φ2 and their product automaton A φ∧ . Here φ 1 = ♦r ∧ ♦g which entails that both r and g needs to be true at least once (order does not matter), and φ 2 = ♦b. The resultant product corresponds to the formula φ = ♦r ∧ ♦g ∧ ♦b.

We provide the following theorem on automata guided skill composition

Theorem 2. Let Q π φ = [Q π 1 φ1 , ..., Q π n

φn ] be a vector with entries Q

π i

φi being the optimal Q-function for the FSA augmented MDP M φi . The optimal Q-function for M φ∧ where φ

∧ = i φ i is Q π ∧ φ∧ = max(Q π φ ).

Proof. For q φ∧ = (q φ1 , q φ2 ) ∈ Q φ∧ , let Ψ q φ ∧ , Ψ q φ 1 and Ψ q φ 2 denote the set of predicates guarding the edges originating from q φ∧ , q φ1 and q φ2 respectively. Equation ( 9) entails that a transition at q φ∧ in the product automaton A φ∧ exists only if corresponding transitions at q φ1 , q φ2 exist in A φ1 and A φ2 respectively. Therefore, ψ q φ ∧ ,q φ ∧ = ψ q φ 1 ,q φ 1 ∧ ψ q φ 2 ,q φ 2 , for ψ q φ ∧ ,q φ ∧ ∈ Ψ q φ ∧ , ψ q φ 1 ,q φ 1 ∈ Ψ q φ 1 , ψ q φ 2 ,q φ 2 ∈ Ψ q φ 2 (here q φi is a state such that p φi (q φi |q φi ) = 1). Therefore, we have

EQUATION

)

where q φ1 , q φ2 don't equal to q φ1 , q φ2 at the same time (to avoid self looping edges). Using the fact that ψ q φ i ,q φ i = ¬ q φ i =q φ i ψ q φ i ,q φ i and repeatedly applying the distributive laws

EQUATION

Let rφ∧ , rφ1 , rφ2 and sφ∧ , sφ1 , sφ2 be the reward functions and states for FSA augmented MDP M φ∧ , M φ1 , M φ2 respectively. s φ∧ , s φ1 , s φ2 are the states for the corresponding MDPs. Plugging Equation (11) into Equation ( 7) and using the robustness definition for disjunction results in rφ∧ (s φ∧ , s φ∧ ) = ρ(s φ∧ , D

q φ ∧ φ∧ ) = ρ(s φ∧ , D q φ 1 φ1 ∨ D q φ 2 φ2 ) = max ρ(s φ1 , D q φ 1 φ1 ), ρ(s φ2 , D q φ 2 φ2 ) = max rφ1 (s φ1 , s φ1 ), rφ2 (s φ2 , s φ2 ) . (12)

Looking at Theorem 1, the log-sum-exp of the composite reward r = α log(|| exp(r/α)|| w ) is in fact an approximation of the maximum function. In the low temperature limit we have r → max(r) as α → 0. Applying Corollary 1 results in Theorem 2.

Having obtained the optimal Q-function, a policy can be constructed by taking the greedy step with respective to the Q-function in the discrete action case. For the case of continuous action space where the policy is represented by a function approximator, the policy update procedure in actorcritic methods can be used to extract a policy from the Q-function.

In our framework, -AN Dand -ORtask compositions follow the same procedure (Theorem 2). The only difference is the termination condition. For -AN Dtask, the final state F φ∧ = F φi in Definition 4 needs to be reached (i.e. all the constituent FSAs are required to reach terminal state, as in state q φ∧,f in Figure 1 ). Whereas for the -ORtask, only F φ∨ = F φi needs to be reached (one of states q φ∧,2 , q φ∧,4 , q φ∧,5 , q φ∧,6 , q φ∧,f in Figure 1 ). A summary of the composition procedure is provided in Algorithm 1.

In Algorithm 1, steps 3 and 4 seeks to obtain the optimal policy and Q-function using any off-policy actor critic RL algorithm. B φ1,2 are the replay buffers collected while training for each skill.

Step Algorithm 1 Automata Guided Skill Composition 1: Inputs: scTLTL task specification φ 1 and φ 2 , randomly initialized policies π φ1 , π φ2 and actionvalue functions Q

π φ 1 φ1 , Q π φ 2

φ2 . State and action spaces of the MDP. 2: Construct FSA augmented MDPs M φ1 and M φ2 using Definition 3

3: π φ1 , Q π φ 1 φ1 , B φ1 ← ActorCritic(M φ1 )

learns the optimal policy and Q-function

4: π φ2 , Q π φ 2 φ2 , B φ2 ← ActorCritic(M φ2 ) 5: Q π φ 2 ∧φ 2 φ2∧φ2 = max(Q π φ 1 φ1 , Q π φ 2 φ2

) construct the optimal composed Q-function using Theorem 2 6: B φ∧ ← ConstructP roductBuf f er(B φ1 , B φ2 )

7: π φ2∧φ2 ← ExtractP olicy(Q π φ 2 ∧φ 2 φ2∧φ2 , B φ∧ ) 8: return π φ2∧φ2

6 constructs the product replay buffer for policy extraction. This step is necessary because each B φi contains state of form (s, q i ), i ∈ {1, 2} whereas the composed policy takes state (s, q 1 , q 2 ) as input (as in Definition 4). Therefore, we transform each experience ((s, q i ), a, (s , q i ), r) to ((s, q i , q j =i ), a, (s , q i , q j =i ), r) where q j =i is chosen at random from the automaton states of A φj and q j =i is calculated from Equation ( 6). The reward r will not be used in policy extraction as the Q-function will not be updated.

Step 7 extracts the optimal composed policy from the optimal composed Q-function (this corresponds to running only the policy update step in the actor critic algorithm).

6. CASE STUDIES:

We evaluate the our composition method in two environments. The first is a simple 2D grid world environment that is used as for proof of concept and policy visualization. The second is a robot manipulation environment.

6.1. GRID WORLD:

Consider an agent that navigates in a 8 × 10 grid world. Its MDP state space is S : X × Y where x, y ∈ X, Y are its integer coordinates on the grid. The action space is A : [up, down, left, right, stay] . The transition is such that for each action command, the agent follows that command with probability 0.8 or chooses a random action with probability 0.2. We train the agent on two tasks, φ 1 = ♦r ∧ ♦g and φ 2 = ♦b (same as in Example 1). The regions are defined by the predicates r = (1 < x < 3) ∧ (1 < y < 3) and g = (4 < x < 6) ∧ (4 < y < 6). Because the coordinates are integers, a and b define a point goal rather than regions. φ 2 expresses a similar task for b = (1 < x < 3) ∧ (6 < y < 8). Figure 1 shows the FSA for each task.

We apply standard tabular Q-learning Watkins (1989) on the FSA augmented MDP of this environment. For all experiments, we use a discount factor of 0.95, learning rate of 0.1, episode horizon of 200 steps, a random exploration policy and a total number of 2000 update steps which is enough to reach convergence (learning curve is not presented here as it is not the focus of this paper).

Figure 2 (a) and (b) show the learned optimal policies extracted by π φi (x, y, q φi ) = arg max a Q φi (x, y, q φi , a). We plot π φi (x, y, q φi ) for each q φi and observe that each represents a sub-policy whose goal is given by Equation 7.

Figure 2 (c) shows the composed policy of φ ∧ = φ 1 ∧ φ 2 using Theorem 2. It can be observed that the composed policy is able to act optimally in terms maximizing the expected sum of discounted rewards given by Equation ( 12). Following the composed policy and transitioning the FSA in Figure 1 (b) will in fact satisfy φ ∧ (-AN D-). As discussed in the previous section, if the -ORtask is desired, following the same composed policy and terminate at any of the states q φ∧,2 , q φ∧,4 , q φ∧,5 , q φ∧,6 , q φ∧,f will satisfy φ ∨ = φ 1 ∨ φ 2 . 

6.2. ROBOTIC MANIPULATION:

In this sub-section, we test our method on a more complex manipulation task.

6.2.1. EXPERIMENT SETUP:

Figure 4 (a) presents our experiment setup. Our policy controls the 7 degree-of-freedom joint velocities of the right arm of a Baxter robot. In front of the robot are three circular regions (red, green, blue plates) and it has to learn to traverse in user specified ways. The positions of the plates are tracked by motion capture systems and thus fully observable. In addition, we also track the position of one of the user's hands (by wearing a glove with trackers attached). Our MDP state space is 22 dimensional that includes 7 joint angles, xyz positions of the three regions (denoted by p r , p g , p b ), the user's hand (p h ) and the robot's end-effector (p ee ). State and action spaces are continuous in this case.

We define the following predicates 1. ψ i = ||p ip ee || < , i ∈ {r, g, b, h} where is a threshold which we set to be 5 centimeters. ψ i constrains the relative distance between the robot's end-effector and the selected object. 2. ψ hand in sight = (x min < p h

x < x max ) ∧ (y min < p h y < y max ) ∧ (z min < p h z < z max ). This predicate evaluates to true if the user's hand appears in the cubic region defined by [x min , x max , y min , y max , z min , z max ]. In this experiment, we take this region to be 40 centimeters above the table (length and width the same as the table ).

We test our method on the following composition task 1. φ traverse = ♦(ψ r ∧ ♦(ψ g ∧ ♦ψ b ))

Description: traverse the three regions in the order of red, green, blue. 2. φ interrupt = (ψ hand in sight ⇒ ♦ψ h ) U ψ b Description: before reaching the blue region, if the user's hand appears in sight, then eventually reach for the user's hand and the blue region, otherwise just reach for the blue region. 3. φ ∧ = φ traverse ∧ φ interrupt Description: conjunction of the first two tasks.

The FSAs for φ 1 and φ 2 are presented in Figure 3 . The FSA for φ ∧ (14 nodes and 72 edges) is not presented here due to space constraints.

6.2.2. IMPLEMENTATION DETAILS:

Our policy and Q-function are represented by a neural network (3 layers each with 300, 200, 100 ReLU units). For tasks φ traverse and φ interrupt , the input state space is 23 dimensional (22 continuous dimensional MDP state and 1 discrete dimension for the automaton state). For task φ ∧ , the state space is 24 dimensional (2 discrete dimensions for automaton states of φ traverse and φ interrupt ). We use soft Q-learning (SQL) (Haarnoja et al., 2017) to learn the optimal policies and Q-functions for φ traverse and φ interrupt and Theorem 2 to compose the Q-functions. The policy for task φ ∧ is extracted from the composite Q-function by running the same algorithm on already collected experience without updating the Q-function (similar procedure as (Haarnoja et al., 2018) ). After each episode, the joint angles, the FSA state, the position of the plates as well as the position of the hand (represented by the yellow sphere in Figure 4 (a)) are randomly reset (within certain boundaries) to ensure generalization across different task configurations. The robot is controlled at 20 Hz. Each episode is 100 time-steps (about 5 seconds). The episode restarts if the final automaton state q f is reached. During training, we perform 100 policy and Q-function updates every 5 episodes of exploration. All of our training is performed in simulation and for this set of tasks, the policy is able to transfer to the real robot without further fine-tuning.

6.2.3. RESULTS AND DISCUSSION:

In Figure 5 (left), we report the discounted return as a function of policy update steps for task φ ∧ . 5 evaluation episodes are collected after each set policy updates to calculate the performance statistics. As comparison, we learn the same task using SQL with FSA augmented MDP. We can see that our composition method takes less update steps to reach a policy that achieves higher returns with lower variance than the policy obtained from learning. Figure 5 (right) shows the episode length as a function of policy update (upper bound clipped at 100 steps). As mentioned in the previous section, a shorter episode length indicates faster accomplishment of the task. It can be observed that both the composition and learning method result in high variances likely due to the randomized task configuration (some plate/joint/hand configurations make the task easier to accomplish than others). However, the policy obtained from composition achieves a noticeable decrease in the average episode length.

It is important to note that the wall time for learning a policy is significantly longer than that from composition. For robotic tasks with relatively simple policy representations (feed-forward neural networks), learning time is dominated by the time used to collect experiences and the average episode length (recall that we update the policy 100 times with each 5 episodes of exploration). Since skill composition uses already collected experience, obtaining a policy can be much faster. Table 1 shows the mean training time and standard deviation (over 5 random seeds) for each task (tasks φ traverse , φ interrupt and φ ∧ (learned) are trained for 80K policy updates. φ ∧ (composed) is trained for 40K policy updates). In general, training time is shorter for tasks with higher episodic success rate and shorter episode length. We also show the task success rate evaluated on the real robot over 20 evaluation trials. Task success is evaluated by calculating the robustness of the trajectories resulting from executing each policy. A robustness of greater than 0 evaluates to success and vice versa. π φ∧ (learned) fails to complete the task even though a convergence is reached during training. This is likely due to the large FSA of φ ∧ with complex per-step reward (D q φ in Equation ( 7)) which makes learning difficult. Figure 4 (b) shows an evaluation run of the composed policy for task φ ∧ .

7. CONCLUSION:

We provide a technique that takes advantage of the product of finite state automata to perform deterministic skill composition. Our method is able to synthesize optimal composite policies for -AN Dand -ORtasks. We provide theoretical results on our method and show its effectiveness on a grid world simulation and a real world robotic task. For future work, we will adapt our method to the more general case of task-space transfer -given a library of optimal policies (Qfunctions) that each satisfies its own specification, construct a policy that satisfies a specification that's an arbitrary (temporal) logical combination of the constituent specifications.

:



Figures and Tables:

Figure 1 : FSA for (a) φ1 = ♦a ∧ ♦b. (b) φ2 = ♦c. (c) φ∧ = φ1 ∧ φ2.

Figure 2 : Policies for (a) φ1 = ♦a ∧ ♦b. (b) φ2 = ♦c. (c) φ∧ = φ1 ∧ φ2. The agent moves in a 8 × 10 gridworld with 3 labeled regions. The agent has actions [up, down, left, right, stay] where the directional actions are represented by arrows, stay is represented by the blue dot.

Figure 3 : The FSA for (a):φtraverse = ♦(ψr ∧ ♦(ψg ∧ ♦ψ b )) b φinterrupt = (ψ hand in sight ⇒ ♦ψ h ) U ψ b . The subscripts φ1 and φ2 are dropped for clarity.

Figure 4 : (a): The upper figure shows our simulation environment in V-REP Rohmer et al. (2013) and the lower shows the corresponding experimental environment. (b) An execution trace of policy π φ∧ where φ∧ = φtraverse ∧ φinterrupt.

Figure 5 : (left) Learning curve of discounted return (discount factor 0.98). The plot shows mean and one standard deviation calculated over 5 episodes. (right) Mean and standard deviation of episode length (steps) averaged over 5 episodes(a smaller number means faster accomplishment of the task).

Results on Training and Evaluation Performanceφ traverse φ interrupt φ ∧ (learned) φ ∧ (composed)

Table content: <table><tr><td colspan="2">Mean training time/std (Hour) 4.2/0.1</td><td>4.7/0.3</td><td>5.8/0.5</td><td>0.8/0.04</td></tr><tr><td>Mean task success rate/std</td><td colspan="3">85.4%/2.3% 80.2% / 2.5% 3.3%/ 1%</td><td>68.9%/ 3%</td></tr><tr><td>Mean discounted return/std</td><td>-13.2/2.3</td><td>-10.3/3.5</td><td>-40.6/10.5</td><td>-20.1/8.3</td></tr></table>

Peer review (iclr2019):
 This work proposed using temporal logic formulas to augment RL learning via the composition of previously learned skills. This work was very difficult to follow, so it is somewhat unclear what were the main contributions (since much of this seems to be covered by other works as referenced within the paper and as related to similar unreferenced works below). Moreover, regarding the experiments, many things were unclear (some of the issues are outlined below). While the overall idea of using logic in this way to help with skill composition is interesting and exciting, I believe several things must be addressed with this work. This includes: situating this work more clearly against existing similar works which use logic in this way, clearly defining the novel contributions of this work as compared to those and others, overall making the methodology more clear and specific (including experimental methodology), and comparing/contrasting against (or at least discussing differences with) methods with similar motivations (e.g., HRL multi-task learning, meta-learning) to emphasize the need/importance of this work — I am aware that at least 1 HRL work is mentioned, but this work is not really contrasted against it to help situate it. Questions/Concerns about Experiments: + Does Figure 5 show the averaged return over 5 runs, sum of discounted rewards averaged over 5 episodes per update step, or 5 episodes, each from a separate run averaged together? It is a bit unclear especially because the main text and the figure caption slightly differ. Also, average discounted return is somewhat different than average return, suggest updating the label to be clear also with the discount factor used. + What were the standard deviations for this across experiments? Even with averaging it seems that these runs are very high variance, would be good to understand what variance bounds to expect if using this method. + Why were average discounted returns reported in Figure 5 and not in Table 1? + What were the standard deviations on success rate and training time? Also what about sample complexity? + To my understanding the benefit here is reusability of learned skills via the automata methods described here. It would have made sense to compare against other HRL or multi-task learning methods in addition to just SQL or learning from scratch. For example how would MAML compare to this? + It is also unclear whether the presented results in Table 1 and Figure 5 are on the real robot or in simulation. The main text says, “All of our training is performed in simulation and the policy is able to transfer to the real robot without further fine-tuning.” So does this mean that Figure 5 is simulated results and Table 1 is on the real robot? Citations that should likely be made: + Giuseppe, Luca Iocchi, Marco Favorito, and Fabio Patrizi. "Reinforcement Learning for LTLf/LDLf Goals." arXiv preprint arXiv:1807.06333 (2018). + Camacho, Alberto, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. "Decision-making with non-markovian rewards: From LTL to automata-based reward shaping." In Proceedings of the Multi-disciplinary Conference on Reinforcement Learning and Decision Making (RLDM), pp. 279-283. 2017. + Camacho, Alberto, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. "Non-Markovian Rewards Expressed in LTL: Guiding Search Via Reward Shaping." In Proceedings of the Tenth International Symposium on Combinatorial Search (SoCS), pp. 159-160. 2017. Typos/Suggested grammar edits: “Skills learned through (deep) reinforcement learning often generalizes poorly across tasks and re-training is necessary when presented with a new task.” —> Often generalize poorly “We present a framework that combines techniques in formal methods with reinforcement learning (RL) that allows for convenient specification of complex temporal dependent tasks with logical expressions and construction of new skills from existing ones with no additional exploration.” —> Sentence kind of difficult to parse and is a run-on “Policies learned using reinforcement learning aim to maximize the given reward function and is often difficult to transfer to other problem domains.” —> ..and are often.. “by authors of (Todorov, 2009) and (Da Silva et al., 2009)” —> by Todorov (2009) and Da Silva et al. (2009) Also several other places where you can use .citet instead of .cite