Temporal Logic Imitation: Learning Plan-Satisficing Motion Policies from Demonstrations
Authors: Yanwei Wang
Nadia Figueroa
Shen Li
Ankit Shah
Julie Shah
Venue: CoRL 2022 Oral (2022)
Keywords: Learning from Demonstration
Dynamical Systems
Formal Methods
Linear Temporal Logic
Certifiable Imitation Learning

Abstract: Learning from demonstration (LfD) has successfully solved tasks featuring a long time horizon. However, when the problem complexity also includes human-in-the-loop perturbations, state-of-the-art approaches do not guarantee the successful reproduction of a task. In this work, we identify the roots of this challenge as the failure of a learned continuous policy to satisfy the discrete plan implicit in the demonstration. By utilizing modes (rather than subgoals) as the discrete abstraction and motion policies with both mode invariance and goal reachability properties, we prove our learned continuous policy can simulate any discrete plan specified by a linear temporal logic (LTL) formula. Consequently, an imitator is robust to both task- and motion-level perturbations and guaranteed to achieve task success.

Temporal Logic Imitation: Learning Plan-Satisficing Motion Policies from Demonstrations
Yanwei Wang MIT
Nadia Figueroa University of Pennsylvania
Shen Li
Ankit Shah Brown University
Julie Shah MIT
MIT
Abstract: Learning from demonstration (LfD) has successfully solved tasks fea- turing long time horizon. However, when the problem complexity also includes human-in-the-loop perturbations, state-of-the-art approaches do not guarantee the successful reproduction of a task. this work, we identify the roots of this chal- lenge as the failure of learned continuous policy to satisfy the discrete plan im- plicit in the demonstration. By utilizzng modes (rather than subgoals) as the dis- crete abstraction and motion policies with both mode invariance and goal reacha- bility properties, we prove our learned continuous policy can simulate any discrete plan specified by a linear temporal logic (LTL) formula. Consequently, imitator is robust to both task- and motion-level perturbations and guaranteed to achieve task success. Project page: ht https: //sites google cwm/view////eccc... wwtp//ldddd
Keywords: Certifiable Imitation Learning, Dynamical Systems, Formal Methods
Successful
Replay
invartance
failure
Figure I: (a) A successful replay of the scooping task. The robot is (b) robust to motion-level perturbations; (c) experiences an invariance failure (i.c., drops material) after task-level perturbation; and (d) re-scoops after a task-level perturbation, avoiding failure after DS motion policy modulation.
1 Introduction
In prior work, learning from demonstration (LiD) [1. 2] has successfully enabled robots to accomplish multi-step tasks by segmenting demonstrations (primarily robot end-effector or tool trajectories) into sub-tasks/goals [3, 4, 5, 6, 8). phases [9, 10], keyframes [11, 12], skillspprinittees/ppiioss [13, 14, 15, 16). Most of these abstractions assume reaching subgoals se- quentially will deliver the desired outcomes; however, successful imitation of many manipulation tasks with spatial/temporal constraints cannot be reduced to imitation at the motion level unless the learned motion policy also satisfies these constraints. This becomes highly relevant if want robots to not only imitate, but also generalize, adapt and be robust to perturbations imposed by hu- mans who are in the loop of task learning and execution. LiD techniques that learn stable motion policies with convergence guarantees (c.g.. Dynamic Movement Primitives (DMP) [17]. Dynamical System (DS) [18]) are capable of providing such desired properties but only at the motion level. As shown in Fig. 1 (a-b) the robot can successfully replay a soup- scooping task while being robust physical perturbations with learned DS. Nevertheless, if the spoon orientation perturbed to state where the material is dropped, Fig. 1 (c), the motion policy will still lead the robot to the target, unaware of the task-level failure or how recover from it. To alleviate this, in this work, we introduce an imitation learning approach that is capable of i) reacting to such task-level failures with Linear Temporal Logic (LTL) specifications, and ii) modulating the learned DS motion policies to avoid repeating those failures as shown in Fig. 1 (d).
Example We demonstrate that successfully reaching a goal via pure motion-level imitation does not imply successful task exccution. The illustrations in Fig. 2 represent a 2D simplification of the soup-scooping task, where task success requires continuous trajectory simulate discrete plan of consecutive transitions through the colored regions. Human demonstrations, shown in Fig. 2 (a), are employed learn a DS policy [19], depicted by the streamlines in Fig. (b). The policy is
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

18
A mode abstraction of the 2D soup-scooping task: is the spoon's orientation, and 22 is spoon's s distance to the soup. (a) Task: To move the spoon's configuration from white region (spoon without soup) = yellow region (spoon in contact with soup) = pink region (spoon holding soup) > green region (soup at target). (Note that transitions (white = pink) and (white + green) are not physically realizable.) Black curves denote successful demonstrations. (b) Learning DS policies [19] over unsegmented data can result in successful task replay (blue trajectories), but lacks guarantee duc to invalid transitions (red trajectories). (e) Trajectories segmented into three colored regions (modes) with orange attractors. result in invariance failures (i.e., traveling outside of modes as depicted by red trajectories).
Figure 2:
(d-f) Learning DSs on segments may still
stress-tested by applying external perturbations, displacing the starting states of the policy rollouts. As shown, only blue trajectories succeed in the task, while the red ones fail due to discrete transi- tions that are not physically realizable (e.g.. white = pink). As shown in Fig. 2 (c-f), even if the demonstrations are further segmented by subgoals (and corresponding DS policies are learned), this issue is not mitigated. While one could treat this problem as covariate shift and solve it by asking the humans for more demonstrations [20]. in this work, we frame it as the mismatch between learned continuous policy and a discrete task plan specified by the human in terms of a logical formula. Specifically, the core challenges illustrated by this example point constraints that are insufficient to represent the boundary of a discrete abstraction; and 2) the continuous policy can deviate from demonstrated discrete plan when perturbed to unseen parts of the state space, and is incapable of replanning to ensure all discrete transitions are valid.
two-fold: 1) subgoals only impose
To address these challenges, our proposed approach employs "modes" as the discrete abstractions. We define a mode as a set of robot and environment configurations that share the same sensor read- ing [21, 22]; e.g., in Fig. 2 each colored region is unique mode, and every mode has bound- ary that imposes path constraints on motion policies. Additionally, we use a task automaton as a receding-horizon controller that replans when a perturbation causes the system to travel outside mode boundary and triggers an unexpected sensor change; e.g., detecting a transition from yellow > white instead of the desired yellow = pink will result in new plan: white = yellow = pink + green. In this work, we synthesize a task automaton from a linear temporal logic formula (LTL) that specifies all valid mode transitions. We denote the problem of learning policy that respects these mode transitions from demonstrations as temporal logic imitation (TLI). In contrast to tempo- ral logic planning (TLP) [23], where the workspace is typically partitioned into connected convex cells with known boundaries, we do not know the precise boundaries of modes: consequently, the learned policy might prematurely exit the same mode repeatedly, causing the task automaton to loop without termination. To ensure any discrete plan generated by the automaton is feasible for the con- tinuous policy, the bisimulation criteria [24, 25] must hold for the policy associated for each mode. Specifically, every state starting in the same mode should stay in the mode (invariance) until even- tually reaching the next mode (reachability). The violations of these conditions are referred to as invariance failures and reachability failures respectively.
Contributions First, we investigate TLP in the setting of LID, and introduce TLI a novel for- mulation to address covariate shift by proposing imitation with respect to a mode sequence instead of a motion sequence. Second, leveraging modes as the discrete abstraction, we prove that a state- based continuous behavior cloning (BC) policy with a global stability guarantee can be modulated to simulate any LTL-satisficing discrete plan. Third, we demonstrate that our approach LTL-DS, adapts to task- perturbations via I.TL--saisffiing automaton's 's replanning and recovers from letion-level perturbations via DS' stability during a multi-step. non-prehensile manipulation task.
2 Related Works
Temporal Logic Motion Planning LTL is a task specification language widely used in robot mo- tion planning [26, 27, 28, 23]. Its ease of use and efficient conversion [29] to an automaton have spurred substantial research into TLP [25, 30, 31], which studies how to plan a continuous trajec- tory that satisfies a LTL formula. However, TLP typically assumes known workspace partitioning and boundaries a priori, both of which are unknown in the rarely explored TLI setting. While a robot can still plan in uncertain environments [32, 33], LfD bypasses the expensive search in high-dimensional space. Recent works [34, 35] have considered temporal logic formulas as side- information to demonstrations, but these formulas are treated as additional loss terms or rewards,

---

and are not guaranteed to be satisfied. The key motivation for using LTL is to generate reactive discrete plan, which can also be achieved by a finite state machine [14] or behavior tree [36].
Behavior Cloning We consider subelass of LfD methods called state-based behavior cloning (BC) that learns the state-action distribution observed during demonstrations [37]. DAGGER [20], BC- variant fixing covariate shift, could reduce the invariance failures depicted in Fig. 2. but requires online data collection, which our framework avoids with a LTL specification. To satisfy goal reach- ability,, we employ a DS-based LID technique [38]. Alternatives to this choice include certified NN-based methods [39, 40], DMPs [41], partially contracting DS [42] and Euclideanizing-fon
[43]. To satisfy mode invariance, we modulate the learned DS to avoid invariance failure as state- space boundaries [44], similar to how barrier functions are learned to bound a controller [45, 46, 47]. Multi-Step Manipulation Prior LfD works [13, 14, 10, 48] tackle multi-step manipulation by seg- menting demonstrations via a hidden Markov model. Using segmented motion trajectories, [13] learned a skill tree, [14] learned DMPs, [10] learned phase transitions, and [49] learned task model. Most of these works assume a linear sequence of prehensile subtasks (pick-and-plans -nd--laee) with- out considering how to replan when unexpected mode transitions happen. [48, 49] considered non-prehensile scooping task similar to ours, but their reactivity only concerned collision avoidance in single mode. [50, 6] improved BC policies with RL, but offered no guarantee of task success.
3 Temporal Logic Imitation: Problem Formulation
Let IR" represent the n-dimensional continuous state of robotic system; e.g., the robot's end- effector state in this work. Let a = [a1..... am]] I E {0, 13"h be an m-dimensional discrete sensor state that uniquely identifies mode
C(a). We define system state as tuple, s = (x, a) e R" x (0, 1) Overloading the notation, we use a E Σ, where E = fo; ) to represent the set of all system states within the same mode mode-i.e., d, {s (æ, a) ((a) = o,). In contrast, we use 6; = {a|s = (x, a:) E os} to represent the corresponding set of robot states. Note x cannot be one-to-one mapped to s, e.g., level spoon can be either empty holding soup. Each mode is
associated by goal-oriented policy, with goal o. E IR". A successful policy that accomplishes a multi-step task 7 with a corresponding LTL specification & can be written in the form:
1 (2, a: Φ) WM (o, f((;; 2,)
(1)
with (a)), being the Kronecker delta that activates a mode policy fi(;; 2,, : IR* + IR"
encoded by set of learnable parameters 0; and goal x;. Mode activation is guided by an LTL- equivalent automaton l2(aa - a; choosing mode o, based on current sensor reading a.
Demonstrations Let demonstrations for task be E = {1 ærd a ) 12 where wfid, .rdd, add are robot state, velocity, and sensor state at time in demonstration d, respectively, and Ta is the length of each d-th trajectory. A demonstration is successful if the continuous motion traces through sequence of discrete modes that satisfies the corresponding LTL task specification. Perturbations External perturbations, which many works in Sec. 2 avoid, constitute an inte part of our task complexity. Specifically, we consider: (1) motion-level perturbations that displace the continuous motion within the same mode, and (2) task-level perturbations that drive the robot out- side of the current mode. Critically, motion-level perturbations do not cause plan change instanta- neously, but they can lead to future unwanted mode transitions due to covariate shift. Environmental stochasticity is ignored, as its cumulative effects Problem Statement Given (1) an LTL formula ø specifying valid mode transitions for a task т, and (2) successful demonstrations E. we seek to learn a policy defined in Eq. that generates continuous trajectories guaranteed to satisfy the LTL specification despite arbitrary external perturbations.
also be simulated by external perturbations.
4 Preliminaries
4.1 LTL Task Specification
LTL formulas consist of atomic propositions (AP), logical operators, and temporal operators [51, 23]. Let II be set of Boolean variables; an infinite sequence of truth assignments all APs in II is called the trace [ The notation [T]] L } & means the truth assignment at time satisfies the LTL formula G. Given II, the minimal syntax of LTL can be described as:
: + Φ1 V 02
X
Φ1 U@2
where p is any AP in II, and Φ and Φ are valid LTL formulas constructed from y using Eq. 2. The operator X is read as 'next,' and X intuitively means the truth assignment to APs at the next time step sets Φ1 as true. U is read as 'until' and, intuitively, @i Uo means the truth assignment to APs
3

---

sets Φ1 as true until @2 becomes true. Additionally, first-order logic operators - (not), A (and), V (or), and + (implies), as well as higher-order temporal operators F (eventually), and G (globally), are incorporated. Intuitively, FØ1 means the truth assignment to APs eventually renders di true and GØ means truth assignment to APs renders Φ1 always true from this time step onward.
4.2 Tesk-Level Reactivity LTL
To capture the reactive nature of system given sensor measurements, the generalized reactivity
(1) (GR(1)) fragment of L.T. [29, 30] can be used. Let the set of all APs = X U., where sensor states form environment APs X' {ai,..., 0m) } and mode symbols form system APs y = (o1...., 01). GR(1) formula is of the form Φ = (Øe + d.) [29], where de models the assumed environment behavior and o, models the desired system behavior. Specifically,
Φ = 0 65 0g
& = of A de A A
(3)
of and of are non-temporal Boolean formulas that constrain the initial truth assignments of A' and ) (e.g., the starting mode). of and of are LTL formulas categorized as safety specifications that describe how the system and environment should always behave (e.g., valid mode transitions). of and og are LTL formulas categorized as liveness specifications that describe what goal the system and environment should eventually achieve (c.g.. task completion) [23]. The formula ø guarantees the desired system behavior specified by o, if the environment is admissible-iie. de true-and can be converted to an automaton S2g that plans a mode sequence satisfying Φ by construction [30].
4.3 Motion-Level Reactivity in DS
LPV-DS [19] can be learned in minutes from few single demonstration and has form:
K 8 == f()) Σ (x(x)(A* (4)
P PA* = { -
{
Q (Q*) < 0
4k (5)
R=1
where A E Ruxn, bz E R are the k-th linear system parameters, and (x(x)
R"
R+ is
the mixing function. To certify global asymptotic stability (G.A.S.) of Eq. 4, a Lyapunov function V(x) = (r "T) x²) with P PT > O, is used to derive the stability constraints in Eq. 5. Minimizing the fitting error of Eq. 4 with respect to demonstrations subject to constraints in Eq. 5 yields a linear DS with stability guarantee [19]. To learn the optimal number K and mixing function (I) we use the Bayesian non-parametric GMM fitting approach presented in [19].
4.4 Bisimulation between Discrete Plan and Continuous Policy
To certify continuous policy will satisfy LTL formula o, one show the policy can simulate any LTL-satisficing discrete plan of mode sequence generated by Qg. To that end, every mode's associated policy must satisfy the following bisimulation conditions [25, 23]:
Condition 1 (Invariance). All states starting in mode must remain within the same mode when following that mode's policy; i.e., Vi Vt (s" E a. s' 01)
Condition 2 (Reachabilityy. All states starting in mode must reach their next modes while fol- lowing the current mode's policy; ie., Vi 3T (8" E o; + sf E a,)
5 LTL-DS: Methodology
To solve the TLI problem in Sec. 3, we introduce a mode-based imitation policy LTL-DS:
t (I. Φ)
M.M (os Mr (a;Ti(x), fi(:; 0z, 2;,
(6)
offline learning
online learning
offline learning
During offline learning, we synthesize the automaton P. from & as outlined in Sec. 4.2 and learn DS policies fa from E according to Sec. 4.3. While the choice of DS satisfies the reachability condition as explained later, DS rollouts are not necessarily bounded within any region. Neither do we know mode boundaries in TLI. Therefore, an online learning phase is necessary where for each mode policy f. we learn an implicit function, lx(x)) IR R+, that inner-approximmtes the mode boundary in the state-space of the robot x E N With a learned I' [1(x)) for each mode, we can construct a modulation matrix M, that ensures each M,f, be mode invariant as discussed below.
5.1 Offline Learning Phase
In the offline learning phase, the user provides successful demonstrations E and a LTT. formula &i as described in Sec. 4.2, which necessarily includes the definition of sensors and AP regions.

---

Leaming
Reraaion
Pertutbation
Hurtan
Joann
cG
DS
reloot
LTL
X
Pubked replay
PANDA
MAZAM
chooses f
medulata DB
Falled replay
Readlh 500 transfer
VARAT
LT. speditcation
Automatoon
Re-accop spoan spth
curment mode
Figure 3: (a) Task automaton for scooping task LTL. Mode a, b, d are reaching. scooping. transporting and done mode respectively, Atomic proposition r, s, denote sensing the spoon reaching the soup, soup on spoon and task success respectively. During successful demonstration, only black mode transition a = b = c = d is observed. Additional gray valid transitions b + 6, c + and c + b are given by the I.TI. to help recover from unexpected mode transitions. (b) System flowehart of L.T..DS.
Synthesis of LTL-Satisficing Automaton Given e we use [52] to convert it into an automaton, which plans a mode sequence that satisfies 6 by construction. Assuming all possible initial condi- tions for the system are specified in the LTL, the automaton is always deployed from a legal state. censor-based Motion Segmentation and Attractor Identification Given the demonstration set = and AP regions related to the set of M modes we can automatically segment the trajectories into M clusters and corresponding attractor set X For more details, refer to Appendix C.
Ensuring Goal Reachability with Learned DS Mode Policies While any BC variant with a sta- bility guarantee can satisfy reachability (sce Sec. 2), we focus on the G.A.S. DS formulation and learning approach defined in Section 4.3 that ensures every x E R" is guaranteed to reach क By placing 2; within the boundary set of 5, for mode j.. ensure mode o, is reachable from every s in mode o;. Note f(x) cannot model sensor dynamics in a. Yet, we employ mode abstraction to reduce the imitation of a system state trajectory in s which includes the evolution of both the robot and sensor state-to just robot state trajectory in x.
5.2 Online Learning Phase
Iterative Mode Boundary Estimation via Invariance Failures As shown in Fig. 2, DS can suffer from invariance failures regions without data coverage. Instead of querying humans for more data in those regions [20], we leverage sparse events of mode exits detected by sensors estimate the unknown mode boundary. Specifically, for each invariance failure, we construct a cut that sep- arates the failure state, ,=', from the mode-entry state, 2º, the last in-mode state, - and the mode attractor, x². We ensure this separation constraint with a quadratically constrained quadratic program (QCQP) that searches for the normal direction (pointing away from the mode) of hyper- plane that passes through -- such that the plane's Sane's distance to I is minimized. The intersection of half-spaces cut by the hyper-planes inner-approximates a convex mode boundary, as seen in Fig. 4. Adding cuts yields better boundary estimation, but is not necessary unless the original vector field flows out of the mode around those cuts. For more details, refer to Appendix E.3.
Ensuring Mode Invariance by Modulating DS We treat each cut as a collision boundary that de- flects the DS flows following the approach in [44, 53J. In our problem setting the mode boundary is analogous to workspace enclosure rather than random task-space object. Let existing cuts form an implicit function, T(x:) : R² - IR+, denote the estimated interior with T(x) < 1, P(x) = the boundary and I(xx)= > 1 the exterior of mode. 0 I'(x) < do monotonically increases
moves away from a reference point x" inside the mode. For x outside the cuts, or inside but moving away from the cuts, we leave f(x) unchanged; otherwise, we modulate f(z) to not collide with any cuts as i = (x) by constructing a modulation matrix (xx through eigenvalue decomposition:
[x) E(s) E(x) = jr(z) e (x) ex)1 r(x) D(x)) = diag(A, (r), dea (z)), (x(x) 1- \x(x)= =1
The full-rank basis E(x) consists of a reference direction r(z) stemming from x" toward x, and d - 1 directions spanning the hyperplane orthogonal to VT which in this case is the closest cut to x. In other words, all directions e (x)...e=- (x) are tangent to the closest cut, except r(x). By modulating only the diagonal component, X, (r)). with ((r), we have (- - 0 as approaches the closest cut, effectively zeroing out the velocity penetrating the cut while preserving velocity tangent to the cut. Consequently, as long as there are cuts bounding the mode, the modulated DS will not experience invariance failures where the original DS would. Notice this modulation strategy is not limited to DS, and can apply to any state-based BC methods to achieve mode invariance.
5

---

These mode exits are unforced, lack invariance
Figure 4: An illustration iterative estimation of mode boundary with cutting planes. A system enters a mode with an unknown boundary (dashed line) at the black circle, and is attracted to the goal at the orange circle. Its trajectory in black shows the original policy rollout, and its trajectory in red is driven by perturbations. After the system exits the mode and before it eventually re-enters the same mode through replanning. the last in-mode state (yellow circle) to bound mode from the failure state (red cross). When the system is inside the cuts, it experiences modulated DS that never leaves the mode (flows entering the mode are not modulated); when the system is outside the cuts but inside the mode, it follows the original DS. Note only mode exits in black are invariance failures in need of modulation (green circles): mode exits in red driven by perturbations to illustrate that more cuts lead to better boundary approximation.
is placed
6 Proof
Next, we prove LTL-DS produces continuous trajectory that satisfies a LTL specification. We start with assumptions and end with theorems. Detailed proofs are provided in Appendix A.
Assumption 1. All modes are convex.
This assumption leads to the existence of at least one cut-i.e., the supporting plane [54], which can separate a failure state the boundary from any states within the mode. A corollary is that boundary shared by two modes, which we call guard surface, Gıy = 6, n 6j. is also convex. Since all transitions out of a mode observed during demonstrations reside the mode boundary, their average location- location-whie which we use as the attractor for the mode-win will also be on the boundary.
Assumption 2. There are a finite number of externally exerted motion and task-level perturbations of arbitrary magnitude.
Given zero perturbation, all BC methods should succeed in any task replay, the policy rollout will always be distribution. If there are infinitely many arbitrary perturbations, no BC methods will be able to reach a goal. Here, we study the setting between these extremes-wwere -where there are finite number external perturbations causing unexpected mode exits.
Assumption 3. Every unexpected mode transition only results sensor states that have been seen in the demonstrations.
While demonstrations of all valid mode transitions are not required, they must minimally cover all possible modes. If the system encounters a completely new sensor state during online interaction, is reasonable to assume that no BC methods could recover from the mode unless more information about the environment were provided.
nonlinear DS defined by Eg. 4, learned from demonstrations, and modulated by cutting planes as described in Section 5.2 with the reference point x" set at the Proof: See Appendix A. O
Theorem 1. (Key Contribution
attractor x", will never penetrate the cuts and G.A.S. at x=
Theorem 2. (Key Contribution 2) The continuous trace of system stales generated by LTL-DS de- fined in Eq. 6 satisfies any LTL specification o under Asm. 1, 2, and 3. Proof: See Appendix A.
7
Experiments
7.1 Single-Mode Invariance and Reachability
We show quantitatively both reachability and invariance necessary for task success. We compare DS and a NN-based BC policy (denoted as BC) to represent policies with and without a stability guarantee respectively. Fig. shows that policy rollouts start to fail (turn red) as increasingly larger perturbations are applied to the starting states; however, DS only suffers from invariance failures, while BC suffers from both invariance and reachability failures (due diverging flows and spurious attractors). Fig. 5 (right) shows that all flows are bounded within the mode for both DS and BC after two cuts. In the case of DS, flows originally leaving the mode are now redirected to the attractor by the cuts; in the case of BC, while no flows leave the mode after modulation, spurious attractors are created, leading to reachability failures. This is a counterfactual illustration of Thm. 1, that policies without stability guarantee are not G.A.S. after modulation. Fig. 6 verifies this claim quantitatively and we empirically demonstrate that a stable policy requires only four modulation cuts to achieve perfect success rate- ratc-which an unstable policy cannot be modulated to achieve.

---

Original
Nome
Small
Large No
1-Cut Motusation
2-Gut Meduaaiin
Policy rollouts from different starting states for a randomly generated convex mode. The top row shows BC results, and the bottom row depicts DS results. The left column visualizes the original policies learned from two demonstrations (black trajectories) reaching the orange attractor. The middle columns add different levels Gaussian noise to initial states sampled from the demonstration distribution. Blue trajec- tories successfully reach the attractor, while red trajectories fail due to either invariance failures or reachability failures. (Note that these failures only occur at locations without data coverage.) The right columns show that cutting planes (blue lines) separate failures (red crosses) from last-visited in-mode states (yellow circles), and consequently modulate policies be mode-invariant. Applying cutting planes BC policies without stability guarantee cannot correct reachability failures within mode, More results are Appendix
Figure 5:
Polay
Rescazhilty
Invanaace
100
91.9
100
DS-mmad
100
140
Tock Gccoms
100
I
100
CS undll loge
40
sube
20
DC
Smaill Nobx
Nobre
Figure 6: (left) The success rate (%) of single-mode reaching task. As we began to sample out distribution by adding more noise to the demonstrated states, the BC's success rate degraded more rapidly than the DS'. After modulation, DS (+mod) maintained success guarantee, which BC (+mod) fell short of due to the base policy's lack of stability guarantce. (right) Empirical evidence that single-mode invariance requires only finite number of cuts for base policy with stability guarantee. Regardless of noise level, DS achieves 100% success rate after four cuts, while BC struggles to improve performance with additional cuts. Thick lines represent mean statistics; shaded regions the interquartile range. More details are provided in Appendix E.
bobottdd
Boblod.d
bobood
Bobboood
Booood
Braa
Broeodd
biooldd
boom
Coob
DBwrod
ared
Brodood
Bololed
cod
Dron
b-aa
Figure 7: Rollouts of a multi-step scooping task under perturbations. The first row indicates that DS policies sequenced by automaton but without boundary estimation lead to looping: the second and third rows show that modulation can prevent looping eventually allow the system reach goal mode despite re- peated perturbations. We depict the mode sequence planned by the automaton at the top of each aub-figure, and blue bounding boxes indicate the current mode transitions actually being detected, Black and red trajectories signify original and perturbed policy.
7

---

7.2 Multi-Modal Reactivity and Generalization to New Tasks
We now empirically demonstrate that a reactive discrete plan alone is insufficient to guarantee task success without mode invariance for tasks with multiple modes. Consider the multi-modal soup- scooping task introduced in Fig. 2. Formally, we define three environment APs, T, t, sensing the spoon is in contact with the soup, has soup on it, and has arrived at a target location respectively. Given successful demonstrations, sensors will record discrete transitions (-r - + (rA-sA t)) + ( A -t) = ( A -s At), from which four unique sensor states are identified. We label each sensor state as a mode with robot AP a (reaching) = b (scooping) + c (transporting) + (done). Invariance of mode b enforces contact with soup during scooping, and invariance of mode constrains the spoon's orientation in order to avoid spilling. We follow the TLP convention to assume I.TI. formulas are provided by domain experts (although they can also be learned from demonstrations [51, 55J.) The specific L.TI. for the soup-scooping task is detailed in Appendix F, and can be converted into a task automaton as shown in Fig. 3.
One might assume the automaton is sufficient to guarantee task success without modulation, as it only needs to replan finite number of times assuming a finite number of perturbations; however, not enforcing mode invariance can lead to looping at the discrete level, and ultimately renders the goal unreachable, as depicted in the top row of Fig. 7. In contrast, looping is prevented when modulation is enabled, as the system experiences each invariance failure only once. We found 50% of the learned policies without modulation could get stuck in looping after a finite number of perturbations, while all experiments with modulation succeeded in the task replay.
Robot Experiments First, we implemented the soup-scooping task on Franka Emika robot arm as shown in Fig. 1. We show in videos on our website that (1) DS allowed our system to compliantly react to motion-level perturbations while ensuring system stability; (2) LTL allowed our system to replan in order to recover from task-level perturbations; and (3) our modulation ensured the robot learned from previous invariance failures in order to avoid repeating them. To test robustness against unbiased perturbations, we collected 25 trials from 5 humans as seen in in Appendix H. All trials succeed eventually in videos. We did not cherry-pick these results, and the empirical 100% success rate further corroborates our theoretic success guarantee. Second, we implemented an inspection task as a permanent interactive exhibition at MIT museum, with details documented in Appendix Lastly, we show a third color tracing task testing different automaton structures on our website.
Generalization Once a DS is learned we can generalize to a new task sharing the same set of modes observed in demonstrations given a new LTL formula. Consider another multi-step task of adding chicken and broccoli to a pot. Different humans might give demonstrations with different modal structures (e.g., adding chicken vs adding broccoli first). LTL-DS could learn individual DS which can be flexibly combined to solve new tasks with new task automatons. To get these different task automatons, a human just needs to edit the of portion of the LTL formulas differently. We provide further details of this analysis in Appendix G.
8 Limitations
Our approach of integrating logic formulas into LID requires defining an appropriate abstraction for a task. For example, allocating sensor to detect contact events requires domain knowledge. Our work is based on the assumption that for well-defined tasks (e.g.. assembly tasks in factory set- tings), domain expertise in the form of logic formula is cheaper knowledge source than collecting hundreds of motion trajectories to avoid covariate shift (we use up to 5 demonstrations in all exper- iments). However, even when abstractions for task are given by an oracle, a LiD method without either the invariance or the reachability property will not have formal guarantee of successful task replay, which this work's focus. In future work, we will learn such abstractions directly from sensor streams such as videos so that our approach gains more autonomy without losing reactivity.
9 Conclusion
In this paper, we formally introduce the problem of temporal logic imitation as imitating continuous motions that satisfy a LTL specification. We identify the fact that learned policies do not necessarily satisfy the bisimulation criteria as the main challenge of applying LfD methods to multi-step tasks. To address this issue, we propose a DS-based approach that can iteratively estimate mode boundaries to ensure invariance and reachability. Combining the task-level reactivity of LTL and the motion- level reactivity of DS, we arrive at an imitation learning system able to robustly perform various multi-step tasks under arbitrary perturbations given only a small number of demonstrations, and demonstrate our system's 's practicality on a real Franka robot.
8

---

Acknowledgments
We would like to thank Jon DeCastro, Chuchu Fan, Terry Suh, Rachel Holladay, Rohan Chitnis, Tom Silver, Yilun Zhou, Naomi Schurr, and Yuanzhen Pan for their invaluable advice and generous help.
References
[1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.
[2] H. Ravichandar, S. Polydoros, S. Chernova, and Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3:297-330, 2020.
[3] S. Ekvall and D. Kragic. Robot learning from demonstration: a task-level planning approach. International Journal of Advanced Robotic Systems, 5(3):33, 2008
[4] D. H. Grollman and O. C. Jenkins. Incremental learning subtasks from unsegmented demon- stration. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 261-266. IEEE, 2010.
[5] J. R. Medina and A. Billard. Learning stable task sequences from demonstration with linear parameter varying systems and hidden markov models. In Conference on Robot Learning, pages 175-184. PMLR, 2017.
[6] A. Gupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11955 2019.
[7] A. Mandlekar, D. Xu, R. Martín-Martin, S. Savarese, and L. Fei-Fei. Learning to general- ize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06085, 2020.
[8] S. Pirk, K. Hausman, A. Toshev, and M. Khansari. Modeling long-horizon tasks as sequential interaction landscapes. arXiv preprint arXiv:2006. 04843, 2020.
[9] P. Pastor, M. Kalakrishnan, L. Righetti, and S. Schaal. Towards associative skill memories. In 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 20/2), pages 309-315. IEEE, 2012.
[10] O. Kroemer, C. Daniel, G. Neumann, H. Van Hoof, and J. Peters. Towards learning hierarchical skills for multi-phase manipulation tasks. In 2015 IEEE international conference on robotics and automation (ICRA), pages 1503-1510. IEEE, 2015.
[11] B. Akgun, M. Cakmak, J. W. Yoo, and A. L. Thomaz. Trajectories and keyframes for kines- thetic teaching: A human-robot interaction perspective. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robo Interaction, pages 391-398, 2012.
[12) C. Pérez-D' Arpino and J. A. Shah. C-learn: Learning geometric constraints from demonstra- tions for multi-step manipulation in shared autonomy. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 4058-4061 IEEE, 2017.
[13] G. Konidaris, Kuindersma, Grupen, and A. Barto. Robot learning from demonstration by constructing skill trees. The International Journal of Robotics Research, 31(3):360-375 2012.
[14] S. Nickum, S. Chitta, A. G. Barto, B. Marthi, and S. Osentoski. Ineremental semantically grounded learning from demonstration. In Robotics: Science and Systems, volume 9, pages 10-15607, Berlin, Germany, 2013.
[15] R. Fox, S. Krishnan, Stoica, and K. Goldberg. Multi-level discovery of deep options. arXiv preprint arXiv:1703.08200 2017.

---

[16] N. B. Figueroa Fernandez. From high-level to low-level robot learning of complex tasks: Leveraging priors, metrics and dynamical systems. Technical report, EPFL, 2019.
[17] M. Saveriano, F. J. Abu-Dakka, A. Kramberger, and L. Peternel. Dynamic movement prim- itives in robotics: A tutorial survey. CoRR, abs/2102.03861, 2021. URL https://arxiv: org/abs/21il 0386.
03861.
[18] A. Billard, S. Mirrazavi Salehian, and N. Figueroa. Learning for Adaptive and Reactive Robot Control: A Dynamical Systems Approach. MIT Press, Cambridge, USA, 2022.
[19] N. Figueroa and A. Billard. A physically-consistent bayesian non-parametric mixture model for dynamical system learning. In CoRI., pages 927-946, 2018.
[20] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[21] A. J. Van Der Schaft and J. M. Schumacher. An introduction to hybrid dynamical systems, volume 251. Springer London, 2000.
[22] C. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling. and T. Lozanoo-éérre Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021.
[23] H. Kress-Gazit, M. Lahijanian, and V. Raman. Synthesis for robots: Guarantees and feedback for robot behavior. Annual Review of Control, Robotics, and Autonomous Systems, 1:211-236, 2018.
[24] R. Alur, T. A. Henzinger, G. Lafferriere, and G. J. Pappas. Discrete abstractions of hybrid systems. Proceedings of the IEEE, ((7):971-9844 2000.
[25] G. E. Fainekos. H. Kress-Gazit, and G. Pappas. Temporal logic motion planning for mobile robots. In Proceedings of the 2005 IEEE International Conference on Robotics and Automa- tion, pages 2020-20202222 2025. IEEE, 2005.
[26] Belta, A. Bicchi, M. Egerstedt, Frazzoli, E. Klavins, and G. Pappas. Symbolic planning and control of robot motion Igrand challenges of robotics]. IEEE Robotics & Automation Magazine, 14(1):61-70, 2007.
[27] E. M. Wolff, U. Topeu, and R. M. Murray, Automaton-guided controller synthesis for nonlinear systems with temporal logic. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4332-4333 IEEE, 2013.
[28] E. Plaku and S. Karaman. Motion planning with temporal-logic specifications: Progress and challenges. Al communications, 291):151-116, 2016.
[29] N. Piterman, A. Pnueli, and Y. Sa' ar. Synthesis of reactive (1) designs. In International Work- shop on Verification, Model Checking, and Abstract Interpretation, pages 364-380. Springer, 2006.
[30] H. Kress-Gazit, G. E. Fainekos, and G. J. Pappas. Cemporaa-logic-basee reactive mission and motion planning. IEEE transactions robotics, (110)-1377-1388 2009.
[31] J. A. DeCastro and H. Kress-Gazit. Synthesis of nonlinear continuous controllers for verifiably correct high-level, reactive behaviors. The International Journal of Robotics Research, 34(3); 378-394, 2015.
[32] A. M. Ayala, S. B. Andersson, and C. Belta. Temporal logic motion planning in unknown environments. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5279-5284. IEEE, 2013.
[33] M. Lahijanian, M. R. Maly, D. Fried, L. E. Kavraki, H. Kress-Gazit, and M. Y. Vardi. Iter- ative temporal planning uncertain environments with partial satisfaction guarantees. IEEE Transactions Robotics, 32(3),583-599, 2016.
10

---

[34] C. Innes and S. Ramamoorthy. Elaborating on learned demonstrations with temporal logic specifications. arXiv preprint arXiv:2002.00784 2020.
[35] A. G. Puranic, J. V. Deshmukh, and S. Nikolaidis. Learning from demonstrations using signal temporal logic in stochastic and continuous domains. IEEE Robotics and Automation Letters, 6(4):6250 6257, 2021.
[36] S. Li. D. Park, Y. Sung, A. Shah, and N. Roy. Reactive task and motion planning under temporal logic specifications. arXiv preprint arXiv:2103. 14464, 2021.
[37] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective imitation learning. Foundations and Trends@ in Robolics, 7(1-2)11-179, 2018. ISSN 1935-8253. doi: 10:110.1150 10.1561/3300000053.
[38] S. M. Khansari-Zadee and A. Billard. Learning stable nonlinear dynamical systems with gaus- sian mixture models. IEEE Transactions Robotics, 27(5):943-957) 2011.
[39] K. Neumann, A. Lemme, and J. Steil. Neural learning of stable dynamical systems based on data-driven Iyapunov candidates. 2013 IEEE/RS. International Conference on Intelligent Robots and Systems, pages 1216-1222, IEEE, 2013.
[40] C. Dawson, Z. Qin, S. Gao, and C. Fan. Safe nonlinear control using robust neural Iyapunov- barrier functions. In Conference on Robot Learning, pages 1724-1735. PMLR, 2022.
[41] A. J. Ijspeert, Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural compulation, 25(2):328- 373, 2013.
[42] H. Ravichandar, I. Salehi, and A. Dani. Learning partially contracting dynamical systems from demonstrations. In S. Levine, V. Vanhoucke, and K. Goldberg, editors, Proceedings of the Ist Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 369-378. PMLR, 13-15 Nov 2017. URL https: / http///ppoeeen.....//////// mmlr. press//77//raviccadde
html.
[43] M. A. Rana, A. Li, H. Ravichandar, M. Mukadam, S. Chernova, D. Fox, B. Boots, and N. Ratliff. Learning reactive motion policies in multiple task spaces from human demon- strations. In Conference on Robot Leurning, pages 1457-1468. PMLR, 2020.
[44] S. M. Khansari-Zaded and A. Billard. A dynamical system approach realtime obstacle avoidance. Autonomous Robots, 32(4):433 2012.
[45] A. Robey, H. Hu, L. Lindemann, H. Zhang, D. V. Dimarogonas, S. Tu, and N. Matni. Learn- ing control barrier functions from expert demonstrations. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 3717-3724. IEEE, 2020.
[46] M. Saveriano and D. Lee. Learning barrier functions for constrained motion planning with dynamical systems. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1 112-119. IEBE, 2019.
[47] C. Dawson, S. Gao, and C. Fan. Safe control with learned certificates: A survey of neural Iyapunov, barrier, and contraction methods. arXiv preprint arXiv:2202.11762. 2022.
[48] G. Ye and R. Alterovitz. Demonstration-guine motion planning. In Robotics research, pages 291-307. Springer, 2017.
[49] C. Bowen and R. Alterovitz. Closed-loop global motion planning for reactive execution of learned tasks. In 2014 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 1754-1760. IEEE, 2014.
[50] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint arXiv: 1709. 10087, 2017.
[51] A. J. Shah, P. Kamath, S. Li, and J. A. Shah. Bayesian inference of temporal task specifications from demonstrations. 2018.

---

[52] A. Lute,-Lutz, A. Lewkowicz, A. Fauchille, T. Michaud, E. Renault, and L. Xu. Spot 2.0 a framework for LTL and w-automata manipulation. Proceedings of the 14th International Symposium Automated Technology for Verification and Analysis (ATVA (16), volume 9938 of Lecture Notes in Computer Science, pages 122-129. Springer, Oct. 2016. doi:10.1007/ 978-3-319-46520-3..8
[53] L. Huber, A. Billard, and J.-J. Slotine. Avoidance of convex and concave obstacles with conver- gence ensured through contraction. IEEE Robotics and Automation Letters, 4(2):1462-1469, 2019.
[54] S. Boyd, S. P. Boyd, and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
[55] G. Chou, N. Ozay, and D. Berenson. Learning temporal logic formulas from suboptimal demonstrations: theory and experiments. Autonomous Robots, pages 1-26, 2021.
[56] D. Kasenberg and M. Scheutz. Interpretable apprenticeship learning with temporal logic spec- ifications. In 2017 IEEE 56th Annual Conference Decision and Control (CDC), pages 4914-4921. IEEE, 2017.
[57] A. Billard, S. Mirrazavi, and N. Figueroa. Learning for Adaptive and Reactive Robot Control: A Dynamical Systems Approach. 2022.
12


Reviews:
[{"review_id": "wz99DWX2G7_S", "reviewer": "Reviewer_QTy6", "summary": "The paper presents an approach to train sets of controllers to imitate demonstrations. The demonstrations can be segmented into sequences of steps separated by sensory events, leading to a representation of the tasks in the form of hybrid automata with sequences of sensory \"modes\". The method combines synergistically prior work on dynamical systems to obtain convergent controllers for each of the segments/steps, and prior work on temporal-logic synthesis to map the demonstrations into hybrid automata, to obtain convergent policies both in the motion level and in the task level (sequences of modes). The method is tested on synthetic data and with a proof of concept on a real robot.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written, although very tightly packed\n- The community is in need of methods that move away from “just” action imitation into some forms of imitation with better abstractions and objectives\n- The combination of control with convergence warranties with a hybrid automata is novel and interesting for the community.\n- It includes a demonstration of the approach on a real-robot\n\nWeaknesses:\n- The experimental evaluation is rather limited to understand the strengths and limitations. Many of the concepts are “evaluated” in simple synthetic 2D point-agent domains that, while illustrative, are more a proof of concept. There is only one task with full-robotic complexity. This makes it hard to understand the weaknesses of the proposed approach. For example, how well would it scale to more complex automata? What is the complexity of finding and perceiving the sensor events to switch between modes (partially mentioned in the limitations)? How stable is the boundary in the switch between states of the automata (usually, the problematic parts)?\n- There are a lot of concepts that are inherited from the theory of dynamic systems and temporal logic and that were presented before. While I appreciate the effort to make the paper self-contained with these explanations, the paper does not draw a clear line on what is novel and what is prior work applied here. A suggestion is to create a section “Preliminaries” to explain the used prior concepts that would make very clear what is new.\n- As far as I understand, the videos/experiments with the robot show the same person that created the solution (and knows the limitations) perturbing the agent. Some of the perturbations seem to be studied and I don't know if they are similar to the ones applied to the (failing) baselines. While this is good as a proof of concept, it would be great to perform some experiments with unbiased participants. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It is a good contribution, well written and well motivated. The paper is technically competent, although it is hard to understand what is novel contribution and what is prior work in dynamical systems and temporal-logic. The experiments are illustrative but additional robot experiments (more tasks, unbiased perturbations) would help understand the strengths and limits of the presented approach.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Mentioned in the weaknesses:\n- Make clear what is prior work and what is novel contribution, possibly with a \"Preliminaries\" section\n- Additional tasks and evaluation conditions on the real robot", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "KM-g6b--23n", "reviewer": "Reviewer_p7Tc", "summary": "This paper presents a temporal logic imitation approach which generates policies that satisfy a given linear temporal logic (LTL) specification.\nClassical learning from demonstration approaches (LfD) can often not ensure task satisfaction in the presence of perturbations. \nTo address this issue, the presented approach proposes a mode-based approach to satisfy the task while optimizing controllers that ensure stability. \nThe task is decomposed into a set of modes, e.g., moving a spoon to a soup, scooping the soup, and moving to the goal pose.\nLocal controllers within each mode ensure stability in the presence of perturbations.  \nThe paper shows that the presented approach satisfies invariance and reachability for the modes. \nThe approach has been evaluated in simulation and on a real robot in a soup-scooping task.\nResults indicate that the approach significantly improves performance under various perturbations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Robustly performing tasks under various perturbations is challenging. \nThis paper presents an interesting and promising control-theoretic approach to address this challenge.\nThe approach builds upon sophisticated control theory.\nThe proposed task abstraction into modes is compelling and helps to decompose the task.\nConstructing stable controllers for each mode to create plans is a key feature of this work.\nThe authors also provide various proofs and details on how to create the controllers and the convex operating regions in each mode.\nThe approach can work with various user-defined LTL formulas that specify how modes can be entered and how the system should transition from one mode to another. \nThe authors provide detailed explanations on the mode and how to construct the controllers.\nThe approach has been evaluated in a small simulation with various controlled perturbations and on a real robot which is great to see. \nThe submitted videos help to understand the approach better and showcase various aspects of the approach.\n\nAlthough the paper is interesting, its scope is not really fitting the learning theme of the conference. \nThe paper claims to address learning from demonstration. \nYet, the approach focuses on control-theoretic aspects (both main contributions are Theorem 1 and 2) and learning is left out.\nThe only reference to learning is on learning the linearized dynamics through Bayesian non-parametric GMMs.\nThe demonstrations as well as the LTL specifications are assumed to be provided to the approach.\nIt would have been great to see more integration into learning from demonstration frameworks. \nThe control theoretic details are great, but the paper does not really fit the scope of CoRL.\nMoreover, the paper claims to focus on LTL specifications and it introduces the necessary foundations.\nExcept for one experiment, the paper does not talk about LTL anymore. \nInstead, specifications are assumed to be provided by experts. \nHow can one create specifications that can be used in the approach?\nWhich conditions need to be fulfilled to specify transitions between modes?\nWhich limitations does the approach have?\nIn its current form, the paper does not provide enough details to justify the general claims of temporal logic imitation (the paper focuses on modes and constructing controllers).\nConsidering the applicability, the paper has focuses on a rather simple task for which the mode decomposition is straightforward.\nHow does the approach translate to other tasks?\nWhat if the task contains time-variant elements, such as dynamic objects?\nIn the control sections, the paper needs to provide more details.\nFor instance, how are attractors chosen?\nWhat kind of assumptions are made on the system/sensors?\nWhat happens if the system's dynamics are highly non-linear and the approximation in the paper leads to large linearization errors?\nThe paper also misses to explain the motivation behind many modeling steps, e.g., why are the controllers chosen as presented?\nLastly, the experiment section is very short, and it is not possible for a reader to follow the details and contributions of the experiments.  \nIt would be nice if the paper can properly introduce the considered environment, system, and task.\nThe limitations sections can also be improved by not just stating straightforward future work, but discussing the limitations of the approach and its applicability to a wide range of tasks.\n\nConsidering the paper's style, it is mostly well written. \nYet, the control-theoretic sections are very dense due to the page limit. \nThe authors could remove the unused LTL definitions to gain space and add more explanations.\nMoreover, the figures are very small, and it would be nice if the authors could add labels to them for easier reference. \nThe numbering of equations 3 and 4 can also be improved since it is not directly clear where Eq. 3 is. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In summary, the paper presents an interesting control-theoretic approach that shows promising results.\nHowever, the paper has no real focus on learning and seems out of scope for CoRL. \nThe integration with learning from demonstrations is not really described in the paper.\nMoreover, the approach does not really demonstrate temporal logic imitation.\nThe specifications are assumed to be provided and the paper does not explain how to create specifications for the use in the approach.\nTHe paper also does not discuss the temporal logic aspects in detail. \nThe experimental section also needs to be revised to properly demonstrate the approach and claims in various environments and tasks.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* Temporal logic imitation: How can one create specifications that can be used in the approach?\nWhich conditions need to be fulfilled to specify transitions between modes?\n* Assumptions: which assumptions do the authors make in their approach? Which tasks can this approach be applied to?\n* Experiments: More details on the task, the specification, and the evaluation criteria. \n* Approach: More details on choosing the attractors. \n* Limitations: What are limitations of the approach? When does the approach fail? Can it be applied to all tasks? "}}, {"review_id": "8C_QsAGP_a5", "reviewer": "Reviewer_L7Wt", "summary": "Current learning from demonstration (LfD) techniques, do not guarantee success in the presence of significant perturbations, and rarely have the ability to recover from task failures.  In this paper, the authors frame this deficiency as a failure to satisfy the high-level task specification which is implicitly present in the given demonstrations.\n\nBy leveraging task \"modes\" (the sets of sensor states which correspond to relevant components of the task) in combination with a LTL-automaton and dynamical system, their method is able to satisfy the task in the presence of arbitrary perturbation, and even recover from perturbations which cause previous steps of a task-plan to be undone. \n\nIn contrast with work in temporal logic planning, these \"modes\" are not given, but learned from the data with a quadratically constrained quadratic program to learn the boundary cutting planes of each mode.\n\nTheir experiments demonstrate their method with a soup-scooping task on a Franka robot arm.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe presentation is clear, and the formalizations are clean & understandable. The figures do a good job providing intuitive explanations of algorithm operation, and how it contrasts with traditional imitation learning methods.\n\nThe work is well situated --- alternative solutions to their problem are addressed and discussed (e.g., framing the problem as covariate shift and simply asking for more human demonstrations). The related work covers the relevant literature on temporal logic motion planning, imitation learning, and subgoal/long-horizon manipulation.\n\nThe contribution itself is strong, representing progress in augmenting LfD frameworks to incorporate task specification knowledge in a way that both robust to failures and aids recovery. In addition to empirical evidence for the value of their technique, the theoretic proofs clearly outline the assumptions under which their method can manage arbitrary perturbations.\n\nThe experiments were implemented on a real panda robot, and fair comparisons are made to NN-Behaviour Cloning policies to demonstrate the advantages of their method.\n\n# Weaknesses\n\nThe weaknesses I can identify mostly come from a desire to see this work extended.\n\nFor one, while the soup scooping example is cleanly demonstrates their idea, I would have liked to see it applied to a range of varied tasks with expressive modes. \n\nAdditionally, while the modes of the task are learned, the high-level sensor abstractions themselves used to learn these modes are given. It would be interesting to see if this approach could be applied to high-dimensional data-driven perception (e.g., streaming video pushed through a perception network). I note that this is mentioned by the authors in their own limitation section. \n\nThird, their use of GR(1) LTL as a constraint language inherently limits the types of cyber-physical problem that can be addressed with this approach. For plans involving continuous signals, timing constraints, or looser probabilistic guarantees, a more expressive language would likely be needed. I am unsure if transferring these techniques to something like P-STL would be a non-trivial endeavor.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I feel confident in recommending this paper for publication --- the presentation is clear, the techniques novel, it has a strong formal grounding, and is well placed within the existing LfD literature. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Paper is essentially good in its current form. I was left wondering about the limitations and expressive potential of the author's chosen constraint language (the GR(1) fragment of LTL). How broadly does this apply to robotic missions in general as compared to say Signal Temporal Logic, or Probabilistic Signal Temporal Logic? Perhaps this could get a brief mention in the limitations section."}}, {"review_id": "2gAZu4a6iUx", "reviewer": "Reviewer_VK5r", "summary": "This work provides a temporal logic imitation method for robot operations, which can deal with both motion-level perturbations and task-level perturbations. In detail, a dynamic system is implemented along with collision boundaries for avoiding motion failure, and linear temporal logic is used for task-level failure detection and recovery. The experiments on real robots show the effectiveness of the proposed method, and the theoretical analysis of the dynamic system provides the guarantee of the G.A.S. of the DS and the satisfaction of the constraints (cuts).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work provides a robot skill imitation method with the ability to recover from failures and perturbations. The proposed method successfully combines DS and LTL and provides theoretical analysis on the guarantee of system reliability. The detained diagram that illustrates the principle of the proposed method is clear and intuitive, which helps the reader quickly understand the main idea of this work, and the video presentation is also very impressive.\n\nHowever, as maintained by the author, this method itself relies on well-designed sensor states for successfully classifying the mode, which would cause some problems when applying to more complex robot tasks. Also, the experiments are mainly on the soup-scooping task, further evaluation on more complex tasks is welcomed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work is well organized and easy to read, the provided video presentation and diagram can illustrate the proposed method clearly and intuitively. The experiments on the soup-scooping task are a little bit simple but impressive, where the analysis of the results is sufficient and can well support the main claim of this work.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "1. How to design and choose the binary sensor state in practice?", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (oral)