Surprise-Guided Search for Learning Task Specifications From Demonstrations
Authors: Marcell Vazquez-Chanlatte
Ameesh Shah
Gil Lederman
Sanjit A. Seshia
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Learning from Demonstrations
Specification Mining
Markov Decision Process
Inverse Reinforcement Learning
Formal Methods
Symbolic Learning

Abstract: This paper considers the problem of learning temporal task specifications, e.g. automata and temporal logic, from expert demonstrations. Task specifications are a class of sparse memory augmented rewards with explicit support for temporal and Boolean composition.  Three features make learning temporal task specifications difficult: (1) the (countably) infinite number of tasks under consideration, (2) an a-priori ignorance of what memory is needed to encode the task, and (3) the discrete solution space - typically addressed by (brute force) enumeration. To overcome these hurdles, we propose Demonstration Informed Specification Search (DISS): a family of algorithms requiring only black box access to (i) a maximum entropy planner and (ii) a task sampler from labeled examples. DISS works by alternating between (i) conjecturing labeled examples to make the provided demonstrations less surprising and (ii) sampling tasks consistent with the conjectured labeled examples. We provide a concrete implementation of DISS in the context of tasks described by Deterministic Finite Automata, and show that DISS is able to efficiently identify tasks from only one or two expert demonstrations.

Surprise-Guided Search for Learning Task Specifications from Demonstrations
Anonymous Author(s)
Affiliation
Address
email
Abstract
This paper considers the problem of learning temporal task specifications, e.g- automata and temporal logic, from expert demonstrations. Task specifications are a class of sparse memory augmented rewards with explicit support for temporal and Boolean composition. Three features make learning temporal task specifications difficult: (1) the (countably) infinite number of tasks under consideration; (2) an a-priori ignorance of what memory is needed to encode the task; and (3) the discrete solution space typically addressed by (brute force) enumeration. To overcome these hurdles, we propose Demonstration Informed Specification Search (DISS): a family of algorithms requiring only black box access to a maximum entropy planner and task sampler from labeled examples. DISS then works by alternating between conjecturing labeled examples to make the provided demonstrations less surprising and sampling tasks consistent with the conjectured labeled examples. We provide a concrete implementation of DISS in the context of tasks descrihed by Deterministic Finite Automata, and show that DISS is able to efficiently identify tasks from only one or two expert demonstrations.
3
7
11
13
15
Introduction
16
Expert demonstrations provide an accessible and expressive means to informally specify task, particularly in the context of human-robot interaction. this work, we study the problem of inferring, from demonstrations, tasks represented by formal task specifications, e.g., automata and temporal logic. The study of task specifications is motivated by their ability to (i) encode historical dependencies; (ii) incrementally refine the task via composition, and (iii) be semantically robust to changes in the workspace. However, learning such symbolic specifications is difficult, due to the often combinatorially large search space and lack of gradient-based feedback that can be leveraged. Prior works this space have used methods ranging from enumeration (20) to mutation- based sampling [II]. In order more efficiently search for explanatory task specifications, this work introduces a family of approximate algorithms, called Demonstration Informed Specification Search (DISS), that systematically reduces the problem of learning from demonstrations into series of
18
20
22
24
25
26
28
specification identification problems, e.g., finding a DFA that is consistent with set of example strings a problem more generally referred as Grammatical Inference 7
29
30 To ground our discussion, we introduce a running example.
31
1.1 Running Example Consider agent operating in environment with different regions,
32 which we present as a colored 8x8 discretized world for demonstrative purposes, shown in Fig The agent can attempt to move up, down, left, or right. With probability 1/2,, wind will push the agent down, regardless of the agent's ' action. The black path is the prefix of an episode, in which the agent attempts to move left, slips into blue region visits brown region and then proceeds
33
34
35
36 downward.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37 Given the black demonstration, Eb. and the prior knowledge that the agent's task implies that it will avoid red regions ). what task, represented as a Deterministic Finite Automaton (DFA), explains the agent's behavior?
38
39
40
41
Upon inspecting the demonstration, one might hypothesize that the complete path formed by extending & with the grey dashed lined to is a positive example of the task. Appealing to Occam's 's razor,
42
43
44
one might conjecture that the task was just to reach and avoid W. However, under this hypothesis and assuming a temporal discount, & quite surprising. For one, the detour to visit seems unjustified. Furthermore, why would the agent not take the red dashed path directly to
45
46
Figure 1
47
48
49
To remedy these concerns, one might conjecture that the agent's true task requires visiting after visiting thus explaining why the agent does not take the red dashed path. Similarly, the demonstration seems less surprising if one assumes that the agent needs avoid red regions
50
51
52
58 thus explaining why the agent does not take the light blue dotted path.
54 The result is the task represented by the DFA shown in Fig 2 We shall later systematize this line of reasoning and provide learner that recovers an explanatory DFA given demonstration.
55
56
Figure 2
57
The above example also illustrates a few features that motivate learning task specifications. First and foremost, task specifications directly encode the set of acceptable behaviors and model temporal dependencies. This stands in contrast to Markovian rewards where the encoded task is intimately tied to the details of the dynamics model such as the transition probabilities, time resolution, and workspace configuration [T]. Second, task specifications have well defined compositions, e.g., conjunction and sequential ordering, side stepping many classes of reward
58
59
80
61
62
63 bugs stemming from ad-hoc composition [21] 2). This avoids the need to fine tune learned tasks, which undereuts the original purpose of learning the task representations (13). Finally, learning task specifications from demonstrations enables learning classes of sparse rewards, a key primitive in feedsek reinforcement learning techniques such as hindsight experience replay (3).
64
85
66
1.2 Related Work The problem of learning objectives by observing an expert has rich and well developed literature dating back to early work on Inverse Optimal Control 110) and more recently via Inverse Reinforcement Learning (IRL) [15]. In IRL, an expert demonstrator optimizes an unknown reward function by acting in a stochastic environment. The goal of IRL is to find a reward function that explains the agent's behavior. A fruitful approach has been cast IRL as a Bayesian inference problem to predict the most probable reward function [16]. To make this inference robust to demonstration/moddlin noise, one commonly appeals to the principle of maximum causal entropy 99] 24]. Intuitively, this results in forecasting model that is no more committed to any given action than the data requires (formalized as bounding the worst-case expected description length of future demonstrations)
68
70
71
72
73
74
75
78
While powerful, traditional IRL provides no principled mechanism for composing the resulting reward artifacts and requires the relevant historical features (memory) to be a-priori known. Furthermore, has been observed that small changes in the workspace, e.g., moving goal location perturbing transition probabilities, can change the task encoded by a fixed reward [2]
78
80
81 To address these deficits, recent works have proposed learning Boolean task specifications, e.g. logic
82 or automata, which admit well defined compositions, explicitly encode temporal constraints, and have workspace independent semantics. The development of this literature mirrors the historical path
84 taken in reward based research, with works adapting optimal control 12 6). Bayesian [17] 23 and maximum entropy [22] IRL approaches.
85
86 key difficulty for the task specification inference from demonstrations literature is how to search an intractably large (often infinite) concept class. In particular, and in contrast to the reward setting, the discrete nature of automata and logic, combined with the assumed a-priori ignorance the relevant memory required to describe the task, makes existing gradient based approaches either intractable or inapplicable. Instead, current literature either (syntactically) enumerates concepts 21
87
88
90
or
91 hill climbs via simple probabilistic (syntactic) mutations 22 5].

---

92
1.3
Contributions Specific contributions of our work include:
93
1. A proxy function whose gradient (i) informs the search for an explanatory task specification; and (ii) is computed with black-box access to maximum entropy planner;
94
85
2. A reduction from learning specifications from demonstrations to learning from labeled examples;
96
A guided hill-climbing algorithm that is agnostic to the underlying task representation and dynamics model. For example, changing the task representation only requires providing a specification identification algorithm for that class of tasks. Examples include learning decision trees, DFAs, symbolic automata, etc., and An open-souree (MIT license) implementation of DISS for learning DFAs that we apply in an empirical setting from prior literature.
97
98
99
100
101
102
The choice of DFAS as the task representation for our experiments was motivated by two main observations. First, DFAs explicitly encode memory, making the contribution of identifying relevant memory more clear. Next, to our knowledge, all other techniques for learning finite path properties from demonstrations focus syntax defined concept classes. As a result, these existing techniques conflate search efficiency with their concept classes' inductive biases. On the other hand, DFAs constitute a very large and mostly unstructured concept class" which allow for learning without user-defined inductive biases.
103
104
105
108
107
108
Algorithm Overview Demonstration Informed Specification Search (DISS) operates by cycling between three components (shown in Fig 3).
1.4
110
Candidate Sampler
tusk: iPt
Surprisal Guided Sampler
112
1. Example Buffer: Given previous iterations, the example buffer yields set of positive and nega- tive example paths. Based on Simulated Anneal-
SSec 2
See
(Sed
114
ing [18].
select labeled examples Xeç1
labeled puth & c sP)))
116
2.
Candidate Sampler: Given a set of labeled ex- amples and an optional previous candidate task, the candidate sampler draws a candidate task that is consistent with set of labeled examples.
Example Bufler (See
118
119
Figure 3: Overview of DISS.
120
3. Surprisal Guided Sampler (SGS): Given task yo, the SGS algorithm samples a labeled path that is suspected to be mislabeled by y.
121
122
Paper Structure In the sequel, will formalize the problem statement and agent model based
129
on the extensive literature on maximum causal entropy agent models (Sec [ (ii) formulate an
approximate solution using simulated annealing (Sec3); (iii) derive proposal distribution which tries to find mislabeled paths; and (iv) provide empirical evidence for the efficacy of this algorithm for learning DFAS.
125
128
2
Preliminaries and Problem Statement
127
128
2.1 Dynamics Model We model the expert demonstrator as operating in a Markov Decision Process (MDP), M AS so, P), where (i) S denotes finite set of states, (ii) A(s) denotes the finite set of actions available at state 8 E S. (iii) 8p is initial state, and (iv) P(s' a, 8) is the probability of transitioning from s' when applying action E A(s). We will make additional assumptions about M. First, we assume a unique (always reachable) sink state, i.e P($ a, $) = 1, denoting "end of episode". Second, we shall assert the Luce choice axiom, which requires that each action, a E A(s), be distinct, i.e., no actions are interchangeable or redundant at a given state [14).
130
132
133
134
135 A path, E, is an alternating sequence of states and actions starting with so: & ==s) so 2g Any path, &. can be (non-uniquely) decomposed into a prefix, p. concatenated with suffix, o, denoted
24 Sn-
136
{ p.a. We allow a to be of length 0. The last state of & is denoted by last(E) del sn- A path complete if it contains $ exactly once, and thus last(E) $. We denote by Pathsę the set of all complete paths, and by Paths the set of all prefixes of Pathsg. i.c., paths that contain $ at most once.
137
138
139
'The number of DFAs grows super exponentially in the alphabet size and maximum number of states, e.g.. given an alphabet of size four, there already many many more than 100,000 DFAs with at most 4 states.
3

---

140
Task Specifications A task specification (or task), sp, is subset of paths equipped with a size function that measures its description complexity, i.e.,
2.2
141
Pathsg
size T R20,
142
where & is a task specifications, called a representation class. A labeled example is tuple, 2 = (g, 1).
143
corresponding to complete path and a binary label, E {0. 1). An example, (E,1), is consistent with task sp if = [s E spl. A collection of labeled examples, X ,,, In, is consistent with a task, ip, if they are all consistent.
144
145
146
Example 2.1. Our running example used DFAs over the alphabet E = representation class. For a DFA task yp a path, E, belongs to ip if the corresponding color sequence ends in an accepting (concentric circle) state. For instance, let &b and &, be the completed black and red paths shown in Fig I and define Xbg = {(Cb, (Er,0) The DFA shown in consistent with Kbg We take the size of yp be the number of bits encode the DFA using stuttering semantics, i.e. default self loops. The concrete encoding is provided by the DFA python library 4
as its
147
148
149
150
151
152
Finally, a candidate sampler (or identifier), is a map from labeled examples, X. and an optional reference task, &p E + to distribution over consistent tasks in o. The lack of task (either because no reference is provided or no task consistent, denoted 1. This distribution is denoted I( yp X).
153
154
155
2.3 Policies and Demonstrations A (history dependent) policy, л(a {), is a distribution over actions, a, given a path, 5, where a E A Allas((E)). A policy, n, is (p. -omccmett if the probability of satisfying yp using is p. i.e., Pr(g E yp n, M) = p. A demonstration, is a path, 5*, generated by M).
156
157
158
a employing a policy π in an MDP M.
Task Inference from Demonstrations Problem (TIDP): Let M, b, and P be a fixed MDP, representation class, and task prior, respectively. Further, let X² be (p",'"") "compmettt policy, RT, where ""pp,, and π are all unknown. Given a multi-set of ii.d. demonstrations, 5i.... Sin ~ (TT M), find:
go E arg max Pr(Ei,.... M) P( | M).
+E
159
By itself, the above formulation ill-posed as Pr(si, Sxp M, sp) left undefined. What remains is to derive suitable agent model and discuss how to manipulate likelihoods in this model.
160
161
Task Motivated Agents Following [22]. we propose using the principle of maximum causal entropy to assign a bias- maa-miiii belief of generating the demonstrations given a candidate task. Here maaimmiiig is taken to mean miniminizing the worst case prediction log-loss (24), i.c., the worst-case number of bits needed to encode the the actions of the agent.
2.4
162
163
185
We start by defining the causal entropy on arbitrary sequences of random variables. Let Nis X....., X, and Vı V1,..., Vd denote two sequences of random variables. The entropy of X1: causally conditioned on Vi: is:
der
166
187
H(a V1:) del Σ Di:e
188
where, H(X) | y) In })), denotes the entropy of X (statically) conditioned on
169 y. Intuitively, causal conditioning enforces that past variables do not condition on events in the future. This makes causal entropy particularly well suited for robust forecasting in sequential decision
170
making problems, as agents typically cannot observe the future (4
172 For MDPs, the unique policy, T. that maximizes entropy subject to a finite horizon and to being (p. competeen exponentially biases towards higher value actions: In <aa { _ Vx(E a) (E).))
174
The state and action values are recursively given by the following smoothed Bellman-backup 24]:
X ((()
if g E Pathss, if E E Paths \ Pathss.
(5) der LSE A (last()) VA(g.a. ES [V>(E. s') | a)
(4)
S if (g z ..) 8 A).
4

---

Here LSEA f(x) =hr In In ef(²) and X, called the rationality, is set such that Pr(g E ip TT TM p. Unfortunately, is typically not known. In such cases, the competency of the agent can be treated as hyper-parameter estimated empirically, e.g., Ps 1/m Di=ls E vp.. The former is useful when given on a few demonstrations and the latter is useful when given a large number of demonstrations. Finally, when À is induced from ip, we shall write Vp. and mip-
176
177
178
179
180
Explainability of a task The surprisal (or information content) of (i.id.) demonstrations, Sm is the negative log likelihood of the demonstrations under (st, M)::
Si....
m
. Sm n. M) set
Σ In Pr(Ei n. M)
(5)
182
Note that the likelihood of i.i.d., demonstrations from (TT, M) is simply exp(-h (m)). Given Sm. we define the task surprisal, 189 a fixed MDP, M, and a fixed collection of demonstrations,
184 9,, as:
h(p)) def h(Ef,-.- .......-pp. Em Tip M)
185
Solving a TIDP requires minimizing h plus the negative log prior, which can be taken as size(sp).
186
3
Example Buffer
187
Given our maximum causal entropy agent model, we employ simulated annealing to approximately solve the TIDP problem. At high level, Simulated Annealing (SA) (18) is probabilistic optimization method that seeks minimize an energy function U : Z + R U [oo]. To run SA, one requires three ingredients: (i) cooling schedule which determines a monotonically decreasing sequence of temperatures; (ii) a proposal (neighbor) distribution g(=' | 2); and (iii) reser schedule; which periodically sets the current state, 2,, to one of the lowest energy candidates seen so far.
188
189
190
191
192
198
A standard simulated annealing algorithm then operates as follows: (i) An initial 20 Z is selected; (ii) T. is selected based on the cooling schedule; (iii) A neighbor z' is sampled from g( 2)): (iv) z'
194
195
accepted (Z+411 f 2²) with probability:
Pr(accept| [=1,20) = min { (
{
if dU > 0 otherwise
196 where dU det U U(z)): (v) Finally, if reset is triggered, 2+11 is sampled from previous 197 candidates, e.g., uniform on the argmin.
198 For DISS, we will start by expressing the posterior distribution tasks in the form:
Pr(p E.....et S.m M)
c ((p))
(8)
199
where the energy, U, is given by:
U(p) th g size((p) + h(p),
(9)
200
and 0 E R determines the relative weight of the size. That is, we appeal to Occam's razor and assert that the task distribution is exponentially biased towards simpler tasks, where simplicity measured by the description length of the task, size((p), and the description length (i.e. surprisal) of ......e, under (T,, M).
201
203
204
Using the language of SA, we define DISS as follows: (i) 2 E Z is tuple, p), of labeled examples and a task specification; (ii) 2 (6. L): (iii) the proposal distribution, q(x', ,p X, sp) is defined to first sample a concept using an identification map, sp I(X), then run SGS (defined below) on sf to conjecture a labeled path &. yielding X" X U ( & w/)}. Next, drop examples from X" with probability Pampi (iv) resets occur every α E N time steps. If a reset is triggered, k is sampled from softmin;g:U((pp). and p sampled from I(. I, 1 This process of accepting, rejecting, and resetting defines the example buffer.
205
206
207
209
4 Surprise Guided Sampler (SGS)
212 The key innovation of DISS is defining a proposal distribution over labeled examples which constrains the concept sampler to find more and more explanatory (less surprising) tasks.
5

---

LSE
LSE
14
LSB
AVG
17
4/2
AVG
1.5
56
%
AVG
10
Figure 4: Prefix tree and computation graph with 12 nodes for the paths shown on the left.
214
We start by discussing the prefix tree of the demonstrations. As we shall see, the prefix tree will serve as mechanism to reason about the various paths not taken.
215
Let f..... Sin be multi-set of demonstrations (paths) and denote by T = (N, E) the prefix tree of the .i.... Sm where N and are the prefixes (nodes) and edges of T. respectively. Each node E N, corresponds to prefix of at least one of the demonstrations. Given two prefixes p. E N, p' is descendent of if p' = p- An edge connects parent p to child p' if p' is the one action (or state) extension of Pt- For each edge, (p,p') ) F., define the edge traversal count, # (p..))) the number of demonstrations, 8* such that 5* p' y. A node, p, is said to be ego node if its prefix ends in a state, i.e. last(p) S. node that is not an ego node is called an environment (env) node. A path, &, pivots at node p if is the longest prefix of & in N. The pivot actions (and pivot states) 224 of a node, P. are the set of available actions (states) that result in pivoting at p. i.e.,
217
219
222
Ap def {a p. E Paths \N}
Sp del {s p.s Paths \, N}.
(10)
Example 4.1. Consider the MDP shown in Fig 4 with two paths E and ES shown as a green dashed and black solid line resp. The prefix tree of {ET {2} is shown on in the middle. For convenience an index is associated with each node (prefix). There is path that pivots every node except node p2, since both possibilities (slipping/not slipping) appear in the demonstrations yielding Sp = 0.
226
227
228
229
Next, observe that because weighted averaging and LSE are commutative, one can aggregate the values of a set of actions or set of states (environment actions). This motivates defining the pivot value of node p as:
230
231
[SE Vo(p.a) (..VV(p.s) s) p. M, $ Spl if i env,
if ego,
VP del
(11)
We shall denote by Ve E R. the node-indexed vector of pivot values associated with task ço under our maximum entropy agent model. We note two properties of pivot values. First, pivot values strictly increase as the language of task specification is made larger:
233
234
Proposition 4.2 (Pivot values respect subsets). Let be a complete path that pivots node i. If V C sb and & E & \ sp, then Vf- V..
236
237
Proof. Follows inductively from the monotonicity of E, Σ, and In.
238
Second, using the soft Bellman backup one sees that the pivot values entirely determine the values, V, of the prefixes of the demonstrations (an example is shown the right of Fig 4p. Namely,
let (V) denote the derived value at node i in the prefix tree, and let Pr(i ~h k V) denote
241 the probability of transitioning from node to node l under the (local) policy: -,() This
motivates defining the surprisal of the local policy induced by the pivot values as follows:
243 Let T (N, F.) be a prefix tree of demonstrations, ...... - The pivot surprisal of given T is
244 map, h: Rd + R, where is the number of nodes that can be pivots and:
i i(V) def
Hus Pr(i ~ VV.
(12)
(L,E)E
Importantly, the task surprisal factors through the pivot surprisal, i.c., h(()) h(V9).

---

246
4.1 Pivot surprisal gradients Motivated by the question "Given candidate task, what counter- factuals still require explanation?", we ask related question: "How could the pivot values change to make the demonstrations more likely?" For example, for ego nodes, one might want to make the value of the observed actions large and the pivot value small. The result would be an agent with no
247
249
250
incentive to pivot. Unfortunately, changing pivot value changes the policy in a non-local way, e.g..
251
changing V9 in Fig also changes the policy for nodes 9, 8, 2, and 1. Fortunately, 4ropp.33shon shows that upstream effects are easily summarized by the gradient of h, with a proof provided in the appendix.
252
Proposition 4.3 (Vh determined by local policy). Letting Pay P denote the probability of starting at node x and pivoting at y:
Th 0V
Σ #(.0) (Va - PyA P2)
(L)E à ego
(13)
Pay VV) Pr(x y/ V) (1-
E (M.A)ER
Pr(y ~ 2 1V) P1())
illustrates that gradients are simple to compute given only access to the policy on the prefix tree. Further, focusing on any given edge one observe that (13) captures the trade-off between (i) making the actions taken more optimal by decreasing the value of other actions; (ii)
Note that Prop 4.3
254
255
256
making the actions taken less risky by increasing the value of possible outcomes.
257
Mislabeled counter-factuals Because of the expressivity of representation classes like DFAs, there is concern that globally optimizing h will overfit to the demonstrations and ignore the prior distribution. Thus, our goal is not to simulate gradient descent under Vh, but to instead help identify counter-factual paths that require explanation. Props 4.2 and
258
259
4.3 yields the following observation.
261
Let & be a complete path with pivot P such that: E +p
O. I is a likely path under Te dh is large in absolute value, then (and thus has large effect on V) and the pivot surprisal gradient 87,
283
E may be mislabeled by y. For example, if the gradient pivot p positive, the surprisal and can be decreased by removing path, & from the candidate task sp.
254
Using these insights propose surprise guided sam- pling (Alg which samples a path to relabel based on (i) how likely it is under Tip and (ii) the magni- tude and sign of the gradient at the corresponding pivot. Combined with an identification algorithm, I, repeated applications of yields an infinite (and stochastic) sequence of tasks resulting from incre- mentally conjecturing mis-labeled paths.
Algorithm 1: Surprise Guided Sampler
286
Input: y., M,
Compute The given M and T.
268
Let D = softmax,
(-
1 ah
8 04
270
Return p ~ 1) and { (Tp, Af) S.L.
271
i { pivots at i.
272
ii E & +
0h 0. 04.
Importantly, note that Alg only requires black box maximum entropy (MaxEnt) planner to enable assigning edge probabilities, Pr(i ~4 V), and sampling suffixes given a pivot. If the satisfaction
ii 3 E + s.t. o' is consistent with:
274
XU p))
276
277
probability of an action is also known, i.e., (r
278
ç E 9p g, M, mp), then one can more efficiently sample suffixes using Baye's rule.
5
Experiments
279
280
In this section, we illustrate the effectiveness of DISS by having it search for a ground truth specifica- tion, represented as a DFA, given the expert demonstrations
281
the workspace from our motivating example (shown in Fig D. This inference problem is derived from previous benchmarks proposed
282
283
in [1 22]. As we expand on below, the key difference is that the representation classes in the prior work are much smaller, i.e., of sizes 930 and 14 tasks respectively. Here we shall work with all DFAS
285
over the alphabet, Σ = (L,
D). As previously discussed, this representation class grows super exponentially, e.g., there many more than 100,000 DFAs with at most 4 states.
7

---

287
To start, denote the (dotted) green path in Fig goes directly to by Eg. Simiarly, denote the (solid) black path by & that immediately slips into E. visits m. then proceeds towards This path is incomplete, with possible extension, ob. shown as dotted line. The ground truth task is the right DFA in Fig 2
286
290
We consider two TIDP instances which vary the representation class and the provided demonstrations. These variants respectively illustrate that (i) the full specification can be learned given unlabeled complete demonstrations; and (ii) our method can be used to incrementally learn specifications from unlabeled incomplete demonstrations. In particular, we will show that DISS supports incorporating prior knowledge about the task, e.g., rules learned from natural language or prior demonstrations.
291
292
294
295
296
1. Monolithic: Eg and Eo.or are provided as (unlabeled) complete demonstrations. The representation class is MinDFA.
297
296
&6 is provided as an (unlabeled) incomplete demonstration. The representation class, R. is variant of MinDFA that incorporates prior knowledge. Let yo denote the three state DFA for avoiding and reaching M. If a task, ip in R, then { ] S concept(p)) S concept(y'). That is, prior knowledge is provided that you must reach you must avoid and you know two positive examples. The size of sp is given by: size(y) = size' (5p) size' (y²). where size' the size function for the MinDFA representation class.
Incremental:
299
300
302
303
304
For both experiments, the size weight, 0, was set 1/50. For reference, the ground truth task uses about RV 40 nats.
305
Candidate Sampler. To implement I, we adapted existing SAT-based DFA identification algorithm [19] to enumerate the first 20 consistent DFAs, To make I(p' | (p, X) respect the size prior on DFA, we sampled DFA from the enumerated DFAs, exponentially weighted by the number of bits needed to describe o' given yp. That is, the sampling was weighted by the change in the number of states and the introduction/removall of labeled edges. Further details our implementation of
307
309
311
I. along with a discussion of other component implementations and their hyperparameters, can be found in the Appendix.
312
5.1 Baselines. As mentioned in the introduction, existing techniques for learning specifications from demonstrations use various syntactic concept classes, each with their own inductive biases. Thus, we implemented two DFA-adapted baselines that act as proxies for the enumerative [2] 17
314
315
316
23 and probabilistic hill climbing style [12 5) algorithms of existing work:
317
1. Prior Guided Enumeration. This baseline uses the same SAT-based DFA identification algorithm to enumerate DFAS in ordered by the size prior. This is done by finding the N smallest DFAs in lexicographic order (node then edges) as above and then ordering by size. N = 80 in the monolithic experiment and N = 40 in the incremental experiment. As an alternative to DISS's competency assumption, we allow the enumerative baseline to restrict the search to task specifications that accept the provided demonstrations.
318
319
320
322
323
2.
Random Pivot DISS. As mentioned above, will evaluate DISS various SGS temperatures, one of which has 3 = Xc. This results in a (labeled example) mutation based search with access to the same class of mutations as DISS, but samples pivots uniformly at random, i.e., gradient based bias. Note that this ablation still samples suffixes conditioned on the sign of the gradient, and thus the mutations are still partially informed by the surprisal.
324
326
327
5.2 Results and Analysis To simplify our analysis, we present time in iterations, i.e., number of sampled DFAs, rather than wall clock time. This is for three reasons. First, for each algorithm, the wall clock-time was dominated by synthesizing maximum entropy planners for each unique DFA discovered, but the choice of planner is ultimately an implementation detai Second, because many DISS iterations correspond to the same DFAs (due to resets and rejections) the enumeration baseline explored significantly more unique DFAS than DISS (a similar effect occurs with the random pivot baseline, since the different pivots give more diverse example sets). This results in the baselines spending more time planning than DISS, thus increasing their time per iterations. Third, the enumeration baseline first enumerates DFAs in lexicographic order (without planning) and then computes the energies in order of increasing size. This incurs a significant (ss (55) overhead. Thus, using wall clock-time would further skew the results below in DISS's 's favor.
329
331
332
333
334
335
336
337
338
e For reference the used for this experiment planner took around 4-10s per DFA.
8

---

speetnatte Morolitec
STATHTES
100
52
-25
-26
-00
-102
(b) Most probable DFA found by DISS in the monolithic experi- ment.
(a) DISS finds explanatory DFAs much faster than baselines.
Search efficiency Fig 5a shows the minimum energy of DFA discovered by iteration for the monolithic experiment. For space, the same plot for the incremental experiment is provided in the appendix. To reduce variance, experiments, DISS was able to significantly outperform the enumeration baseline (recall that energy the negative log of the probability) and tended to degrade in its search efficiency as / increased. For example, in the incremental setting, I < -5 typically required only 1-2 iterations (compared the 13 iterations of enumeration)! For reference, the benchmark this experiment was based on (21) used a syntactic variant of the incremental representation class and evaluated 172 (out of 930) tasks.
340
take the median of 5 runs for each B. We see that for both
342
343
344
346
The key takeaways are that: (I) DISS is significantly more (cycle) efficient at finding explanatory DFAs than prior based enumeration; (2) Relying on the surprisal gradient (by decreasing the pivot temperature) enables efficient exploration in large concept classes; (3) Using stronger inductive bias such as asserting partial knowledge of the specification increases the search efficiency of DISS; DISS can be effective even with few incomplete and unlabeled examples.
348
350
352
Diversity of DFAS
In addition to finding the most probable DFAs much faster than the baselines, DISS also found more high probability DFAs. The most probable DFA found by DISS for monolothic experiment shown in Fig 5b] The incremental variant is similar, with an additional edge to the sink failure state enforced by the prior knowledge. We observe that for both experiments, DISS is able to leann that if the agent visits M. it needs to visit before W. Nevertheless, our learned DFAS differ from ground truth, particularly when comes the acceptance of strings after visiting m. We note that a large reason for this is that our domain and planning horizon make the left most effectively act as a sink state. That is, the resulting sequences are effectively indistinguishable with many even having the exact same energy. In Fig 5b] we make such edges lighter, and note that the remainder of
354
356
357
358
359
380
361
the DFAs show good agreement with the ground truth. This limitation standard in learning from
demonstrations.
6
Conclusion
384
This paper considers the problem of learning history dependent task specifications, e.g. automata and temporal logic, from expert demonstrations. We empirically demonstrate how to efficiently explore intractably large concept classes such as deterministic finite automata to find probable task specifications. The proposed family of algorithms, Demonstration Informed Specification Search (DIS)) requires only black box access to (i) a Maximum Entropy planner; and (ii) an algorithm for identifying concepts, e.g., automata, from labeled examples. While we showed concrete examples for the efficacy of this approach, several future research directions remain. First and foremost, research into faster and model-free approximations of maximum entropy planners would enable a much larger range of applications and domains. Similarly, while large, the demonstrated concept class was over small number of pre-defined atomic predicates. Future work thus includes generalizing to large symbolic alphabets and studying more expressive specification formalisms such as register automata, dosh automata, and (synchronous) products of automata.
365
386
387
368
369
370
372
374

---

References
378
[1]
David Abel, Will Dabney, Anna Harutyunyan, Mark K. Ho, Michael L. Littman, Doina Precup, and Satinder Singh. "On the Expressivity of Markov Reward". In: NeurlPS. 2021.
378
[2] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. "Concrete problems in Al safety". In: arXiv preprint 1r0xiv:/06..6630 (2016).
380
381
[3] Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. "Hindsight experience replay". In: Advances in neural information processing systems. 2017, pp. 5048-5058.
382
383
384
[4]
Anonymous, Anonymous. Version 1. May 2022.
385
[5] Estefany Carrillo. "Controller synthesis and Formal Behavior Inference in Autonomous Sys- tems". PhD thesis. University of Maryland, College Park, MD, USA, 2021.
387
[6] Glen Chou, Necmiye Ozay, and Dmitry Berenson. "Explaining Multi-stage Tasks by Learning Temporal Logic Formulas from Suboptimal Demonstrations". In: Robotics: Science and Systems. 2020.
389
Colin De la Higuera. Grammatical inference: learning automata and grammars. Cambridge University Press, 2010.
391
8
Marijn Heule and Sicco Verwer. "Exact DFA Identification Using SAT Solvers". In: ICGI. Vol. 6339. Lecture Notes in Computer Science. Springer, 2010, pp. 66-79.
393
394
99]
Edwin Jaynes. "Information theory and statistical mechanies". In: Physical review 106.4 (1957), P. 620.
385
396
[10]
Rudolf Emil Kalman. "When is a linear control system optimal". In: (1964).
397
[11]
Daniel Kasenberg and Matthias Scheutz. "Interpretable Apprenticeship Learning with Tempo- ral Logic Specifications". In: arXiv preprint arXiv: 1710. 10532 (2017).
388
[12]
Daniel Kasenberg and Matthias Scheutz. "Interpretable apprenticeship learning with temporal logic specifications". In: CDC. IEEE, 2017, pp. 4914-4921.
400
[13]
Michael L. Littman, Ufuk Topcu, Jie Fu, Charles Isbell, Min Wen, and James MacGlashan. Task Specificcations via GLTL". In: 17X4v:1704.04341 [cs] (Apr. 2017). arXiv: 1704.04341. URL: httt:://:::::i//// org/ abs/1704 04341
402
wwwiironnent-Indeeeeddee
404
[14]
R Duncan Luce. "Individual choice behavior." In: (1959).
[15]
Andrew Y. Ng and Stuart J. Russell. "Algorithms for Inverse Reinforcement Learning.. In:
406
ICML. Morgan Kaufmann, 2000, pp. 663-670.
407
[16]
Deepak Ramachandran and Eyal Amir. "Bayesian inverse reinforcement learning". In: IJCAI
408
(2007).
409
[17]
Ankit Shah. Pritish Kamath, Julie A. Shah, and Shen Li. "Bayesian Inference of Temporal Task Specifications from Demonstrations", In: NeurIPS. 2018, pp. 3808-3811
410
411
[18]
Christopher C. Skiscim and Bruce L. Golden. "Optimization by simulated annealing: A preliminary computational study for the TSP". In: WSC. ACM, 1983, pp. 523-535.
413
[19]
Vladimir Ulyantsev, Ilya Zakirzyanov, and Anatoly Shalyto. "BFS-Based Symmetry Breaking Predicates for DFA Identification". In: LATA. Vol. 8977. Lecture Notes in Computer Science. Springer, 2015, pp. 611-622.
415
[0
Marcell Vazquez-Chanlatee Jyotirmoy V Deshmukh, Xiaoqing Jin, and Sanjit A Seshia. "Log- ical clustering and learning for time-series data". In: International Conference on Computer Aided Verification. Springer. 2017, pp. 305-325.
417
419
[21]
Marcell Vazquez-Chanlatte, Susmit Jha, Ashish Tiwari, Mark K. Ho, and Sanjit A. Seshia, "Learning Task Specifications from Demonstrations". In: NeurIPS. 2018 pp. 5372-5382.
420
421
[22]
Marcell Vahnuez-Chanlatte and Sanjit A. Seshia. "Maximum Causal Entropy Specification Inference from Demonstrations" In: CAV (2). Vol. 12225. Lecture Notes in Computer Science. Springer, 2020, pp. 255-278.
422
423
424
[23]
Hansol Yoon and Sriram Sankaranarayanan. "Predictive Runtime Monitoring for Mobile Robots using Lagic-Based Bayesian Intent Inference". In: ICRA. IEEF, 2021, pp. 8565-8571.
425
426
[24]
Brian D Ziebart, Andrew Bagnell, and Anind K Dey. "Modeling interaction via the principle of maximum causal entropy". In: (2010).
10

---

Checklist
428
429
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope? [Yes]
431
432
(b) Did you descrihe the limitations of your work? [Yes] See end of experiment analysis.
433
(c) Did you discuss any potential negative societal impacts of your work? [No] The work discussed inherits the societal impacts of general imitation learning, and in particular inverse reinforcement learning. One benefit is the potential to generate auditable and verifiable artifacts. As such, compared to existing work, we thing the societal impact is comparatively positive.
434
436
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
439
440
2. If you are including theoretical results...
411
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
442
(b) Did you include complete proofs of all theoretical results? [Yes] Complete proofs are provided in the appendix (supplemental material).
444
3. If you ran experiments..
445
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] Provided in supplemental material.
447
448
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
449
450
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] We report the median over five runs and several runs with similar parameters.
452
453
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [No] The results were given in terms of iterations and the experiments were designed to be runnable on a laptop without a GPU.
454
455
456
4. If you using existing assets (e.g., code, data, models) or curating/releusing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] All code was MIT licensed.
460
(c) Did you include any new assets either in the supplemental material as a URL? [Yes]
461
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
464
465
5. If you used crowdsourcing or conducted research with human subjects...
466
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
467
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
469
470
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
11


Reviews:
[{"review_id": "wm-flttTvmM", "reviewer": "Reviewer_Y88L", "summary": "Continuing the tradition of making the powerful goal modeling language LTL amendable to end-user usability, this paper gives an efficient approach to solve for LTL formulas (expressed using a DFA) from expert demonstrations. The crux of their algorithm is incorporating in the fact that the optimal planner is an maximum-entropy planner, using this knowledge to derive an algorithm more efficient than an enumerator agnostic to this fact.\n", "questions": "line53 : where is the light blue dotted path?\n\n", "limitations": "it is discussed well", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## str\n\nThis paper shows a promising inference algorithm that is efficient in converting expert-demonstration trajectories into DFA specifications over an MDP environment. Most importantly, it does so by considering fewer numbers of DFAs than prior algorithms. The crux of their algorithm is incorporating in the fact that the optimal planner is an maximum-entropy planner, and leveraging this information to derive an algorithm that is more efficient than enumerating a DFA and using the maximum-entropy planner afterwards to check.\n\n## weakness\n\n### additional baselines are required\n\nThis task can be formulated as a straight-forward program synthesis task, and one should seriously consider standard neural-symbolic approaches on it. \nHere are few suggestions:\n1. For instance, in this simulated environment, one can easily generate paired data of the form (Env, DFA, Traj_from_max_ent_planner) and use supervised learning to learn a mapping from (Env,Traj_from_max_ent_planner)->DFA. One can use hugginface API on a pre-trained language model (gpt-neo) and fine tune it over some structured/textual representation of these abstract objects. Then at inference time simply use the top-k sampling to generate DFA from this fine-tuned language model, and see if it works\n2. You can also develop a PCFG capable of generating DFA, then, using a learned neural network, predict the probability distributions of this PCFG, then sample top-k DFA from this grammar, and see if it works\n\nI wouldn't be surprised there will be some very strong patterns in the datapoints (Env, DFA, Traj_from_max_ent_planner) so that a learned neural model can pick it up, and can generate some fairly good (low energy) DFAs. And you won't even have to worry about having a train/test data split since the space of DFA (like you said) is very large, so the likelihood of encountering the same DFA from training to testing is unlikely.\n\n### motivation can be better\n\nSynthesizing a DFA in this manner is very different from synthesizing a DFA for (for instance) regular expression matching. The key difference lies in the evaluation of the DFA, whereas in regex synthesis it is trivial to check if it is consistent with a set of accept/reject input strings, to validate if a DFA is good for specifying a goal, we need to check whether a maximum-entropy planner would produce similar trajectories as the ones given by the expert. This is important and sets this task apart from many other program synthesis tasks, where checking the validity of a program is cheap.\n\n### few writing changes\n\nrunning example 1.1 is a good attempt (I'm happy to see ML papers are adopting more of a PL paper convention with a running example section), but is still confusing. I still do not know what is the DSL that governs the \"goal space\" of the robot. For instance, I can infer the robot's trajectory as simply \"reach the yellow square as soon as possible, but do so while using as many horizontal movements as vertical movements, while not changing directions too much\" <-- is this in the goal space? probably not, as DFA cannot do counting, but how would I know that? In general conter-factuals are easier to describe if the reader knows of the space of hypothesis beforehand. Instead example 1.1 did not explain what this space is, and the reader is lost. Figure 1 has no caption, and Figure 2 has no caption. I have to guess that figure 2 is in fact the solution that explains the behaviour of the robot, i.e. it cannot visit yellow directly after visiting blue, or it will be in the \"death\" state. A short sentence describe the overall hypothesis space, something like \"The goal space of the robot is formulated as a DFA, where edges denote traversing through multiple same-colored blocks (i.e. a blue means visiting multiple blues in succession)...\". . . Okay I just took a glance at reference [21], \"learning task specification from demonstrations\". They explained the problem so much better. The agent moved in the way it moved because it accidentally touched water (due to wind), so it has to dry itself (brown) before touching yellow (electricity), or else it'll be zapped. That is a wonderful story and made much more sense. I don't know why you took their example and removed the most reader-friendly affordance (fire, zap, water, drying) from it. Having to go read that [21] personally only to find out an easier explanation was already there is frustrating at best.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WQyuFFxMIcO", "reviewer": "Reviewer_GoLB", "summary": "The paper investigates the interesting problem of learning formal task specifications (i.e., a DFA) from (expert) demonstrations in a MDP. The main motivation is that symbolic structures (DFAs in this case) offer better properties (e.g., non-Markovian goals, goal composition, less sensitive to perturbations in the environment, etc.) compared to previous \"inverse control\" reward-based approaches like IRL. The main difficulty is the large search space of DFAs (even for the small toy example considered in the paper). The paper proposes the DISS (demonstration informed specification search) as an approximate solution to learning DFAs from demonstration. The key idea is to construct a structured search space over labeled examples and tasks. A knowledge-injected hill-climbing algorithm can operate efficiently in this search space to uncover the task that best \"explains\" the demonstrations. The key algorithmic contribution in DISS is the \"surprise-guided sampler\" which constructs labeled examples. Experiments are conducted on a gridworld and show the proposed method outperforming 2 baselines. \n\n----------\n\nUPDATE: I thank the authors for their detailed responses and being open to suggestions. I generally agree with Reviewer Y88L's comments, especially regarding the experimental aspects (baselines, user studies) and overall clarity of the paper. The authors have addressed most reviewer feedback, incorporated a number of reviewer comments into the paper and have included some more empirical data in the Appendix. I'm more positively inclined towards the paper although I think it remains difficult to assess impact with the current set of experiments. I've revised my score upwards to incorporate all the new information.", "questions": "  1. What's the definition of $\\varphi(\\xi)$? (I'm assuming the same as [21].)\n  2. What's the correctness / optimality of the final DFAs wrt to ground truth?\n  3. Are there any non-grid-world applications or domains in which these methods can be evaluated?\n  4. What's the computational complexity of DISS? How does it compare with the baselines?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n  + The problem of learning formal representations of goals directly from expert trajectories (without reward functions) is challenging and important. The proposed approach seems to extend the state of the art in this area. The primary contribution of the paper is algorithmic with the introduction of the DISS algorithm for identifying specifications and the SGS component for selecting labeled examples.\n  + The algorithmic ideas seem rigorously developed and novel (although I'm not sure I fully understood the details in Section 4.).\n  + Experiments show that the method does perform better than reasonably strong baselines (enumeration and mutation-based).\n\nWeaknesses\n  - I found the paper to be a bit hard to read (especially compared to the closely related work in [21], which was much clearer). This paper assumes or omits important definitions (e.g., $\\varphi(\\xi)$, LSE). Combined with the large amount of new notation, interchangeable terminology (e.g., specification vs task) and typos, I found it challenging to grasp the key ideas on my first reading. Overall, I'd suggest this paper be edited to be more self-contained and clearer.\n  - The experiments are somewhat limited. I was looking forward to a deeper analysis into the learned DFAs and in particular their correctness. For example, how \"close to optimal\" are the DFAs (compared to an optimal DFA)? The DFA for a single example is somewhat analyzed but a more detailed \"error\" analysis would strengthen the paper.\n  - Given the use of grid worlds in this paper and previous works [21, 22] and the above concerns, the overall impact of the paper is somewhat unclear to me. (That said, I'm open to revising my score based on the feedback and other reviewers.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5XM7zZ0Z1lG", "reviewer": "Reviewer_yBeH", "summary": "The authors introduce a method, Demonstration Informed Specification Search (DISS), for learning DFAs from examples. Their method consists of simulated annealing over DFAs and labeled examples simultaneously, with a very complex proposal distribution they call the \"surprise guided sampler\" (SGS) that involves among other things defining an analogue of \"gradient\" and incrementally conjecturing paths likely to be mislabeled by candidate DFAs. They compare against a SAT-based algorithm and a more random ablation of DISS on a toy synthetic grid-world domain and find their method performs best.", "questions": "Why is this an important problem? Also, could the kernel of the main SGS idea be demonstrated more clearly in a more abstract setting?", "limitations": "Yes", "rating": 3, "confidence": 1, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method introduced is very complicated and the toy results are extremely meager. This ratio is highly unfavorable. In general, the paper seems unmotivated. The paper is also highly symbolic and more generally does not seem relevant to the NeurIPS community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject