Chain of Thought Imitation with Procedure Cloning
Authors: Sherry Yang
Dale Schuurmans
Pieter Abbeel
Ofir Nachum
Venue: NeurIPS 2022 Accept (2022)
Keywords: Imitation Learning
behavioral cloning
generalization
sequence modeling

Abstract: Imitation learning aims to extract high-performance policies from logged demonstrations of expert behavior. It is common to frame imitation learning as a supervised learning problem in which one fits a function approximator to the input-output mapping exhibited by the logged demonstrations (input observations to output actions). While the framing of imitation learning as a supervised input-output learning problem allows for applicability in a wide variety of settings, it is also an overly simplistic view of the problem in situations where the expert demonstrations provide much richer insight into expert behavior. For example, applications such as path navigation, robot manipulation, and strategy games acquire expert demonstrations via planning, search, or some other multi-step algorithm, revealing not just the output action to be imitated but also the procedure for how to determine this action. While these intermediate computations may use tools not available to the agent during inference (e.g., environment simulators), they are nevertheless informative as a way to explain an expert’s mapping of state to actions. To properly leverage expert procedure information without relying on the privileged tools the expert may have used to perform the procedure, we propose procedure cloning, which applies supervised sequence prediction to imitate the complete series of expert computations. This way, procedure cloning learns not only what to do (i.e., the output action), but how and why to do it (i.e., the procedure). Through empirical analysis on navigation, simulated robotic manipulation, and game-playing environments, we show that imitating the intermediate computations of an expert’s behavior enables procedure cloning to learn policies exhibiting significant generalization to unseen environment configurations, including those configurations for which running the expert’s procedure directly is infeasible.

Chain of Thought Imitation with Procedure Cloning
Mengjiao Yang Google Brain, UC Berkeley
Dale Schuurmans
Google Brain, Unversity of Alberta
sherryydgoogle.com
schurrmans@google.com
Pieter Abbeel
Ofir Nachum
UC Berkeley pabbeel@es berkeley..eeeee..............ee. edu
Google Brain of irrachum@googgee.coo com
Abstract
Imitation learning aims to extract high-performance policies from logged demon- strations of expert behavior. is common frame imitation learning super- vised learning problem in which one fits function approximator to the input-output mapping exhibited by the logged demonstrations (input observations to output actions). While the framing of imitation learning supervised input-output learning problem allows for applicability overly simplistic view of the problem in situations where the expert demonstrations provide much richer insight into expert behavior. For example, applications such as path navigation, robot manipulation, and strategy games acquire expert demonstra- tions via planning, search, or some other multi-step algorithm, revealing not just the output action to be imitated but also the procedure for how to determine this action. While these intermediate computations may use tools not available to the agent during inference (e.g., environment simulators), they nevertheless informative as a way to explain an expert's 's mapping of state to actions. To properly leverage expert procedure information without relying the privileged tools the expert may have used to perform the procedure, we propose procedure cloning, which applies supervised sequence prediction imitate the series of expert computations. This way, procedure cloning learns not only what to do (i.e., the output action), but how and why do it (i.e., the procedure). Through empirical analysis on navigation, simulated robotic manipulation, and game-playing environments, we show that imitating the intermediate computations of an expert's behavior enables procedure cloning to learn policies exhibiting significant generalization to unseen environment configurations, including those configurations for which running the expert's 's procedure directly is infeasible.'
wide variety of settings, is also an
1 Introduction
The idea of learning by imitation in autonomous agents closely resembles how humans (especially children) learn real life - by watching and mimicking how someone else performs a certain task [1] While humans are able to generalize exceedingly well from small number of demonstrations, today's 's amitation-leaman autonomous agents often struggle in situations that only slightly differ from the demonstrations, e.g., opening door in different shape or color [2]. One explanation for such difference that while humans imitate, we understand the task at high level as opposed to only remembering mapping from images actions [3], and indeed, generalization failures also occur
'https://github ctttaau..ooomoooggeee con//gogle--eseeee reweereseeraaee shaseaarchhgoolle-eessrr http://wwww./..eeeeeeeeeee///ena
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Datasets
Training
Behavioral cloning: State
TCC
Action
State
Chain of thought:
of ((x 19-1
Procedure cloning: State
TPC
Action
Inference
Procedure e.g., search,
State
BC
planning
PC
Action R (right)
Figure I: Visualization of the dataset collection, training, and inference of BC and PC on a maze navigation task, During dataset collection, the expert uses a search procedure to determine the optimal action to generate a path to the goal location (red star). During training. BC discards these intermediate search outputs and learns to map states to actions directly. In contrast, PC learns the complete sequence of intermediate computations (i.e., branches and backtracks) associated with the search procedure. During inference, PC generates a sequence of intermediate search outcomes emulating the search procedure on new test map before outputting the final action.
in humans when there is a lack of understanding of the underlying reasons for the behavior, such as solving complex math problem. As result, students are told to "lear principles, not formulas" and "understand, do not memorize" to encourage better generalization - to be able to solve problems that are similar but different from the ones taught in lectures [4, 5, 6]. Just students taught the step-by st--bypstep derivations of a math problem during a lecture, is it possible to have an equivalent "chain of thought" supervision for training autonomous agents?
While solving complex math derivations may not be the predominant application of today's imitation- learned autonomous agents, the tasks they are commonly applied to 10---- e.g., path navigation, robot manipulation, and strategy games often do employ chain-of-thought reasoning procedures in the form of planning. search, or other multi-step algorithms when collecting expert data [7, 8, 9]. Since such multi-step algorithms application-speciin and therefore lack common representation which can be systematically characterized, it is typical for imitation learning to assume access to only logged demonstrations of state-action pairs, leaving out the much richer insight into expert behavior provided by the algorithm's 's intermediate computations. In addition, the planning or search procedures used by the expert may rely on tools not available to the agent during inference (e.g., environment simulators), so how these intermediate computations should be used to facilitate imitation learning is not clear.
In this work, we formulate chain of thought imitation as an extension of traditional imitation learning. Different from the traditional imitation learning setup where an agent only has access to expert state-cction pairs as demonstrations, chain of thought imitation also has access to the intermedi- ate computations that generated the expert state-action pairs in the training data (Figure 1). We then propose procedure cloning (PC), an alternative to behavioral cloning (BC) [10], which applies supervised sequence prediction to imitate the complete series of of expert computations before outputting an expert action (Figure 1). Procedure cloning learns a policy by maximizing the likeli- hood of the joint distribution of procedure observations and expert actions, which can be modeled autoregressively using transformer-Iie like architecture [11]. During inference, a procedure cloned policy autoregressively generates procedure observations from given input state, mimicking the computations of a search or planning algorithm before outputting the final action, thus avoiding any reliance on privileged tools or information used by the expert's procedure. From modeling perspective. procedure cloning can employ a more expressive model (c.g., transformer) trained on more data (i.e., procedure observations), which leads better generalization according to the new "scaling law" of large language models [12]. Intuitively, procedure cloning learns not only what to do (i.e., the output action), but how and why do (i.c., the procedure), which further resembles how humans learn complex tasks.
We demonstrate how to conduct procedure cloning and leverage the generalization ability of procedure cloned agents on variety of path navigation, robotic manipulation, and strategy game tasks. For example, in path navigation an expert trajectory may be determined by using a BFS search algorithm on an annotated map (e.g., x. y coordinates of obstacles), which are expensive obtain [13]. We

---

show that procedure cloning agent can successfully learn to imitate BFS on a previously unseen test map without requiring additional annotations, achieving 100% test accuracy while BC agent completely fails to navigate (0% accuracy) when the maze layout changes. Similarly, in an image- based robotic manipulation task, we observe that BC quickly overfits to the set of training images, whereas procedure cloning learns to predict the intermediate computation outcomes of a scripting policy, thus generalizing much better and achieving a success metric of 83.9% compared to 78.2% from the previous state-of-the-arr [14]. Finally, in strategy games expert trajectories are collected by running MCTS [15], which requires access to a simulator and can be extremely slow [16]. We show that procedure cloning can effectively learn from the path traversed by MCTS collected in a deterministic environment to then successfully generalize in a zero-shot manner to stochastic environments and more difficult game settings, where running MCTS, even if given access to simulator, performs poorly.
2 Related Work
Generalization in sequential decision making, Learning decision making policies with good generalization properties has a long-standing history in bandit [17, 18, 19], imitation learning [20], and reinforcement learning (RL) [21, 22. 23, 24] settings in the form of regret or Probably Approximately Correct (PAC) bound analysis. These studies are concerned with generalization in continual learning setup without explicit separation of training and testing stages. We are more interested in an agent's ability to generalize to separate "test" environment whose configuration is different from the training environment, which is often what happens when an agent is deployed to the real world after being trained in a simulator. With aighccapccity neural network parametrized policies being the norm in training image-based agents, the risk of overfitting to the training environment is nontrivial [25, 26]. Various works have injected stochasticity into the training process including using stochastic policies [27], random starts [28, 29], sticky actions [30], and frame skipping [31] as a means to prevent overfitting to the specific environment dynamics experienced by the agent during training. A variety of image-based regularization techniques have also been applied to deep RL including dropout and (-2 regularization [32], data augmentation and batch normalization [33], domain randomization [34], and network randomization [35]. Instead of improving generalization through regularization or data augmentation like in existing work, we instead ask the question of whether learning the sequence of computations opposed to only the final expert action during training can help an agent generalize better.
Access to additional task information Many previous works in imitation learning and RL have observed that access to additional task information can help an agent learn better policies. For instance, [36] relics on access to the simulator for collecting more expert trajectories to reduce the quadratic error in task horizon to linear. [37] assumees that expert demonstrations contain both high-level and low-level trajectories, and learns a hierarchical policy explicitly. While our procedure observations can appear to be high-level labels sub-goals similar [37, 38), we do not make assumptions about the structure of procedure observations, which can simply be scalar variable values computed during program execution. There is also large body of literature that assumes access to a suboptimal offline dataset in addition to the expert demonstration data collected from the same environment, and conduct representation learning [39, 40, 41, 42, 43], hierarchical skill extraction [44, 45, 46], or dynamics model learning [47, 48, 49] on the suboptimal offline data followed by imitation learning from an expert. These works commonly assume good coverage in the offline data, which requires large amounts of state-action pairs being collected from running additional policies in the environment. We instead propose observing additional information from only running the expert policy, which requires far fewer state-action pairs being collected for training. Another line of existing work in the direction of multi-task learning suggests that learning an auxiliary task enhances imitation learning or RL [50, 51, 2, 52, 2, 53, 54]. For instance, [55] showed that predicting internal features of the emulator (e.g., enemy on the screen) is beneficial, [56] found predicting scene depth helps navigation, and [57] found predicting explanations helps relational tasks with causal structure. Chain of thought imitation differs from multi-lask learning in that procedures directly influence the action outcomes through computations, in which case learning the procedure directly (as opposed to treating it as an auxiliary objective similar to existing work) is more beneficial. The graphical model in Figure visualizes this distinction.
Chain of thought sequence modeling. The idea of decomposing multi-step problems into interme- diate steps (the so-called chain of thought [58]) and learning the intermediate steps using a sequence model has been applied to domain specific problems such as program induction [59], learning to
3

---

solve math problems [60], learning to execute [61], learning to reason [62, 63, 64, 65, 66, 67], and language model prompting [58]. The chain of thought imitation learning problem we formulate is domain agnostic and applicable to many sequential decision making task traditionally solved by imitation learning a Markovian setting such as robot locomotion, navigation, manipulation, and strategy games. Unlike language-basee tasks. problems in decision making have only recently started being explored by language models [68, 69, 70, 71, 72, 73], as the Markovian nature of these problems brings the value of sequence modeling into question. Our work contributes to bridging the gap between learning memoryless policies in Markovian environments and the intuition that large sequence models should help in reasoning based decision making.
3 Preliminaries
In this section, we define relevant notations and introduce the problem of imitation learning. We also review behavioral cloning (BC), which our work aims to improve.
MDP notations. Consider a Markov decision process (MDP) [74] M := (5, H. 7) consisting of state space S, an action space A, a reward function R S A + R. transition function T S x A + ASS)?. an initial state distribution μ E ASS), and a discount factor initial state so ~ J.. At each timestep t 2 O, an action ar π (s;)) is sampled and applied to the environment, and the environment transitions into the next state St+1 T(s,, at) while producing a scalar reward R(),,a)). The state visitation distribution d' induced by a policy π is defined as d' (s) ==
Y E [0, 1). A policy π. : S - (A) interacts with the environment starting
(1 (1-7) Pr [sp = M)). We use (s, a) d= to denote d a ~ T(S).
Imitation learning. In the standard imitation learning setting, R, T, and ş are unknown to the agent. Learning solely takes place
fixed dataset of expert state-action pairs D. = {(s(9) a()) >}; collected by expert policy π. by sampling s( d and (t)) (s()). The goal of imitation learning is to train a policy π : S - ((A) on D., such that ι closely approximates т. according to some metric such as the Hullback-Leibler (KL) divergence of their state visitation distributions Pxi((d= (s) Idd (s)). Regardless of the choice of this metric, it is computed over states including those outside of the training data D., and requires an agent generalize over unseen states to achieve good test performance.
Behavioral cloning (BC). The nature of learning from an expert dataset D. leads to the common framing of imitation learning as supervised problem of learning state to action mappings. Under this framing, behavioral cloning (BC) [10] proposes to solve the imitation learning problem by learning π that minimizes
Jnc(m) == ((e.a) .x)) (a|s))
using samples from the training data (s.a) D.. Equation 1 can also be viewed as the cross entropy loss for multi-class classification of state to action mappings. Agents trained with this vanilla BC objective (i.e., direct max- mix-likel on the training dataset D.) can quickly fail to generalize to unseen states when the expert data size is small compared to the state-action space of the environment [36, 75], which is often the case when the states are based on images. Image-based inputs also pose the challenge of learning the right invariances for the agent to generalize from training to testing environments with different configurations (e.g., maze layouts and environment stochasticity).
4 Procedure cloning
In this section, we first observe that in many imitation learning situations. expert demonstrations can provide much richer insight into the desirable behavior than just the final optimal action. We formulate learning under these situations as chain of thought imitation, where an agent can learn from not just the optimal action but the "thought process" an expert goes through before arriving at the final decision. We then formalize such thought process as procedures, and propose procedure cloning for learning such thought process through supervised sequence prediction.
4.1 Chain of thought imitation
Depending on the form of the expert policy π. used generate the training data D., an agent potentially has access to rich set of information about a task which can facilitate learning. For
2 (2)) denotes the simplex over set X.

---

Vanilla BC
Auxiliary BC
Procedure cloning
XO
Autoregressive
Conditional independence
Figure 2: Graphical models of vanilla BC, auxiliary BC, and procedure cloning with autoregressive and conditionally independent factorization. Node s represents an input MDP state, represents an expert action, and x represents the sequence of procedure observations (Jo...., FL)
instance, when is some scripting policy following fixed set of rules [76]. the rules (e.g.. "if close to enemy then fire") reveal causal information between firing and the enemy disappearing. In other settings when is search algorithm, the induced search tree exposes the path for finding the optimal action. When n, is multi-step hand-coded algorithm, the breakdown of the steps (e.g.. "first move to objects then sweep") reveals important ordering information about the task. Even in the case where n, is a human demonstrator, we can ask human to explain thought process that led to their decision. We refer to learning from the procedures (e.g.. planning, search, multi-step algorithm) that generated the final action as chain of thought imitation.
4.2 Procedures and procedure observations
To formulate chain of thought imitation learning problem, define procedure II S ((A) as a sequence of computations (Il, II...., II, IT++1) 1) that first transform an input state E S into some computation state (c.g., a variable value) using 1I : - R followed by repeatedly applying each subprocedure, 1L, Rd + Fd, Ve [1, L]. to the computation state before mapping the last computation state back to the MDP action space using II Rd - (A)) in sequential order. A procedure can be broken down to subprocedures at any user-defined granularity (e.g., function or for loop), depending on how frequently the computation state is retrieved for procedure learning. Each pair of training data, (8, a) D., is acquired through executing such procedure, i.e., n, II.
We define procedure observations as the sequence of computation states captured from procedure: IT (s()), VZ E [1, L... The training data with procedure observations is denoted Dn {(s(), x00 a {
(No, ... E (LL+1)xd where xo (s)s)) and ry П, [
j Note that we do not assume any procedure or procedure observation is available at test time, hence we still
need learn policy ı that only takes as input.
4.3 Procedure cloning
With the procedure observations defined above, we are now ready for learning procedures through procedure cloning (PC). We model a PC policy by estimating the joint distribution of the procedure observations and the final action conditioned on the input state p(a, x|s), which we can factorize autoregressively as:
p(a, x|s) plla x) IT/=e(( 8) p(zols).
Under this factorization, estimating p(a, x|s) reduces estimating each conditional factor, which be parametrized using a transformer model [11]. The autoregressive factorization is highly flexible, but if the amount of expert demonstrations is small, and each procedure observation z; only depends on the previous procedure observation æ;-1 (i.e., the computation states are fully observed), and the final action only depends the last computation state, conditionally independent factorization can be more desirable. In other words, autoregressive models need more data to train, so the conditional independence factorization below preferable if the procedure information available fully captures the computation state:
p(a, x|s = p(a)æ1) D/LIP(ze(De-)) p(zols).
(3)
The graphical models of the two factorizations of PC policies are shown in Figure To learn a policy, maximize the empirical likelihood of the joint distribution in Equation 2 on given samples
5

---

Du = ((s xx(1) (( ){=)
min Joc 0,) = E B(xxaa) log x|s]] 6,0,0
(4)
(e(x log qu (a|x, 8)
Σ 1=1
log po (xe) s) log p& (xols)
(5)
Connection to BC with auxiliary tasks. The PC objective in Equation 5 can be reduced to the vanilla BC objective in Equation 1 by discarding the second and third term inside the expectation of Equation 5 and setting go (a]xx,))) s) π (als). We note that several previous works [51, 50, 2, 52, 55, 56] have used procedure information learning p(a, a. x|s) under the assumption that a and x are independent conditioned on s: p(a, = v [a]s))p(({ss) s). Procedure cloning instead focuses on situations where such a conditional independence assumption does not hold (i.e., a is directly computed by the procedure represented by x), and in these situations, as we will show in our experiments, treating the procedure information as a precursor to can perform better than using it as an auxiliary task. The graphical models of vanilla BC and BC with auxiliary task objective are shown in Figure 2.
auxiliary tasks BC, which may be interpreted as
Proof of concept: Synthetic maze navigation
In this section, we study a tabular maze navigation task with synthetically generated maze layouts (see Figure 3). We describe the task setup, followed by how to extract procedure data from a breadth- first search (BFS) path planning algorithm to train a procedure cloning agent, and empirically show that procedure cloning indeed generalizes much better to unseen maze layouts than other BC baselines. While this proof of concept illustration might seem domain-specific to BFS in mazes, we will see in Section 6.3 that procedure cloning can be applied to many types of search or multi-step algorithms.
Task description and evaluation protocol. We use a gridworld maze environment in which an agent seeks to navigate to a goal location in a maze from a random starting location using 4 discrete actions including up (U), down (D), left (L), and right (R). The input to the agent is a multi-channel "image" with the maze wall, goal location, and agent location encoded in separate channels. The maze layout is algorithmically generated with random internal walls that form tunnel-shaped map (see example maze in Figure 4 and details in Appendix A.1). We generate set of mazes Sq C S and split Sn into disjoint training Sgain and testing Si sets. We then generate expert trajectories by running BFS on only the training set of mazes SQ Strain At test time, the agent is evaluated on Steat by
their success rate of navigating to the goal. Figure 3 visualizes the data and training pipeline.
Procedure data collection. BFS is common path planning algorithm for navigation [77, 78, 79, 80], which we use to generate expert trajectories for training an imitation learning agent. To compute the optimal action at each time step, BFS keeps track of a visited 2D array (colored cells in Figure 3) that marks whether each position (1) has been visited by the search, (2) if so, which action visited it, and (3) has a position been backtracked. We simply take snapshot of the entire visi visited array as procedure observations zt every time BFS expands the search perimeter, resulting in a series of procedure data x (#1, (.1...., Lr.) as shown in Figure 3. Io is the identity map and ar = 8. See pseudocode for collecting procedure observations in Appendix A.2.
Procedure learning. Since each of the visited 2D arrays ze only depends on the previous visited array zı-1. and the final visited array after backtracking uniquely identifies the expert action on its own, we choose the conditionally independent factorization of p(a, x(s) (Equation 3) described in Section 4.3. Specifically, we parametrize po( A E [1, T.] using deep con- volutional neural network that takes in the current visited array xz-1 as input and produces the next visi ted array ze as output. We optimize 0 using the cross-entropy loss between the predicted and true next visited array. Since the procedure observations xa and the original input s are in the same image space, (xr) shares the same parameters as pa (zz- During inference when new test maze layout is given, we apply do- = o(s) and äe ~ po (-=-( repeatedly until an array år. is predicted for which the entry in á corresponding to the agent's 's current location is labelled as "backtracked", and we return the backtracked action as the final output action.
Generalization results. We compare procedure cloning to applying data augmentation with random crop, translation, and zoom (Aug BC) or auxiliary objective of predicting the visited array (Aux BC) to vanilla BC. BC policies are parametrized with convolutional neural networks (CNN) and multi-layer perceptrons (MLPs) (see hyperparameters in Appendix A.5), Aux BC receives the same

---

BFS procedure execution
Datasets
Procedure cloning
1y (1,49)
R
State
R
R
R
R
R
R
State
Run BFS expand (koep track of actions to each call).
[TL
pule Wdle ligh wiggle putey
T1:
X
Action
R
Action R
Figure 3: In a discrete maze, the expert employs BFS by first expanding a search perimeter until it encounters the goal cell, at which point it backtracks to find the optimal action at the starting state (cells in light blue visited and dark blue are backtracked). We encode this algorithm sequence of procedure observations (T...... z6) of the intermediate computation states, with each I; represented by a 2D array and each cell of the array containing BFS-relevant information (i.e., whether this cell is being expanded or backtracked and the action recorded when expanding to this cell). Procedure cloning is trained to predict the entire sequence of computations from input state to output action using sequential model p(a|zL) I[ 1)) p(zuls).
Aug
Train maze
Test maze
Discrete
16x11, #Trall
3lsc232x32.. #Trall
#Traj5
Ant13x13, #Traj5.
1.0
R
0.5
AntMaze
0.0
10
20
40
5
20 Train mags
10
40
10
20
40
10
20
40
Train maps
Train maps
Troin maps
Figure 4: [Left] Visualization of the discrete maze (4 discrete actions) and AntMaze (8 continuous actions). [Right] Average success rate of PC and BC agents navigating the goal from random start locations over 10 test mazes. Agents are trained on 5, 10, 20, 40 mazes of and 5 expert trajectories on discrete maze and AntMaze, respectively. We find that procedure cloning leads much better test. maze generalization compared to alternative approaches.
information PC when computing the auxiliary loss, i.e., the visitation maps are given both PC and Aux BC. Figure (Disc. 16 x 16 and Disc.32 x 32) shows the average success rate (over trajectories) of reaching the goal from random start locations on 10 test maze layouts unseen during training. Procedure cloning successfully generalizes to test mazes, whereas vanilla BC completely fails learn (0% sucess rate) in bigger maze 32 32. Aux BC and Aug BC help the smaller maze but not in the bigger maze. The poor test performance of BC is due to generalization failure as opposed insufficient model capacity BC's success rate the training mazes are close to 100% (see Figure 10 in Appendix B.2). We also include additional experiments using VIN [8] as baseline in Appendix B.1, showing that even an approach specialized to the structure gridworld tasks exhibits poor generalization.
Experiments
We now evaluate procedure cloning in larger scale settings on tasks including simulated robotic navigation [82] and manipulation [83, 14], and learning to play MinAtar [84] miniature version of Atari [85]). Procedure cloning exhibits significant generalization to previously unseen maze layouts, positions of objects being manipulated, and environment configurations such as transition stochasticity and game difficulty in each of the tasks, respectively. See Appendix B for more results.
6.1 Evaluating continuous robot navigation in AntMaze
Task description. Following the discrete maze navigation task in Section 5, we now consider more realistic setting of mimicking real-world robotic navigation. We adopt the AntMaze environment from D4RL [82], where an 8-DoF "Ant" quadruped robot with continuous state and action spaces is placed in a 2D maze environment. The original task designed for offline RL only has one maze layout which not revealed to the agent (the agent only sees its current position and joint-based state measurements). To adopt the task for evaluating generalization across maze layouts, we also pass the algorithmically generated maze layouts from Section the agent as inputs. During evaluation, the agent needs to navigate the goal in previously unseen maze layouts.
7

---

BC
Bimanual sweep
Data size
Data sice 100
Data size 1000
PC
BC
Aun BC
Aug mC
Succcsss Metric
Seccos Metric
Soccone Metrii
Success Metric
Results with early stopping
Training step
Lock 204 Training step
Wel sels Gol
Ful pee, Training stop
Dataset
10
size (# expert demos)
109
1000
Figure 5: [Left] Visualization of the bimanual sweep task. [Middle] Average success metric (propor- tion of particles in bowls at the end of the episode) of PC and BC agents completing the bimanual sweeping task after learning on 10, 100, 1000 expert trajectories; each variant is an aggregate of 10 runs. All of our algorithm implementations use the implicit loss function described in [14] for this task. [Right] When using 1000 expert demonstrations with early stopping, PC achieves 83.9% compared to 78.2% success of the existing state-of- tfate-of-the-- ff-thhe-rt art achieved by implicit BC.
PC implementation. The procedure that generated the expert training trajectories for goal- reaching in AntMaze involves a low-level PID controller that is good for navigating to local locations close to the robot and a high-level waypoint generator which uses BFS search to find the next local waypoint [82]. We apply procedure cloning to the high-level waypoint generator following the same steps as in discrete maze described in Section 5. The predicted waypoint is then passed to Gaussian parametrized policy (optimized with max- max-likelihood) together with the agent's joint measurements. For BC and variants, we use a convolutional neural network to embed the maze layout into fixed dimensional vector and concatenate the robot joint measurements together with the agent and goal locations to a Gaussian policy.
Generalization results. Figure (Ant x 10 and Ant x 13) shows the average (over trajectories) success rate of the robot reaching the goal from random starting locations on 10 test mazes. Procedure cloning consistently provides significant benefits over vanilla BC, whereas data augmentation on the maze layout and predicting the visited array as an auxiliary loss are less helpful. We know the poor test performance of BC (and other baseline methods) is more due to generalization failure as opposed to insufficient model capacity because the success rate the training mazes between procedure cloning and auxiliary BC are similar (Figure 11 in Appendix B.3).
6.2 Evaluating image-based robot manipulation
Task description. The bimanual sweeping task [83, 14] requires two 7-DoF robot arms equipped with spatula- like end-effectors to sweep pile of particles evenly into two bowls while avoiding dropping particles between the tips of the spatulas. The scripted oracle for collecting expert trajectories uses access to privileged information including object poses and contact points, which are not accessible at test time. Rather, only high-resolution (96 x 96) images in conjunction with end-effector positions and orientations are given to the imitation-learned agent during inference. Actions end-effector positions for each robot arm (6 dimensions for each arm for total of 12 dimensions for the action); for how to incorporate kinematic actions into this environment, we refer the reader to [83]. Random seeds determining the initial position of the particles are partitioned so that test-time configurations are seen in training.
PC implementation and results. The scripted policy first scoops the particles by moving the spatulas to their computed geometric center and then moves the spatulas to bowl before releasing the particles. At each state of a trajectory, we collect the Cartesian coordinates of one of these two goal points geometric center of the particles or position of the bowl - depending on the expert's behavior mode at that state, and use these as procedure observations. We supervise the PC agent to first predict these coordinates, then predict the actions conditioned on the predicted coordinates and end-effector positions and orientations. For both BC and PC we parameterize log- legelikll using energy-base based models, also known as an implicit loss, which is state-of-th of-the-art for this task [14]. We compare PC to BC trained directly on image inputs and auxiliary BC which learns to predict oracle coordinates as an auxiliary objective. We see that the BC variants quickly overfit to the training set of observations, whereas PC generalizes much better (Figure 5, left). Despite early stopping (taking the maximum evaluation success rate over all training steps), PC still outperforms all of the BC variants (Figure 5, right), improving significantly over the sfate-of-the-art.
8

---

Game envs
MCTS Procedure execution
Datasets
Procedure cloning
MGTS root
State
Ayc
Polals) pylti Xas, q,tal
State
Goal
Action
Action (left)
R
x = [Goal, R.
GPT
State
Goal
R
Action (left)
Goal
Figure 6: In the MinAtar game-playing environment, the expert uses MCTS (Llo,... ..., 1l,,) find an optimal future trajectory [L. R, Goal]. We treat this future trajectory in reverse order [Goal, R, L] as procedure observations, so that procedure cloning is trained to first predict the goal image (MCTS leaf node) and then predict the optimal action sequence backwards from the goal using a GPT-like autoregressive model, ultimately predicting the expert's 's output action as its last prediction.
Asx
Ang
MCTS
Antecto, Sticky
Benakest, Sticky
Frewway, Slicky
Seaqueei, Sticky
Specelorddara, Sticky
Antecia, Bamp
Sauqueai Ramp,
Spacelmnaders Rump
Trale
swuds
sedds
Train sends
Tralis needs
Train meds
Figure 7: Average episode reward (over 50 episodes) of PC and BC agents playing MinAtar games over 3 test environments using sticky actions (left) and game difficulty ramping (right) not see in the training environments.
6.3 Evaluating strategy games in MinAtar
Task description. MinAtar is a minature version of the Atari Arcade Learning Environment [85] consisting 5 games with simplified 10 x 10 multi-channel images as inputs. To generate the expert trajectories for training, run a AlphaZero-style [15] Monte-Carlo tree search (MCTS) algorithm on the deterministic version of the environments to collect expert trajectories (see Appendix A.3 for details). During evaluation, we test the initation-learned agents on set of test environments with different seeds from the training environments where expert trajectories are collected. To further evaluate generalization, we apply sticky actions with probability 0.1 and game difficulty ramping to the test environments where such an option is available.
PC implementation. In contrast running BFS in maze navigation where the visited array cleanly captures the entire state of the search, running MCTS in MinAtar is more convoluted, involving a number of MCTS simulation each with selection, expansion, rollouts, and backtrack steps and different tree structures, making capturing the full search state difficult. Fortunately, procedure cloning with autoregressive factorization (Equation 2) allows procedure data be partial observations of the computation state, and so we elect to use only a subset of the MCTS computation states as supervision for PC. Namely, simulation run (from the final search tree) and the goal image at the tree leaf as procedure observations (highlighted in blue in Figure 6); this is effectively the optimal future trajectory determined by MCTS. A PC policy is trained to use the input state to first predict the goal image using CNN and then use the goal as input to an autoregressive action sequence model p(л, Neć, s) where I, the optimal action that is f-steps away from the goal (i.e., predicting the optimal action sequence backwards from the goal). Figure illustrates the data and training pipeline of procedure cloning.
record the optimal action sequence after the last MCTS
Generalization results. Figure 7 shows the average reward (over 50 episodes) collected by running PC and BC policies in 3 test environments with different environment seeds than the environments used to collect training trajectories. Figure 7 (left) evaluates generalization to stochastic environments, where we found sticky actions caused MCTS (gray) to struggle to search for good actions without extensive tuning. Figure 7 (right) evaluates generalization to more difficult game settings (available in out of MinAtar games). Autoregressive PC policy generalizes the best across all games and all settings.

---

7 Conclusion
We have identified a major gap between the imitation learning problem setup and how training data for imitation are often obtained in practice. Specifically, applications that planning, search, or other multi-step algorithms reveal the procedure for determining an expert action addition to outputting the expert action itself. This led to formulate chain of thought imitation learning, where an agent is also given access to the intermediate computations that generated the expert state-action pairs. In order generalize test environments where intermediate computation is not available, propose procedure cloning, which learns to predict the intermediate computation outcomes step-by- step using a sequence model, and emulates the intermediate computations during inference through autoregressive generation. Evaluation on a variety of navigation, manipulation, and game play tasks shows that procedure cloning generalizes much better test environments of different configurations than alternative improvements over behavioral cloning. We hope the success of applying sequence modeling to imitation learning achieved by procedure cloning opens up new research directions in the intersection of large-scale sequence modeling and sequential decision making.
Limitations. One major limitation of procedure cloning compared to traditional BC is in the computational overhead, since PC needs to predict intermediate procedures. Furthermore, the choice of how to encode the expert's s algorithm into a form amenable to PC is up to the practitioner. While we have presented ways to encode variety of policies here (BFS, MCTS, scripted robotic policies), applying PC to other domains may require some amount of trial-and-error in designing the ideal computation sequence for PC.
Acknowledgments and Disclosure of Funding
Thanks to Hanjun Dai, Kimin Lee, and Olivia Watkins for reviewing draft versions of this manuscript. Thanks to Hanjun Dai for discussions on the MinAtar task. Thanks to Pete Florence, Andy Zeng, and Jake Varley for assistance with the bimanual sweep task, and Justin Fu and Aviral Kumar for assistance with the AntMaze task.
References
[1] Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6):233-242, 1999.
[2] Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvu- nakool, János Kramár, Raia Hadsell, Nando de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint
2018.
[3] Peter Jarvis. Towards comprehensive theory of human learning. Routledge, 2012.
[4] George Katona. Organizing and memorizing: studies in the psychology of learning and teaching. 1940.
[5] Michael Crawford. Teaching contextually. Research, rationale, and techniques for improving student motivation and achievement in mathematics and science. Texas: Cord, 2001.
[6] Heidi L. Lujan and Stephen E. DiCarlo. Too much teaching, not enough learning: what is the solution? Advances in physiology education, 30(1): 2006.
[7] James Bruce and Manuela M Veloso. Real-time randomized path planning for robot navigation. In Robot soccer world cup, pages 288-295. Springer, 2002.
[8] Richard M Murray, Zexiang Li, and S Shankar Sastry. A mathematical introduction to robotic manipulation. CRC press, 2017.
[9] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[10] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Technical report, CARNEGIE-MELLONN UNIV PITTSBURGH PA ARTIFICIAL INTELLIGENCE AND PSYCHOLOGY... .... 1989.
10

---

[11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[12] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:2001.08661 2020.
[13] Charles Thorpe and Jay Gowdy. Annotated maps for autonomous land vehicles. In 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings, pages IEEE, 1990.
[14] Pete Florence, Corey Lynch, Andy Zeng. Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158-168. PMLR, 2022.
[15] David Silver, Thomas Hubert, Julian Schrittwieser, Joannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint 1rXiv:1712.01815. 2017.
[16] Tristan Cazenave and Nicolas Jouandeau. A parallel monte-carlo tree search algorithm. In International Conference on Computers and Games, pages 72-80. Springer, 2008.
[17] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256 2002.
[18] Sébastien Bubeck and Nicolo Cesa-Bianchi Regret analysis of stochastic and nonstochastic multi-armed bandit problems. arXiv preprint 1rX1v 2012.
[19] Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Muchine Learning, pages 1638-1646. PMLR, 2014.
[20] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668. JMLR Workshop and Conference Proceedings, 2010.
[21] Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21, 2008.
[22] Alexander L Strehl, Lihong Li, and Michael L Littman. Reinforcement learning in finite mdps: Pac analysis. Journal of Machine Learning Research, 10(11), 2009.
[23] Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
[24] Mohammad Gheshlaghi Azar, Ian Oshand, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272.
PMLR, 2017.
[25] Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. arXiv preprint arXiv: (00v: 1804.0689, 2018.
[26] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference machine learning, pages 0048-2056. PMLR, 2020.
[27] Matthew Hausknecht and Peter Stone. The impact of determinism on learning atari 2600 games. In Workshops at the Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
[28] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 51897540):59-553, 2015.
11

---

[29] Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Pannecrshelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning, arXiv preprint arXiv: 1507.04296, 2015.
[30] Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61: 61:523-562, 2018.
[31] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint (.06,06600.015444 2016.
[32] Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv: 1810.00133, 2018.
[33] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1282-1289. PMLR, 2019.
[34] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[35] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: simple technique for generalization in deep reinforcement learning. arXiv preprint arXiv: 1910.559, 2019.
[36] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[37] Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé III. Hierarchical imitation and reinforcement learning. In International conference machine learning, pages 2917-2926 PMLR, 2018.
[38] Michael H Lim, Andy Zeng, Brian Ichter, Maryam Bandari, Erwin Coumans, Claire Tomlin, Stefan Schaal, and Aleksandra Faust. Multi-task learning with sequence-conditioned transporter networks. arXiv preprint arXiv:2109.07578. 2021.
[39] Sanjeev Arora, Simon Du, Sham Kakade, Yuping Luo, and Nikunj Saunshi. Provable represen- tation learning for imitation learning via bi-level optimization. In International Conference on Machine Learning, pages 367-376. PMLR, 2020.
[40] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
[41] Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. In International Conference Machine Learning, pages 1 PMLR, 2021.
[42] Ofir Nachum and Mengjiao Yang. Provable representation learning for imitation with contrastive fourier features. Advances Neural Information Processing Systems, 34, 2021.
[43] Mengjiao Yang, Sergey Levine, and Ofir Nachum. Trail: Near-optimal imitation learning with suboptimal data. arXiv preprint arXiv:2/ 14770, 2021.
[44] Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzalez, Edward Grefenstette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pages 3418-3428, PMLR, 2019.
12

---

[45] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Of- fline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 13611, 2020.
[46] Kourosh Hakhamaneshi, Ruihan Zhao, Albert Zhan, Pieter Abbeel, and Michael Laskin. Hi- erarchical few-shot imitation with skill transition models. arXiv preprint arXiv:2107.0898/ 2021.
[47] Arun Venkatraman, Martial Hebert, and Andrew Bagnell. Improving multi-step prediction of learned time series models. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
[48] Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems, 34, 2021.
[49] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Visual adversarial imitation learning using variational models. Advances in Neural Information Processing Systems, 34, 2021.
[50] Long-Ji Lin and Tom M Mitchell. Memory approaches to reinforcement learning in non- Markovian domains. Citeseer, 1992.
[51] Xiujun Li, Lihong Li, Jianfeng Gao, Xiaodong He, Jianshu Chen, Li Deng, and Ji He. Recurrent reinforcement learning: hybrid approach. arXiv preprint arXiv:1509.0311 00004, 2015.
[52] Max Jaderberg. Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.005 1.0597,, 2016.
[53] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, s(47))eab, 2020.
[54] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference Robot Learning, pages 66-75. PMLR, 2020.
[55] Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence. 2017.
[56] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.0367. 2016
[57] Andrew K Lampinen, Nicholas A Roy, Ishita Dasgupta, Stephanie CY Chan, Allison C Tam, James L McClelland, Chen Yan, Adam Santoro, Neil C Rabinowitz, Jane X Wang, et al. Tell me why.-explanntiins support learning of relational and causal structure. arXiv preprint arXiv:2/ 12.03753, 2021.
[58] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[59] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by ratio- nale generation: Learning to solve and explain algebraic word problems. arXiv preprint 170.::1705.04146 2017.
[60] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ 10. 14168, 2021.
[61] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 2021.
13

---

[62] Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. arXiv preprint arXiv:2002.0588 0556677 2020.
[63] Mohammed Saeed, Naser Ahmadi, Preslav Nakov, and Paolo Papotti. Rulebert: Teaching soft rules to pre-trained language models. arXiv preprint arXiv:2 13006, 2021.
[64] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv: 1906.02361, 2019.
[65] Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Teaching pre-trained models to systematically reason over implicit knowledge. 2020.
[66] Zhengzhong Liang, Steven Bethard, and Mihai Surdeanu. Explainable multi-hop verbal reason- ing through internal monologue. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1225- 1250, 2021.
[67] Eric Zelikman, Yuhuai Wu, and Noah D Goodman. Star: Bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203. 14465, 2022.
[68] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayaku- mar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, pages 7487-7498. PMLR, 2020.
[69] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas. and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
[70] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
[71] Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint 1xXiv:2111.10364 2021.
[72] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint arXiv:2202.05607. 2022.
[73] Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? arXiv preprint arXiv:2201.12122 2022.
[74] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
[75] Stephen Tu, Alexander Robey, and Nikolai Matni. Closing the closed-loop distribution shift in safe imitation learning. arXiv preprini arXiv:2102.09161. 2021.
[76] Steven Rabin. Game Al pro: collected wisdom of game Al professionals. CRC Press, 2013.
[77] Don Murray and James Little. Using real-time stereo vision for mobile robot navigation. autonomous robols, 1(2):161-171, 2000.
[78] Wael Sabra, Matthew Khouzam, Arnaud Chanu, and Sylvain Martel. Use of 3d potential field and an enhanced breadth-fire search algorithms for the path planning of microdevices propelled in the cardiovascular system. In 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, pages
IEEE, 2006.
[79] Soumabha Bhowmick, Abhishek Pant, Jayanta Mukherjee, and Alok Kanti Deb. novel floor segmentation algorithm for mobile robot navigation. In 2015 Fifth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG), pages 1-4. IEEE, 2015.
14

---

[80] Hrudaya Kumar Tripathy, Sushruta Mishra, Hiren Kumar Thakkar, and Deepak Rai. Care: collision-awart aware mobile robot navigation grid environment using improved breadth first search. Computers & Electrical Engineering, 94: 107327, 2021.
[81] Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks Advances in neural information processing systems, 29, 2016.
[82] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
[83] Aditya Ganapathi, Pete Florence, Jake Varley, Kaylee Burns, Ken Goldberg, and Andy Zeng. Implicit kinematic policies: Unifying joint and cartesian action spaces in end-to-end robot learning. arXiv preprint arXiv:2203.0193 2022.
[84] Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint 19Xiv:9003003770 2019.
[85] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-277 279, 2013.
[86] Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pages 282-293. Springer, 2006.
[87] Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, et al. Acme: research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.0099 2020.
[88] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[89] Scott Reed and Nando De Freitas. Neural programmerrinterrreeerr arXiv preprint 1r1iv:1511.066 2015.
[90] Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3795-3802. IEEE, 201 8.
[91] Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing xystems, 32, 2019.
[92] Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit roinforcement without interaction at scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420, IEEE, 2020.
[93] Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal plan- ning networks: Learning generalizable representations for visuomotor control. International Conference on Machine Learning, pages 4732-4741. PMLR, 2018.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
15

---

Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See last paragraph in Section 6.3.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See Appendix A and supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See all plots Section 6.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 6.
(b) Did you mention the license of the assets? [Yes] The licenses are Apache License 2.0 and GNU general public license.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "Q0cy1ZXImlK", "reviewer": "Reviewer_4JKy", "summary": "This paper seeks to improve traditional imitation learning, by exploiting additional information that a demonstrator or expert may have during training. It does so by developing a framework to imitate the “procedure’’ used by the expert to arrive at their decision of action, rather than imitating the expert action alone. This procedure is specified by a sequence of computations, that the framework estimates. They demonstrate the results of this framework on several different domains, and show it outperforms traditional behavior cloning based approaches to imitation learning. \n", "questions": "###  Primary Concerns: \n1. Validity of access to expert chain of thought: The paper to a large extent is based on having few expert demonstrations, but with access to the entirety of the experts chain of thought. How reasonable is this assumption? \n\n    a. When is expert chain of thought reasonable to assume access to? When is this infeasible? \n\n    b. When the expert chain of thought is available readily (such as in the graph examples provided), isn’t it also feasible to generally set up a learner with an equivalent amount of knowledge at inference time (such as the ability to query a graph for traversability, expand a search, etc.)? \n\n    c. When is it reasonable to assume expert chain of thought is easier to collect than simply additional demonstrations from the expert? In particular, if (state, action) pairs are easier to collect, would collecting more data that better spans the state-action space outperform PC? \n\n2. Compounding errors of estimators: At inference time, the approach relies on predicted estimates of intermediate observations $x_l$, given only $s$, using a learnt estimator $p_{\\theta}(x_l | x_{<l} ,s)$. This suggests that the approach suffers from the compounding error problem, where a small error in prediction of $x_0$ (more generally, $x_l$), would result in the input to the subsequent estimator being out of its training distribution. This in turn would lead to a (potentially larger) error in its prediction, eventually leading to potentially catastrophic error in prediction of final action $a$. \n\n    a. To what extent does this problem occur in the proposed approach? What can be done to mitigate this? \n\n    b. Would following a DAGGER like approach mitigate this? \n\n3. Manually specified procedures: A key feature of the proposed approach is the procedure assumed to be used by the expert and the learner policy. This procedure is manually specified. This suggests the eventual imitation policy is a highly structured domain-specific policy, that is supervised at every intermediate level.  This marginally detracts from the applicability of the approach. That said, the introduction of such structure is acceptable, but requires discussion of the following points that is currently lacking - \n\n    a. It is unclear how much effort is spent on specifying these procedures (particularly when it is unclear what the true chain of thought of the expert is), and how much PC benefits from this compared to the unstructured BC / AugBC approaches. For example, the BC / AugBC approaches are performing a much more difficult inference problem, estimating $p(a | s)$, whereas the individual inference problems that PC needs to perform ($p(a | x_L)$, $p(x_l | x_{l-1})$, $p(x_0 | s)$) are each much easier. How much does PC benefit from this? This is only briefly discussed in the limitations section. \n\n    b.  When the true chain of thought is unavailable, or not easily approximated, what happens? This seems analogous to the problem of estimating the structure of a graphical model vs. inference on a given graphical model - perhaps the authors could provide directions of automatic inference of this chain of thought.\n\n    c. If a significant effort is spent in specifying this procedure, could other baseline approaches benefit from equivalent domain-specific engineering work? For example, could a leaner agent on the graph traversal problem be provided with a graph structure, and the ability to query the graph for traversability, etc.? Would PC still outperform such a classical search? \n\n4. Soundness of evaluation: It is unclear how fair some of the evaluations are. For example:\n\n    a. Are the model capacities comparable across the PC, BC, AuxBC, AugBC approaches? Or does the PC approach have larger model capacity due to estimating the additional $p_{\\theta}(x_l)$ estimators? \n\n    b. It is also unclear if the parameterizations of PC vs BC / AuxBC approaches are entirely fair. For example, the BC and AuxBC models are parameterized with MLPs / CNNs in the maze traversal domain. In this case, the PC approach has inherent access to sequential / history information over time via the visited array, but AuxBC and BC approaches do not, since they are single-timestep models. What if the parameterizations were recurrent across all approaches, thus providing all approaches with temporal information? Does this help the AuxBC / BC approaches? \n\n    c. It is unclear if the AuxBC approach is provided with sufficient or comparable information to the PC approach. Comparing AuxBC and PC aims to determine how beneficial procedural cloning as a central objective is compared to an auxiliary objective to behavior cloning. If the AuxBC approach does not have sufficient input information (such as search states, or object states in manipulation, that PC is provided in the form of supervision of the intermediate observations), it is not entirely fair to compare it against PC. \n\n    d. It would also be interesting to see how much BC / AugBC benefit from having their intermediate layers supervised by an expert (say, with supervision in the form of feature matching losses on each layer of their architecture), and how PC compares to this. \n\n\n###  Other concerns: \nIt is unclear when the $p_{\\theta}$ estimators can be made auto-regressive vs. conditionally independent. Can this be made arbitrarily? \nLine 206: It is unclear why the number of demonstrations dictates the conditional dependence of factorization. \nThe paper considers the cross-entropy objective; it is unclear if this is the best objective here. For example, one could also consider the joint distribution $p(s,a)$, and maximize the likelihood of the observed states and actions, with an according factorization. \n\n###  Typos / minor edits:\nLine 115: “many” fewer → far fewer\nLine 120-121: Could be more specific about the “procedures” that directly influence actions through “computations”. Providing examples of these procedures and computations, their form, etc., would be useful. \n\n", "limitations": "### Limitations \nThe authors do discuss limitations of their paper briefly. However, the paper could benefit from a more thorough discussion of the limitations and their implications. For example - \n1. How limiting is the manual specification of procedures? What problems would open up should this be resolved? \n2. How limiting is the requirement of additional information from the expert? When is such information hard to access? \n3. When is the cost of getting such additional information go beyond the cost of collecting additional demonstrations, thus making the problem setting irrelevant? \n\nAnother concern I have with the framing / claims of the paper is the language of learning the thought process / chain of thought of the agent used is very general, and allows for misattribution of the proposed frameworks capabilities. More precise language here could prevent adjacent communities from misunderstanding / overestimating the frameworks' capability. \n\n###  Conclusion: \nWhile there are several concerns voiced, I believe this paper introduces an interesting way of thinking about imitation learning problems where access to expert thought processes is reasonable. I believe this is a valuable contribution to the imitation learning community. \n\n\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Pros: \n1. Well motivated: The paper is well motivated. It discusses how traditional imitation learning approaches work, the drawbacks with such approaches, and how exploiting additional information that demonstrators / experts often have access to could improve imitation learning. It thus sets up the motivation for chain of thought imitation well. The paper, in general, is also well written and easy to follow, with minor exceptions (see below). \n2. Sound and elegant problem set up: The paper sets up the problem well. The framing of learning the sequence of computations as a factored joint distribution is sound, and is an elegant (if simplistic) way of systematically representing these computations. Consequently, the presented chain of thought imitation is a fairly broadly applicable framework, and unifies similar ideas in various domains well in the imitation learning setting.\n3. Performance on variety of tasks: The paper also does a good job evaluating on varied domains, and explaining the instantiations of its procedural imitation in each of these domains. The proposed method also outperforms classical imitation learning approaches on these domains, albeit with some concerns (see below). \n\nFor concerns / weaknesses and questions, please see below. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M_77O52Sl2h", "reviewer": "Reviewer_13Cs", "summary": "This paper proposes to go beyond the conventional observations->actions setup of imitation learning, and instead learn a more meaningful mapping of observations - > procedure (intermediate states/obs) -> actions. The hypothesis is that this harder prediction problem results in agents learning a more understanding of the task at hand, rather than action level mimicking. The authors draw parallels to chain of thought prompting in NLP literature, and perform a series of experiments in navigation, manipulation, and discrete control tasks to demonstrate the efficacy of procedure cloning. ", "questions": "* Can you explain the key idea of procedure cloning could be explained in a simpler manner (see above). \n* This is not a requirement, but I would encourage the authors to try their method of harder experiments which are currently not feasible to solve with Behaviour Cloning alone. Improved metrics on simple tasks is fine, but what new capabilities does procedure cloning give us? Some ideas are trying on MineRL or StarcraftUnplugged.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* A very elegant and intuitive idea that should be generally applicable to any RL domain, provided procedure demonstrations are available. \n* The experiments cover a wide range of domains, and are not limited to particular setting. There seems to be clear benefit of using Procedure Cloning over standard Behaviour Cloning. \n\nWeaknesses: \n* Section 4 needs a lot of re-writing to explain the key idea (which is pretty simple but has been very tersely explained). In particular, I think having a Python like pseudocode would make it very easy for readers to quickly grasp the idea and re-implement. The section relies on un-necessary notation and graphical models when simply {inputs, outputs, loss function, architecture} would have worked.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DcOmN5_zSBe", "reviewer": "Reviewer_CmHn", "summary": "The paper attempts to introduce a new formulation to imitation learning. The key insight is that in order to achieve strong generalization, the imitation learner needs to model not only the action at a given state, but the computational procedure that determines the action. The authors name such learning problem procedure cloning (PC). The paper cites a typical 2D navigation problem as an example where the learner can benefit from PC: the procedure policy would effectively clone a BFS search procedure, including the cell expansion and backtracking steps. The paper proposes to model the PC problem using a transformer-like structure. During inference time, the procedures are predicted auto-regressively, similar to a seq-to-seq generation problem. The formulation is validated on four different types of domains ranging from 2D navigation to robot manipulation. The main findings are that PC is particularly helpful in low-data regime and out-of-distribution generalization thanks to its strong inductive biases. \n", "questions": "My main question is regarding the technical contribution of this paper given that there already exists a large body of works that has proposed various form of procedural cloning. See the full context in the main comment above. ", "limitations": "I believe the paper has adequately addressed the limitation of the PC method.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I enjoyed reading the manuscript – the writing is superb. And I echo strongly with the message that imitation learning needs to go beyond modeling actions at a given state. I also appraise that the paper uses various types of domains to validate the main claim that modeling procedure is more advantageous than merely cloning actions. \n\nHowever, as much as I like the key message of the paper, I do have strong doubts about the technical contributions of this paper, especially when there already exists a large body of work in imitation learning that has proposed various forms of procedural cloning. \n\nThe first thing that came to mind is the seminal work Neural Programmer-Interpreter (NPI) by Reed et al (2016). The paper proposes to learn the intermediate computations of any procedure that can be represented by a program-like structure. Similar to PC, the intermediate computation does not need to take any specific form – it can be a sorting algorithm or a policy that rotates a 3D model to its canonical view, hence the learning algorithm is domain-agnostic. And the model itself is also effectively a seq-to-seq architecture (LSTM as opposed to Transformer). A follow-up work of NPI (Xu et al., 2018) also shows that NPI can be used to perform long-horizon manipulation tasks where hierarchical imitation learning can be cast as a program induction problem.\n\nThe second type of algorithms are goal-conditional imitation learning (Mandlekar et al., 2020), or a more generalized form of Decision Transformers (DT). The PC formulation in Section 6.2 is effectively goal-conditional IL where the goal is expressed in goal points. There are many existing works that show that hierarchical or goal-conditional imitation learning methods show better sample efficiency and / or stronger generalization (Lynch et al., 2020) due to its inductive biases. \n\nThe third type of algorithms are learning-to-plan methods that clone the planning procedure itself. For example, Universal Planning Networks (Aravind et al., 2018) attempts to recover the planning procedure through a bi-level gradient-based optimization procedure. Similarly, various works have attempted to make planning faster by learning the stopping condition or plan feasibility.\n\nOne may argue that PC formulation may encompass all the works above. However, this argument runs the risk of overgeneralization, as any type of fully-supervised algorithms for decision making that exploits inductive biases can be considered as procedural cloning. And applying Transformer-like architectures to all the problems above mostly demonstrates the versatility of Transformers. Hence my main question for the author is: what’s the main contribution of this paper on top of existing works that leverage domain-specific supervisions to improve policy learning?\n\nReferences:\n- Reed, Scott, and Nando De Freitas. \"Neural programmer-interpreters.\" arXiv preprint arXiv:1511.06279 (2015).\n- Xu, Danfei, et al. \"Neural task programming: Learning to generalize across hierarchical tasks.\" 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.\n- Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" Advances in neural information processing systems 34 (2021): 15084-15097.\n- Mandlekar, Ajay, et al. \"Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data.\" 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.\n- Lynch, Corey, et al. \"Learning latent plans from play.\" Conference on robot learning. PMLR, 2020.\n- Srinivas, Aravind, et al. \"Universal planning networks: Learning generalizable representations for visuomotor control.\" International Conference on Machine Learning. PMLR, 2018.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4am6ZzLZTwr", "reviewer": "Reviewer_mA33", "summary": "Paper introduces a new paradigm for imitation learning -- procedure cloning. In addition to expert demonstrations, agents will also be provided with privileged information on how exactly the experts draw these plans, ex. the intermediate states of some search/planning algorithms. Then the learned policy will have to predict these states in a row and finally the action. The authors acknowledge some similar endeavors that happened before, which also leverage the extra information in RL and IL but in auxiliary tasks fashion. Experiments in various domains including path finding, robotic motion planning, and video game playing demonstrate the effectiveness of their proposed method, especially on zero-shot generalization. ", "questions": "See [Weaknesses]", "limitations": "The authors mentioned the limitation on computation cost and the requirement of domain knowledge in applying their approach. However, I also find it fails to properly measure the extra overhead brought to experts as more information is yet to be required. It might not be a serious issue for some established algorithms, but as I mentioned above, we must consider what will happen when it is learning from humans; otherwise, the merit of this method could be on thin ice.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strenghs]\n\n+The paper is overall clear and well-written. The presentation is exceptionally illustrative and easy to follow. I like how figure 2 showcases their main contribution to prior arts in such a simple but efficient fashion.\n\n+The idea is intuitive but reasonable. Although it is straightforward to me that more privileged information could lead to better performances, the authors did a good job comparing Aux BC and therefore justified the value of their proposed \"procedure cloning\".\n\n+I like how they perform PC on atari with MCTS as an expert. Very well executed and the backward chaining style planning with GPT is impressive.\n\n[Weaknesses]\n\nHaving said these above, I do have some concerns regarding the method itself and empirical results. I hope the author could clarify them in a rebuttal.\n\n-The approach relies on the intermediate state of some other established algorithms to serve as the source of \"thoughts\". However, I'm wondering how can it learn from humans? The authors did mention this \"where \\pi* is a human demonstrator, we can ask the human to explain the thought process that led to their decision.\" but provide no results. All the experiments are about learning from some established algorithms, which I presume with perfect I.I.D. and even O.O.D. generalization on the selected domains. I do think the value of this method could be questionable without the proclaimed human case.\n\n-Zero-shot generalization should be a central topic in the experiment part as claimed by the authors. But in general, it is not very well conducted and the results don't seem to be comprehensive, some of them are even less convincing. As far as I can tell, most of these zero-shot settings are simply created by using different random seeds, ex. new maze structures, novel initial states, etc, and there is no generalization test for the robotics planning domain. These generalization tests are still largely I.I.D. and comparably simple, considering the amount of extra information the proposed agents received. I will suggest re-designing the testing envs by following the protocol that aims at testing algorithmic generalization, ex. testing regimes proposed by [1, 2], where training and testing problems are distinctive on structures but still share the same primitives. In the pathfinding domain, it could be larger mazes during testing; For video games then it could be more opponents than training tasks. The results could be much more convincing if these challenging tests are included.\n\n-Since the action is finally generated after predicting a few steps of \"thoughts\", it will be pivotal to also examine whether these \"thoughts\" are reasonable, or even interpretable, as they're trained to be so. However, I didn't find any examples in the main paper. Can the authors provide some of them in the rebuttal and the next revision of this manuscript? It will be better to also compare them with the prediction made by Aux BC.\n\n[1] Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents\n\n[2] HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept