Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to go beyond the conventional observations->actions setup of imitation learning, and instead learn a more meaningful mapping of observations - > procedure (intermediate states/obs) -> actions. The hypothesis is that this harder prediction problem results in agents learning a more understanding of the task at hand, rather than action level mimicking. The authors draw parallels to chain of thought prompting in NLP literature, and perform a series of experiments in navigation, manipulation, and discrete control tasks to demonstrate the efficacy of procedure cloning. 
strengths_and_weaknesses: Strengths:
* A very elegant and intuitive idea that should be generally applicable to any RL domain, provided procedure demonstrations are available. 
* The experiments cover a wide range of domains, and are not limited to particular setting. There seems to be clear benefit of using Procedure Cloning over standard Behaviour Cloning. 

Weaknesses: 
* Section 4 needs a lot of re-writing to explain the key idea (which is pretty simple but has been very tersely explained). In particular, I think having a Python like pseudocode would make it very easy for readers to quickly grasp the idea and re-implement. The section relies on un-necessary notation and graphical models when simply {inputs, outputs, loss function, architecture} would have worked.

questions: * Can you explain the key idea of procedure cloning could be explained in a simpler manner (see above). 
* This is not a requirement, but I would encourage the authors to try their method of harder experiments which are currently not feasible to solve with Behaviour Cloning alone. Improved metrics on simple tasks is fine, but what new capabilities does procedure cloning give us? Some ideas are trying on MineRL or StarcraftUnplugged.
limitations: N/A
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper attempts to introduce a new formulation to imitation learning. The key insight is that in order to achieve strong generalization, the imitation learner needs to model not only the action at a given state, but the computational procedure that determines the action. The authors name such learning problem procedure cloning (PC). The paper cites a typical 2D navigation problem as an example where the learner can benefit from PC: the procedure policy would effectively clone a BFS search procedure, including the cell expansion and backtracking steps. The paper proposes to model the PC problem using a transformer-like structure. During inference time, the procedures are predicted auto-regressively, similar to a seq-to-seq generation problem. The formulation is validated on four different types of domains ranging from 2D navigation to robot manipulation. The main findings are that PC is particularly helpful in low-data regime and out-of-distribution generalization thanks to its strong inductive biases. 

strengths_and_weaknesses: Overall I enjoyed reading the manuscript – the writing is superb. And I echo strongly with the message that imitation learning needs to go beyond modeling actions at a given state. I also appraise that the paper uses various types of domains to validate the main claim that modeling procedure is more advantageous than merely cloning actions. 

However, as much as I like the key message of the paper, I do have strong doubts about the technical contributions of this paper, especially when there already exists a large body of work in imitation learning that has proposed various forms of procedural cloning. 

The first thing that came to mind is the seminal work Neural Programmer-Interpreter (NPI) by Reed et al (2016). The paper proposes to learn the intermediate computations of any procedure that can be represented by a program-like structure. Similar to PC, the intermediate computation does not need to take any specific form – it can be a sorting algorithm or a policy that rotates a 3D model to its canonical view, hence the learning algorithm is domain-agnostic. And the model itself is also effectively a seq-to-seq architecture (LSTM as opposed to Transformer). A follow-up work of NPI (Xu et al., 2018) also shows that NPI can be used to perform long-horizon manipulation tasks where hierarchical imitation learning can be cast as a program induction problem.

The second type of algorithms are goal-conditional imitation learning (Mandlekar et al., 2020), or a more generalized form of Decision Transformers (DT). The PC formulation in Section 6.2 is effectively goal-conditional IL where the goal is expressed in goal points. There are many existing works that show that hierarchical or goal-conditional imitation learning methods show better sample efficiency and / or stronger generalization (Lynch et al., 2020) due to its inductive biases. 

The third type of algorithms are learning-to-plan methods that clone the planning procedure itself. For example, Universal Planning Networks (Aravind et al., 2018) attempts to recover the planning procedure through a bi-level gradient-based optimization procedure. Similarly, various works have attempted to make planning faster by learning the stopping condition or plan feasibility.

One may argue that PC formulation may encompass all the works above. However, this argument runs the risk of overgeneralization, as any type of fully-supervised algorithms for decision making that exploits inductive biases can be considered as procedural cloning. And applying Transformer-like architectures to all the problems above mostly demonstrates the versatility of Transformers. Hence my main question for the author is: what’s the main contribution of this paper on top of existing works that leverage domain-specific supervisions to improve policy learning?

References:
- Reed, Scott, and Nando De Freitas. "Neural programmer-interpreters." arXiv preprint arXiv:1511.06279 (2015).
- Xu, Danfei, et al. "Neural task programming: Learning to generalize across hierarchical tasks." 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.
- Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." Advances in neural information processing systems 34 (2021): 15084-15097.
- Mandlekar, Ajay, et al. "Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data." 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.
- Lynch, Corey, et al. "Learning latent plans from play." Conference on robot learning. PMLR, 2020.
- Srinivas, Aravind, et al. "Universal planning networks: Learning generalizable representations for visuomotor control." International Conference on Machine Learning. PMLR, 2018.

questions: My main question is regarding the technical contribution of this paper given that there already exists a large body of works that has proposed various form of procedural cloning. See the full context in the main comment above. 
limitations: I believe the paper has adequately addressed the limitation of the PC method.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Paper introduces a new paradigm for imitation learning -- procedure cloning. In addition to expert demonstrations, agents will also be provided with privileged information on how exactly the experts draw these plans, ex. the intermediate states of some search/planning algorithms. Then the learned policy will have to predict these states in a row and finally the action. The authors acknowledge some similar endeavors that happened before, which also leverage the extra information in RL and IL but in auxiliary tasks fashion. Experiments in various domains including path finding, robotic motion planning, and video game playing demonstrate the effectiveness of their proposed method, especially on zero-shot generalization. 
strengths_and_weaknesses: [Strenghs]

+The paper is overall clear and well-written. The presentation is exceptionally illustrative and easy to follow. I like how figure 2 showcases their main contribution to prior arts in such a simple but efficient fashion.

+The idea is intuitive but reasonable. Although it is straightforward to me that more privileged information could lead to better performances, the authors did a good job comparing Aux BC and therefore justified the value of their proposed "procedure cloning".

+I like how they perform PC on atari with MCTS as an expert. Very well executed and the backward chaining style planning with GPT is impressive.

[Weaknesses]

Having said these above, I do have some concerns regarding the method itself and empirical results. I hope the author could clarify them in a rebuttal.

-The approach relies on the intermediate state of some other established algorithms to serve as the source of "thoughts". However, I'm wondering how can it learn from humans? The authors did mention this "where \pi* is a human demonstrator, we can ask the human to explain the thought process that led to their decision." but provide no results. All the experiments are about learning from some established algorithms, which I presume with perfect I.I.D. and even O.O.D. generalization on the selected domains. I do think the value of this method could be questionable without the proclaimed human case.

-Zero-shot generalization should be a central topic in the experiment part as claimed by the authors. But in general, it is not very well conducted and the results don't seem to be comprehensive, some of them are even less convincing. As far as I can tell, most of these zero-shot settings are simply created by using different random seeds, ex. new maze structures, novel initial states, etc, and there is no generalization test for the robotics planning domain. These generalization tests are still largely I.I.D. and comparably simple, considering the amount of extra information the proposed agents received. I will suggest re-designing the testing envs by following the protocol that aims at testing algorithmic generalization, ex. testing regimes proposed by [1, 2], where training and testing problems are distinctive on structures but still share the same primitives. In the pathfinding domain, it could be larger mazes during testing; For video games then it could be more opponents than training tasks. The results could be much more convincing if these challenging tests are included.

-Since the action is finally generated after predicting a few steps of "thoughts", it will be pivotal to also examine whether these "thoughts" are reasonable, or even interpretable, as they're trained to be so. However, I didn't find any examples in the main paper. Can the authors provide some of them in the rebuttal and the next revision of this manuscript? It will be better to also compare them with the prediction made by Aux BC.

[1] Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents

[2] HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
questions: See [Weaknesses]
limitations: The authors mentioned the limitation on computation cost and the requirement of domain knowledge in applying their approach. However, I also find it fails to properly measure the extra overhead brought to experts as more information is yet to be required. It might not be a serious issue for some established algorithms, but as I mentioned above, we must consider what will happen when it is learning from humans; otherwise, the merit of this method could be on thin ice.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper seeks to improve traditional imitation learning, by exploiting additional information that a demonstrator or expert may have during training. It does so by developing a framework to imitate the “procedure’’ used by the expert to arrive at their decision of action, rather than imitating the expert action alone. This procedure is specified by a sequence of computations, that the framework estimates. They demonstrate the results of this framework on several different domains, and show it outperforms traditional behavior cloning based approaches to imitation learning. 

strengths_and_weaknesses: ### Pros: 
1. Well motivated: The paper is well motivated. It discusses how traditional imitation learning approaches work, the drawbacks with such approaches, and how exploiting additional information that demonstrators / experts often have access to could improve imitation learning. It thus sets up the motivation for chain of thought imitation well. The paper, in general, is also well written and easy to follow, with minor exceptions (see below). 
2. Sound and elegant problem set up: The paper sets up the problem well. The framing of learning the sequence of computations as a factored joint distribution is sound, and is an elegant (if simplistic) way of systematically representing these computations. Consequently, the presented chain of thought imitation is a fairly broadly applicable framework, and unifies similar ideas in various domains well in the imitation learning setting.
3. Performance on variety of tasks: The paper also does a good job evaluating on varied domains, and explaining the instantiations of its procedural imitation in each of these domains. The proposed method also outperforms classical imitation learning approaches on these domains, albeit with some concerns (see below). 

For concerns / weaknesses and questions, please see below. 


questions: ###  Primary Concerns: 
1. Validity of access to expert chain of thought: The paper to a large extent is based on having few expert demonstrations, but with access to the entirety of the experts chain of thought. How reasonable is this assumption? 

    a. When is expert chain of thought reasonable to assume access to? When is this infeasible? 

    b. When the expert chain of thought is available readily (such as in the graph examples provided), isn’t it also feasible to generally set up a learner with an equivalent amount of knowledge at inference time (such as the ability to query a graph for traversability, expand a search, etc.)? 

    c. When is it reasonable to assume expert chain of thought is easier to collect than simply additional demonstrations from the expert? In particular, if (state, action) pairs are easier to collect, would collecting more data that better spans the state-action space outperform PC? 

2. Compounding errors of estimators: At inference time, the approach relies on predicted estimates of intermediate observations $x_l$, given only $s$, using a learnt estimator $p_{\theta}(x_l | x_{<l} ,s)$. This suggests that the approach suffers from the compounding error problem, where a small error in prediction of $x_0$ (more generally, $x_l$), would result in the input to the subsequent estimator being out of its training distribution. This in turn would lead to a (potentially larger) error in its prediction, eventually leading to potentially catastrophic error in prediction of final action $a$. 

    a. To what extent does this problem occur in the proposed approach? What can be done to mitigate this? 

    b. Would following a DAGGER like approach mitigate this? 

3. Manually specified procedures: A key feature of the proposed approach is the procedure assumed to be used by the expert and the learner policy. This procedure is manually specified. This suggests the eventual imitation policy is a highly structured domain-specific policy, that is supervised at every intermediate level.  This marginally detracts from the applicability of the approach. That said, the introduction of such structure is acceptable, but requires discussion of the following points that is currently lacking - 

    a. It is unclear how much effort is spent on specifying these procedures (particularly when it is unclear what the true chain of thought of the expert is), and how much PC benefits from this compared to the unstructured BC / AugBC approaches. For example, the BC / AugBC approaches are performing a much more difficult inference problem, estimating $p(a | s)$, whereas the individual inference problems that PC needs to perform ($p(a | x_L)$, $p(x_l | x_{l-1})$, $p(x_0 | s)$) are each much easier. How much does PC benefit from this? This is only briefly discussed in the limitations section. 

    b.  When the true chain of thought is unavailable, or not easily approximated, what happens? This seems analogous to the problem of estimating the structure of a graphical model vs. inference on a given graphical model - perhaps the authors could provide directions of automatic inference of this chain of thought.

    c. If a significant effort is spent in specifying this procedure, could other baseline approaches benefit from equivalent domain-specific engineering work? For example, could a leaner agent on the graph traversal problem be provided with a graph structure, and the ability to query the graph for traversability, etc.? Would PC still outperform such a classical search? 

4. Soundness of evaluation: It is unclear how fair some of the evaluations are. For example:

    a. Are the model capacities comparable across the PC, BC, AuxBC, AugBC approaches? Or does the PC approach have larger model capacity due to estimating the additional $p_{\theta}(x_l)$ estimators? 

    b. It is also unclear if the parameterizations of PC vs BC / AuxBC approaches are entirely fair. For example, the BC and AuxBC models are parameterized with MLPs / CNNs in the maze traversal domain. In this case, the PC approach has inherent access to sequential / history information over time via the visited array, but AuxBC and BC approaches do not, since they are single-timestep models. What if the parameterizations were recurrent across all approaches, thus providing all approaches with temporal information? Does this help the AuxBC / BC approaches? 

    c. It is unclear if the AuxBC approach is provided with sufficient or comparable information to the PC approach. Comparing AuxBC and PC aims to determine how beneficial procedural cloning as a central objective is compared to an auxiliary objective to behavior cloning. If the AuxBC approach does not have sufficient input information (such as search states, or object states in manipulation, that PC is provided in the form of supervision of the intermediate observations), it is not entirely fair to compare it against PC. 

    d. It would also be interesting to see how much BC / AugBC benefit from having their intermediate layers supervised by an expert (say, with supervision in the form of feature matching losses on each layer of their architecture), and how PC compares to this. 


###  Other concerns: 
It is unclear when the $p_{\theta}$ estimators can be made auto-regressive vs. conditionally independent. Can this be made arbitrarily? 
Line 206: It is unclear why the number of demonstrations dictates the conditional dependence of factorization. 
The paper considers the cross-entropy objective; it is unclear if this is the best objective here. For example, one could also consider the joint distribution $p(s,a)$, and maximize the likelihood of the observed states and actions, with an according factorization. 

###  Typos / minor edits:
Line 115: “many” fewer → far fewer
Line 120-121: Could be more specific about the “procedures” that directly influence actions through “computations”. Providing examples of these procedures and computations, their form, etc., would be useful. 


limitations: ### Limitations 
The authors do discuss limitations of their paper briefly. However, the paper could benefit from a more thorough discussion of the limitations and their implications. For example - 
1. How limiting is the manual specification of procedures? What problems would open up should this be resolved? 
2. How limiting is the requirement of additional information from the expert? When is such information hard to access? 
3. When is the cost of getting such additional information go beyond the cost of collecting additional demonstrations, thus making the problem setting irrelevant? 

Another concern I have with the framing / claims of the paper is the language of learning the thought process / chain of thought of the agent used is very general, and allows for misattribution of the proposed frameworks capabilities. More precise language here could prevent adjacent communities from misunderstanding / overestimating the frameworks' capability. 

###  Conclusion: 
While there are several concerns voiced, I believe this paper introduces an interesting way of thinking about imitation learning problems where access to expert thought processes is reasonable. I believe this is a valuable contribution to the imitation learning community.
review_title: null

Meta-review: metareview: All reviewers acknowledged the rebuttal or replied to the author responses.
The paper proposes to clone procedures in imitation learning rather than blindly mimicking actions. One reviewer is still not convinced that the experiments are sufficient, while I believe they are solid enough for a conference paper. The relation to other methods that take a similar conceptual approach on a high-level is now clear, but should be incorporated in the main paper.
confidence: Less certain
Decision: Accept