Neuro-Symbolic Procedural Planning with Commonsense Prompting
Authors: Yujie Lu
Weixi Feng
Wanrong Zhu
Wenda Xu
Xin Eric Wang
Miguel Eckstein
William Yang Wang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Procedural Planning
Commonsense Knowledge
Prompting
Neuro-Symbolic

Abstract: Procedural planning aims to implement complex high-level goals by decomposition into simpler low-level steps. Although procedural planning is a basic skill set for humans in daily life, it remains a challenge for large language models (LLMs) that lack a deep understanding of the cause-effect relations in procedures. Previous methods require manual exemplars to acquire procedural planning knowledge from LLMs in the zero-shot setting. However, such elicited pre-trained knowledge in LLMs induces spurious correlations between goals and steps, which impair the model generalization to unseen tasks. In contrast, this paper proposes a neuro-symbolic procedural PLANner (PLAN) that elicits procedural planning knowledge from the LLMs with commonsense-infused prompting. To mitigate spurious goal-step correlations, we use symbolic program executors on the latent procedural representations to formalize prompts from commonsense knowledge bases as a causal intervention toward the Structural Causal Model. Both automatic and human evaluations on WikiHow and RobotHow show the superiority of PLAN on procedural planning without further training or manual exemplars.

Published as a conference paper at ICLR 2023
PROCEDURAL PLANNING WITH COMMONSENSE PROMPTING
Yujie Lu', Weixi Feng', Wanrong Zhu', Wenda Xu', Xin Eric Wang? Miguel Eckstein', William Yang Wang' 'University of California, Santa Barbara, CA, USA
{yujielu, wwwww king,, wanrongzhu, wendaxu) edu (miguelecksteing wangwilliamyang @ucgj edu Unniversity of California, Santa Cruz, CA, USA xwang3 66ucscc.eu
ABSTRACT
Procedural planning aims to implement complex high-level goals by decomposition into simpler low-level steps. Although procedural planning is a basic skill set for humans in daily life, it remains challenge for large language models (LLMs) that lack a deep understanding of the cause-effect relations in procedures. Previous methods require manual exemplars acquire procedural knowledge from LLMS in the zero-shot setting. However, such elicited pre-trained knowledge in L.MSs induces spurious correlations between goals and steps, impairing the model's gen- eralization to unseen tasks. In contrast, this paper proposes a neuro-symbolic procedural PLANNer (PLAN) that elicits procedural knowledge from the LLMs prompting. To mitigate spurious goal-step correlations, we use symbolic program executors on the latent procedural representations to for- malize prompts from external knowledge bases as a causal intervention toward the Structural Causal Model of procedural planning. Both automatic and human evalu- ations on WikiHow and RobotHow show the superiority of PLAN procedural planning without further training or manual exemplars
with commonsense-infusee
INTRODUCTION
How to make a cup of coffee? As humans, we can easily specify a procedure to solve this task, using innate ability of commonsense reasoning. However,
endow machines with the same ability to construct a sequential plan? As depicted in Figure[] procedural planning (Pearson] 1996) Zhang al. 2020b Huang al. 2022) aims to decompose high-level goal (Task: Watch TVV into a sequence of temporally extended steps (Procedural Plan: Step at all five time-steps).
We study procedural planning as the conditional text generation problem since resembles real-world scenarios. Previous approaches (Huang et al. 2022: Ahn et al., 2022) require a small number of carefully written held-out exemplars to acquire procedural knowledge. However, these manual exemplars evolved from task data are impossible to cover the ever-changing task setups and the flexible dependency relations among goals and steps. In fact, the biased data may cause the model to learn spurious correlations and hinder the model from generalizing well in zero-shot scenarios. Studies in cognitive science show that humans rely on chunking mechanisms (Gobet et al., 2001 Miller 1956) which turn primitive stimuli into conceptual groups to solve novel and complex problems. Inspired by this, we hypothesize that generalizable procedural planning ability can be achieved by learning cause-effect relations among complex goals and simpler steps using external knowledge.
To reveal the cause-effect relations in procedural planning, we devise a Structural Causal Model (SCM) (Peters et al. 2017). directed acyclic graph commonly used to describe the causal relation- ships within a system Pearl (2009). As depicted in Figure the pre-trained knowledge (D) (c.g.. TV and living room is highly correlated) in L.MMs confounds (?) influences T. S and S,, resulting in spurious correlations) the system to make biased decisions toward an unreasonable step (e.g., Find
Source code and datasets are publicly available at http:///tte.ggoggge...///wwwwaap

---

Published as a conference paper at ICLR 2023
VirtualHome
RobotHow Task
ConceptNet
WikiHow Task
External Aroweedge
Gout Jiput
Symbolic Rule
Hnsalratio in
LLMs
LLMs
Decompoation
Commonsenss-inttsee Prompt-
decompantiinn
Procedural Plan robotHow
Procedural Plan
wiki How
Step Walk living
Step 1: Practice on the ground first.
Step 2: Switch on TV:
Step Practice getting up.
Step 3: Grab Remote Control.
Step Learn to stand the board properly.
Step Walk Sofa.
Step Paddle around and get comfortable in the water.
Step S: Sit on Sofa.
Step to more experienced surfers instructors you can.
Figure Two independant procedural planning task examples from RobotHow and WikiHow. PLAN construct commonsense-iniee inffuue prompt from external knowledge (e.g., ConceptNet) to elicit procedural planning ability of the Large Language Models (LLMM)) without training or exemplars.
Television). Thus, adopt front-door adjustment (definition in Appendix A.3 which utilizes mediator (P:) that blocks all directed paths from the cause (T or S;-1) to the effect (S,). In this way, T (or 1) affects S; by flowing through indirect paths: T (or S:-1) affects P. and P. affects S, And we can identify the causal effects among goals and steps by investigating the indirect effect (Equation 3). which computed by multiplying the effect of T (or S,-1) on P--1 (Equation I with the effect of P. on S; (Equation 2 With the above front-door adjustment, we can mitigate the spuri- ous correlations (e.g., between "television" and "living room") and thus make reasonable decisions on steps (e.g.. Find book). Please refer to A.1 for causal preliminaries (including explanation for SCM, confounder, mediator, spurious correlations), A.3 for the front-door adjustment definition.
Guided by the above causal analysis of procedural planning, we need to construct the mediator P; and then intervene on task and prompt P., which required to compute the conditional probability in 3quatior3] As depicted in Figure 3 we seek to automatically construct commonsense-infuss prompts as the mediator P. by concatenating the task, previous steps with commonsense knowledge extracted from external resources (c.g., ConceptNet (Speer al. 2017)). First, modify the goal input by sampling a task-relevant knowledge subgraph (Stagel in Section 3.1 to implement interventions on T. Then, we modify the prompt by adapting the edge weight to implement interventions P; (Edge-Wist Wise Adoption Stage2 in (..l However, directly incorporating knowledge of graph structure into LLMs leads to the loss of the logical order in eliciting procedural knowledge from LLMs. Thus, we apply symbolic executors (Mao et al. 2019 Yi al. 2018) that execute the sequential mapping program on latent knowledge representations (e.g., the subevent of), In this way, we transit graph structure knowledge into natural language that preserves procedural structure, such as the sequential order of two low-level steps (Symbolic Structuring of Stage2 in Section [3.Tp. The procedural prompt Pa (e.g, "please get the remote control") is further translated into admissible one Po (e.g.. "grab remote control") from available steps in certain domain (RobotHow or WikiHow in our case). Finally, utilize the commonsense-infuseee prompt Pa to control the generation of procedural plans in LLMs in a zero-shot setting (Section 3.2).
We conducted experiments RobotHow (Puig et al. 2018 and WikiHow (Koupace & Wang 2018
under original and counterfactual situations. Our major contributions can be summarized as:
We develop the first causal framework for procedural planning by 1) defining a temporally extended Structural Causal Model and 2) resolving spurious correlation between high-level goals and low-level steps via front-door adjustment with a prompt- cooset-based mediator.
We propose neuro-symbolic approach to construct commonsense-infused prompts for LLMs to tackle the procedural planning task without manual exemplars or further training.
Extensive evaluations show the superiority of PLAN in terms of reasoning about the cause- effect relations among goals and steps and achieving promising planning ability.
2 EXTERNAL KNOWLEDGE MATTERS IN PROCEDURAL PLANNING
As depicted in Figure ! procedural planning requires generating the Plan (e.g., Step I: Walk to the living room.) conditioned the Task (e.g., Watch TV). We first describe the problem definition
2

---

Published as a conference paper at ICLR 2023
Living and highly correlated P[TV]living room) -
D
P3
"Read book in the living room"
T
Pi
S
Sz
S
"Find Television" (w/o adjustment) =Find book" [w/ adjustment)
living room
(a) Full temporal causal graph
(b) SCM at step 8
Figure 2: Structural Causal Model (SCM) for Procedural Planning. (a) The full temporal causal graph. T' denotes the task query, and S; is the sub-goal step at timestep i. D is the unobservable confounding variable introduced by the LLMs. P. denotes the mediating variables we construct to mitigate the spurious correlation. (b) The SCM at timestep i. Without causal intervention, the model produces a sub-goal step "find television" due to the spurious correlation between "television" and "living room" caused by the confounding variable D. With our causal intervention, the constructed mediating variable P. (Section 3.1) can block the backdoor paths for T + S; and S - S; (opened by D) and generate the causal sub-goal "find book" precisely (Section 3.2
and then show why external knowledge matters in procedural planning through the lens of causality, Finally, we show how we elicit procedural ability from the Large Language Models (LLMs).
2.1 PROBLEM DEFINITION
Given the high-level task (e.g. watch television in the living room) sampled from a task domain Mr (e.g. RobotHow), a procedural planner aims to decompose it into lower-level temporally extended steps ST = {S1, ..... S E 5). There exists certain admissible plans S, which is fixed set constrained by the task domain Mr (e.g., the affordance of the interacted objects). The plan S; at timestep is generated as π (S, (o..)))
2.2 A CAUSAL LOOK AT PROCEDURE PLANNING WITH LLMS
We seek to empower the LLMs with the ability to reason cause-effect relations in procedural planning. Thus, we devise causal framework by first defining a Structural Causal Model (SCM) of procedural planning Figure 2 The SCM describes the temporal dynamics and procedural cause-effect relationship. Our causal assumption in SCM indicates that there is backdoor path from task to step, which must blocked with front-door adjustment. Therefore, we model the input prompt mediator which is created from external knowledge. More specifically, Temporal Causal Graph as Figure 2a] which is an unrolled Structural Causal Model (SCM) for sequential decision-maken and plan procedures Sz = {S1, S2........ from LLMs. Initially, there are direct paths + S, and Se S., k because S; relies on the LLM attended task entities and previous accomplished steps. D is an unobserved confounder from learned knowledge during pre-training. D builds a backdoor path between 7 and S, and misguides the LLMs attend to false entities generate the next step (see Fig. 2b). Note that D is unobservable as we directly adopt the LLM without knowing the pre-training data. To mitigate the spurious correlation, we then introduce mediator P. for each S; as shown in Figure 2a To achieve our front-door adjustment, we inject external knowledge into LLMs with neuro-symbolic approach by adopting three stages described in Section 3.1
define our Full
Our goal is to identify the causal relations between the attended task
OUR APPROACH
Although LLMs have strong general language intelligence, they still perform poorly in reasoning the cause-effect relations in procedural plans due to lack of daily life experience. We propose to elicit the unbiased procedural planning knowledge from the LLMs using the created commonsense-ine infuued Prompt P as क (S.|7, Soa-1, P). Figure and and Algorithm depict how PLAN tackles the procedural
3

---

Published as a conference paper at ICLR 2023
Task (T): Wetch TV the living room Gino Inpul
Semantic Parsing
Concept
Te watch TV, Bg: living room)
Retriewe
External Knowledge
Step (8 Walk Walk the living room
+ Jask and Fresous Steps
Prompt
[r: Ps]
Commoorensse-iffore
Tosk-relevont Subgraph
Admissible Knowledge
Symbolic Rule Set op H..asubeeeeent ogLAtLocation Op_ Hop_HasPrrrequisiee
Node:
Node: Living room
Temporai Eetondedd Pooe
Generation
Prompt (P&)
g.gg with
Step 1: walk to the living room.
Sofa
Score
Semantic
Step 2: grab remote control.
sit chaii.
Sdge R, Weightt E,
eg Watch TV
Translation LM,
Stop 4: switch remote control.
Step S: watch tv
Symbolic Structured
Admissible
Admisaible
Node: Turn TV on
Node:
Step 2 (S)]: Find Remote
Control
Translation LMy
Procedural Prompt
(P&)
Remote
Control
The Overview of Procedural Planning, Our five-stage pipeline includes: 1) semantically parsing the task into concept set Tg retrieve subgraph G, from the external knowledge base G. 2) formalize procedural prompt P& and then translate into the admissible one Pç. 3) aggregate task, previous steps and Pa as final commonsense-infuse prompt P. (Section 3.1) 4) and generating and translating time-extended procedural plans until triggering the termination condition. (Section 3.2)
Figure 3:
planning in a five-stage manner. We illustrate the commonsense-infuser prompt construction (the first three stages) in Section 3.1 planning with LLMs (the last stage) in Section 3.2
3.1 COMMONSENSE--NFUSEEE PROMPT CONSTRUCTION
Overview Inspired by the causal analysis in Section 2.2 we propose to construct commonsense- infused Prompt P that helps reveal the cause-effect relations among the goals and steps during procedural planning within 3 stages: 1) Stagel sample a subgraph G, from the external knowledge nodes. 2) Stage2 adapt the edge weight Ew, in G. and apply
base G by extracting task(?) symbolic structuring to get the admissible knowledge prompt Pg. 3) Stage3 acquire the temporal order by temporally aggregated the prompt P, with previous steps So:
Stagel: : Rels-eelevnnt Knowledge Subgraph Sampling First, we investigate the causal effect T - P; and S - P. (Figure 2) S; is a collider that blocks the association between D and P; in the path T + D - S, < P.. Let n, denote (--1P.-) that represent the probability density function conditioned on P:-1. Since there is no backdoor path for T + P; and similarly for S + P.. we simply have the conditional probability after applying do-operators:
3. (P, p)d)(T)) (P: = plT), (PP = p)d 7,(PP pp(S,-)
(1)
We achieve the do-operation in prompting way by modifying the goal input so that the model attends the task-relevant entities. To implement, we use NLTK to tokenize and pos_tac the task text T' Then we use the noun (e.g. television), noun phrases (e.g. remote control), and verb phrases (e.g. watch television) as the concept node. In this way, the task name T is Semantically Parsed into the Concept Set Tg. Each concept E Tg is used as query for sampling the H H-hop task-relevant subgraph G. S Ne. x R, x Ne from the external knowledge base G S A R X., where N and R represent the number of concept nodes and commonsense relations respectively. When extracting G., we keep the triplets with relation type in the household domain (e.g., Locac in, UsedFor) and filter out ones in the linguistic domain (e.g.. Di st i InnF Deri vedFrom) for the procedural planning task. No is maintained in a set of top-k task-relevan nodes using the weight of each Re.
which is updated with edge-wise adaption in Stage2.
Stage2: Edge-Wise Adaption and Symbolic Structuring Second, we need to find the causal T + D + S, contains backdoor from P. to S, cannot
effect for P. S,. Since the path P. < rely on the conditional probability. Instead, we intervene on P. using do-operator to cut off D - T:
क; (S.ldo(P p)) Σ (S. S. S) 1, 1, 8)
(2)
Σ (S. -) s/T = (T =
4

---

Published as a conference paper at ICLR 2023
The retrieved concept-centered graph has multiple edges representing various relationships with other actions/entities. Therefore, the summation over intervened T can be achieved by incorporating these edges into the prompt. For instance, "living room" can be "walked to" and "used for reading" while "book" can locate in "living room" and "bedroom". Similarly, we extrapolate over the edges for i --1 hops aggregate the intervened S; i.e. P(S,-1 = s|T t). Directly ranking the retrieved nodes Ne with the annotated weight (Ew) in the external knowledge base will result in spurious correlation. Because such retrieved local subgraphs tend capture the task-invariant concept nodes as the causal factors. To mitigate this, we propose to adapt the weight of each triplet (Edge-wise Adaption). The adapted weight is the addition of the original edge weight and the cosine similarity between the tail node embedding Fuut of the edge R., and the task embedding Dtask as: Eu E. + cosine (nt Erat Btask). The embeddings are projected from the node text and task name using the
(Reimers & Gurevych, (019). The nodes N₂ are finally retrieved by ranking the adapted weight Ew. To better track the utilized external knowledge during inference, we construct the task-dependent commonsense prompt with a Symbolic Executor (Symbolic Structuring) guided by the relation type of cach triplet in G. with the adapted edge weight beyond threshold de. Specifically, the Symbolic Executor acquires the neural information of each natural language node and executes the sequential mapping program by sampling the operation Op from the Symbolic Rule Set R according the edge relation type. The Symbolic Rule Set R is obtained by mapping the description of the relations (e.g.. AtLocation represent 'A is a typical location for B, or is the inherent location of B. Some instances of this would be considered meronyms in WordNet.') in the external knowledge graph (e.g.. ConceptNet) to symbolic operations (e.g., Op_AOp_AALLa Atocaatii)) For instance, the Location edge samples the operation Op At ion from R. which takes the commonsense relation of the triplet from G. as the parameters to query the procedural concept output given the natural language meaning of the linked nodes (e.g., go to the location of Start_N t___odeeef((@ Nod_NoodeOf(rr and use it for St Start Wwort_Nooee..o (frr)"" And operators Op Haseeveene and Hp_HasPre rerquui site will recursively navigate the subgraph G.. After navigating the subgraph, we linearize the transformed triplets as the Procedural Prompt Pg. which is then translated to Admissible Knowledge Prompt Pa by the Translation Language Model L.Mr-
in this case). Similarly, Op_ up_UsedFo or may refer to "go find
Stage3: Camporally-Extendee Aggregation To acquire temporal order the procedure, we obtain the Prompt P at timestep with the aggregation of task T, history steps So.i1 1 and current external knowledge Pc. The underlying causal mechanism is combination of Eq. | and Eq.
(o = Σ n. (S. = (p)
(3)
= Σ (p)T) Σ T. (S.lp. T () 11-1
The adjustment and marginalization in Eq. 3 achieved in the input space by forming the Procedural Prompt Pg that allows the LLM to attend on the causal entities instead of the highly correlated ones for the next step generation. The LLM can reason over the most relevant edges to link the concepts with the task entities as context. The prompts from knowledge bases are independent of the pre-training data distribution so that P: is independent of 1) and satisfies the front-door criterion. Please refer to Appendix and and Figure 4 for the simplification of our structural causal model.
3.2 PROCEDURAL. PLANNING WITH LARGE LANGUAGE MODEL.
Stage4: SSmantic Generation The external knowledge is further concatenated with the goal input (T) as the initial prompt. Given the prompt, the language model Generation LMG E (PAR. PAÆ) (e.g., GPT3, BART) generates the next sentence, and the most confident prediction is then appended to previous prompts. The Termination Condition is either reaching the max step or the matching score is below threshold 0. The joint probabilities of auto-regressive (PAR) and auto-encoder (PAR (Pag)
model is factorized as:
A (x) = I Ipls.lPo.smm-L.
AAE(D) = II p(s. le, (-1m-1, [MASK]]) (4)
where Po represent the commonsense knowledge and T represent the task name.
5

---

Published as a conference paper at ICLR 2023
Algorithm 1 Neuro-Symbolic Procedural Planning using Commonsense--nfussed Prompting
Require:
Task Sample T, Admissible Step Set S, External Knowledge Graph G; Language Model for Generation LMg and Translation LMr, Symbolic Rule Set Rt
Ensure:
1: [Stagel] Semantically parse T into entity set TE:
2: Maintain top-k task-relevant nodes N₂ in Te:
3: Retrieve subgraph G, C R, x Ne from G X R N for each TE;
4: [Stage2] ] Edge-wise adaption as E.: t- Em + cosine(n Fu Utask) and re-rank No in Tw:
5: Map the description text of the relations RS in Ge as Symbolic Rule Set R:
6: Construct procedural prompt PG by verbalizing the re-weighted Gg using R:
7: Translate Pg in Admissible Knowledge Prompt Pg = LMM((P)); comporally-extendde zero-shot inference for Procedural Plan ST {S ..... S 8: for each timestep do
9: [Stage3] Aggregate Prompt P. f So:: Pal:
10: [Stage4] and [Stage5] S, LMM(LLM(PP));
11: Update Procedural Plan ST t- S;
12:
end for
Stage5: : Admissible Step Translation To ensure that the generated procedural plans are grounded to the environment, we should avoid producing the steps that inadmissible (e.g. Toast the table). In other words, the generated steps should be fully constrained to the admissible composite
action
and object in certain task domain. Thus previous works (Huang et al. 2022; Ahn et al.] (022) have explored using the model (which is LMT in our case) to score a step selected from a fixed set of available options, instead of directly sampling from the output distributions of the language model (which is LMg in our case). Specifically, we match the generated step by LMg to the most similar admissible step in the embedding space encoded by the Translation Language Model LMT. Following (Huang et al.) (2022), we utilize Sentence-Trane Transformer (Reimers & Gurevych 2019) to calculate the cosine similarity as x (s; (a) = (LM(())) which translates LMg((*) into the admissible step s; E S that is the closest in the embedding space measured by the cosine similarity.
3.3 COUNTERFACTUAL PROCEDURAL DATA CONSTRUCTION
To investigate the counterfactual reasoning ability, design three families of intervention methods: 1) Initial Configuration: intervene in the initial configuration, such as the location for implementing the task. 2) Intermediate Step, randomly select one step from the ground truth program as an additional constraint of implementing the task and append it to the task name for generating the procedural plan. 3) Final Goal, intervene the task goal as the composite of another randomly sampled task. Table 5 in the Appendix summarizes the category and description. The counterfactual dataset construction details and post-intervention examples are provided in Appendix [B.2
4 EXPERIMENTS
4.1 PROCEDURAL PLANNING SETUP
Datasets We conduct zero-shot experiments on two datasets with procedural information, WikiHow (collected following (Koupaee& & Wang 2018) and RobotHow (Puig et al. 2018) without training. WikiHow is a large-scale text summanization dataset that is constructed from a Juuman-writhe knowledge base, involving procedural tasks that spans various topics. We utilize "how to" title as the task names and the summarized headlines as the steps. RobotHow is large knowledge base of common household tasks collected in the VirtualHome (Puig et al., 2018)) simulator. The dataset contains the programs with high-level task names and low-level steps. MT is composed of 292 and 2000 distinct tasks from RobotHow and WikiHow respectively. Human evaluations use randomly sampled 50 task examples for each dataset. Automatic evaluations use 150 and 1000 task examples randomly sampled from RobotHow and WikiHow respectively. Please refer to Appendix and and Appendix B.2 for dataset details.

---

Published as a conference paper at ICLR 2023
Model.oee
Original-Cowrrge
Original-Oddrr
Countefactual-oowerss
Couttefactual-Orlee
Win'11
Tic
Lose(_)
Warth)
Tic
Losel.)
Winth)
Tic
Losel]]
Loser_)
BART
Lewis
2000
50,00
22.67
42.00
22.67
50.00
18.67
51.35
RobotHow
GPT2
22.00
44.00
18.67
56.67
11.33
38.67
2000
23.33
53.33
23.33
23.33
54.67
16.57
50,00
76.67
46.00
32.67
WikiHow
Eationi
46.67
16.57
75.17
72.67
3.67
44.00
22.67
Table I: Percentages of procedural planning results of PLAN that are better than, tied with, or worse than Planner (Huang al., 2022). in coverage and order metrics under the original and counterfactual setting.
RobotHow
WikiHow
Architecture
Model
Original
Consterfactual
Original
Counterfactual
Caverage
Order
Coverage
Order
Coverage
Order
Coverage
Order
Chain Wei et 2022
2,99
2.80
2.71
2.76
2.88
2.97
BART Lewis et al. 2020
M. (Haang e 2023
3.06
2.84
2.96
2.82
2.78
3.35
3.46
3.02
PLAN (Ous)
3,16
3.10
3.07
2.98
3
3.47
3.62
3.18
Wi 2022
2.88
GPT2 Radford et al.
MaP (Haang al 2022
3,09
2.94
2.93
2.901
3.20
3,63
3.24
PLAN (tur))
3.12
2,99
3.43
2.88
3.67
3,69
3.81
3.31
Chain Wei al 2022
3.26
3.18
3:45
3.58
3,29
3:16
3.70
3.71
Brown al. 2020
[Tlaang crall
3.71
PLAN (Cwrs)
3.72
3.70
3.67
3.56
3.72
3,82
3,85
3.75
Table 2: Averaged 5-point Likert scale human evaluations on "coverage" and "order" aspects.
Baselines We compare our approach with three vanilla generative pre-trained language models (BART, GPT2, and GPT3) and two powerful generation baselines (Zero-shot Planner (Huang et al.
2022) noted as "LLMaP" and Chain of Thought (Wei et al. 2022) noted as "Chain"), More method and configuration details of the models can be found in Appendix B.3 and Appendix(B.4 B.4
Metries We ask human annotators on the Amazon Mechanical Turk platform to rate model perfor- mance on two aspects: 1) Coverage: depicts which set of steps can better complete the target task (captures semantic completeness). 2) Order: depicts which sequenee covers more steps that are necessary to complete the target task (captures sequential order correctness). In addition, we use Bentence-BLEU (S-BLEU) (Papineni et al. 2002),, BERTScore (Zhang* * et al., 2020). ROUGE- 1 (Lin) 2004) and Word Mover's Distance ( (WMD) (Kssne et al. 2015) automatic evaluation metrics. These metrics are used to compute the semantic scores between the annotated programs and the predictions. Details of the crowdsourcing human evaluation can be found in Appendix C.I
4.2 HUMAN EVALUATION RESULTS WITH COVERAGE AND ORDER METRIC
Each example rated by crowdsourcing annotators. For the Win-Lose Comparison, we ask the human rater to choose between ours and the baseline LLMaP (Huang et al. 2022). Averaged results reported Table [ show that our PLAN is more frequently rated as better for both coverage and order metrics, outperforming baselines over the winning ratio by 21% in coverage and 26% in order, across two datasets. We report the average results of Human Ratings with 5-point Likert scale in Table 2 The consistent performance boost of PLAN indicates the superiority of injecting external commonsense knowledge into the procedural planning task. The performance drop of LLMaP and Chain in the counterfactual setting indicates the vulnerability of fixed holdout knowledge and the pre-defined manual exemplars in causal procedural planning. Please refer to Appendix I for the crowdsourcing human evaluation interface details. Table shows two examples for Qualitative Comparison. More examples can be found in Appendix D
4.3 AUTOMATICALLY MEASURING THE PROCEDURAL PLANNING
Main Results Table 4 summarizes The automatic evaluation results. PLAN achieves the best results regardless of the architecture of the language model architecture, either autoregressive or autoencoder based. The performance gain of "LLMaP" over "Chain" may probably be due to direct exposure to the holdout task from the dataset. While the "Chain" baseline still outperforms the vanilla baseline that only takes the high-level task name the prompt. Note that the annotated program is not the
7

---

Published as a conference paper at ICLR 2023
Dataset
Task
Model
Steps
Chain
Step 1:
Step Step S: Clra howl. Step G: Pur boul oven.
oven. Step Switch oven, Step Open oven.
RobotHow
Toast Beaad
M.
Step Pui broad in owen. Step Put beend oven. Step 3: food bread on table.
Step 1:
Walk kitchen counter, Step Pull bread slice. Step 3: bread slice on tousler.
PLAN
Step 4 Swith on neweler Step S: Look at Insster. Step Turn to toveler.
Chain
Step I:
Take care of your pets. Step 2: Screll to the end of the test file
Wikillow
[oo Take
Care of Per
M.M
Make sure your pel always has enough food and water Step Be sure to feed your dog at lesst three times day, Stop 3: Feod your pet ligh quality "complete and balanced" dog food. Step 4: Gave your dog fruits and vegetables snacks. Step S: Eat variety food. Step Feed your healthy food.
Sup I:
Step Research the type get, you
the esposially if you childien.
your
and lifestyle.
PLAN
Step Exercise your if's's the kind pet
Step Choose your
dog. Step Watch your per's canefully. Step 5: Monitor your pet for illness injury. Step Mike time your pel.
Table 3: Showcases of procedural steps predicted by different models with GPT2 as the base LLM.
RobotHow
WikiHow
Model
S S-BLEU
WMD
BERT-II
ROUGE-I1
S-BLEU
WMD
BERT-I1
ROUGE-FI
BART Lewis el al. 2020
0.069
0.923
0.870
0.442
0
0.937
0.836
0.379
BART Chai Wer at 2022
0.079
0,913
0.862
0.448
0,095
0.939
0.782
0.377
BART + LLMaa Wunnng el 2022
0.094
0.940
0.870
0.467
0.131
0.950
0.816
0.371
BART PLAN (Ours)
0.110
0,951
0.890
0,528
0,142
0,958
0.833
0.400
w/o Adaption
0.104
0.929
0.886
0.492
0.132
0,952
0.824
0.398
w/o Symbolic
0.062
0.858
0.835
0,392
0,087
0.939
0.828
0,386
GPT2 (Railord al.
0.056
0.891
0.846
0.356
0
0,925
0.826
0.345
GPT2 + Cisam Wer 2022
0.079
0,906
0.861
0.405
0,124
0.937
0.817
0.352
GPT2 L.LMal (Hungg of 2022
0.115
0.931
0.885
0.481
0,115
0.957
0.833
0.363
GPT2 PLAN (Ours)
0.148
0,945
0.898
0,547
0,133
0,971
0.835
0.373
w/o Adaption
0.142
0.944
0.896
0.542
0,123
0,965
0.830
0.360
w/o Symbolic
0.343
0,942
0.895
0,538
0,121
0.967
0.829
0.357
GPT3 (Brown et al. 2020
0.072
0.882
0,855
0.416
0,077
0,936
0.832
0.366
GPT3 + Cisan Wer 2022
0.089
0,905
0.860
0.471
0.094
0.943
0.839
0,393
GPT3 M.L.MlP HHinng
123
0.931
0.894
0.539
0.1
0.946
0.842
0.401
GPT3 PLAN (Ours)
0.155
0,939
0.902
0,561
0,155
0,961
0,849
0,433
wroo Adaption
0.139
0.923
0.887
0.517
0.144
0.955
0.830
0.420
w// Symbolic
0.5
0,933
0.898
0.536
0,140
0.959
0.84,
0.414
Table 4: Automatic evaluation results on the Original RobotHow and WikiHow. Metrics are computed between the annotated programs and the predictions.
only solution, thus these automatic metrics provide limited absolute performance information. Details for the correlation between automatic metries and human evaluation can be found Section 4.5
Effects of Edge-wise Adaption and Symbolic Program Execution The variant "w/o o Adaption" maintains the top-k task-specific nodes ranked by the annotated weight Ew in the external knowledge base G without adaption. The variant "w/o Symbolic" directly takes the extracted concept nodes from external knowledge base as prompt. The performance drop of these two variants in Table 4 with significance test in Appendix C.2 demonstrate the importance of adaption and symbolic modules.
Effects of the Large Language Model Architecture We use GPT2 and GPT3 as autoregressive ar- chitecture and BART (Lewis et al. 2020) as autoencoder architecture. The autoregressive architecture achieves better results than the autoencoder one. Since the pre-training objective of autoregressive- based GPT is predict the next token given the previous input tokens. We assume the performance gain of GPT is due to smaller gap between the objective of pre-training and procedural planning.
Level of Complexity We show report results that use the test set which is separated into several buckets according to the number of steps in the procedural planning task. The step number reflects the difficulty of the task. In Table and Table jn Appendix C.2. we show that the averaged performance gain of PLAN over the baselines are consistent or more significant in more complicated procedural planning settings. This indicates the superiority of PLAN in solving long-horizon tasks.
4.4 RESULTS ON COUNTERFACTUAL TASK SAMPLES
We apply Initial Configuration, Intermediate Step, Final Goal interventions RobotHow and Intermediate Step on WikiHow. Human evaluations under counterfactual setting are summarized in Table 1 and Table 2 PLAN consistently outperforms baselines by large margin and experiences
8

---

Published as a conference paper at ICLR 2023
much smaller performance drop compared with the powerful baselines when switching to the counterfactual setting. We assume it's due to the biased knowledge of the holdout examples and manual exemplars utilized in the baselines, which are vulnerable to counterfactual samples. Automatic evaluations on counterfactual RobotHow are summarized in Table 13 in Appendix C.2 Aligned with human evaluations, PLAN achieves the best performance. The overall poor performance in Final Goal category indicates the challenge for long-horizon and composite procedural planning. While the overall better performance in Intermediate Step category benefits from the intermediate guidance.
4.5 CORRELATION BETWEEN AUTOMATIC AND HUMAN EVALUATION
We evaluate segment-level Pearson Correlation between human and automatic metrics. We observe that BERTScore has a moderate correlation to the human coverage score and WMD has moderate correlation to the human order score, with 23.3% and 32.3% respectively. Similar to the prior findings (Xu et al. 2021), D. n-gram-based metrics (Sentence-Ble BLUU and ROUGE) have relatively weaker correlation to the human coverage score, with a Pearson correlation of 16.4% and 21.1 1.1%. Overall, our automatic and human evaluation scores are consistent with the main claim of this paper. However, human evaluation is still irreplaceable for procedural planning at the current stage.
5 RELATED WORK
Procedural Planning Learning to generate procedural plan (Zhang et al. 2020a; Lyu al. 2021
Zhang et al. 2020b] Chang et al. 2020 Wu et al. [0022] Huang et al. (022) is important for embodied agen Tellex et al. (2011 1): Jansen (2020); Ahn et al. (2022) and conversational assistants (lliesski
ct al. 2018: Yang et al. 2022). Previous work views procedural script learning structured
form of commonsense knowledge Gupta et al. (2004); Regneri et al. (0110); Wanzare et al. (2016),
while more recent work strengthens its association with the changing environments for executable
action planning Puig et al. (2018); Shhrdhhar et al. (2020). Some works (Sun et al., 2020: Zhao et al.)
202T explore utilize human written programs precisely specify tasks. Our method tackles the problem with aware of cause-effect by utilizing commonsenseiinfuse prompts via a neuro-symbolic
approach (Ma et al., 2019; Nye et al. 2021 Yi et al. 2018 for zero-shot procedural planning.
Causality for Language Generation The integration of causality and machine learning has been an intriguing topic for many problems Pearl (2009); Scchlkopp (2022). Previous studies focusing on
causal inference for natural language understanding Chen et al. (2020); Keith et al. (2020): Wood-
Doughty et al. (2018) and generating counterfactual text representations Feder et al. (02T.. Weber
et al. (2020) proposes intervention method for script learning. However, these methods cannot be directly applied to procedural planning which requires a formal structure. Our method is based on
mediation analysis VanderWeele (2015)
and causal intervention Pearl (2009) Peters et al. (2017).
Prompt for Large Language Model There is an emerging interest in using prompts to extract
knowledge from large language models (Chen et al. 2022 Le Scao & Rush 2021: Su et al. 2022
Ye et al. 2022; Zhou et al. 2022, Kojima et al. 2022)
Cao el al. (2022) treats the prompt as a cause of the task-specific predictor and investigates biases prompt- based probing evaluations. Chain of
thought Wei et al. (2022) discovers that LLM can perform better on reasoning tasks when the prompt designed as series of short sentences that mimic the reasoning process of humans.
CONCLUSION AND FUTURE WORK
Procedural planning is newly emerged research area of great importance to various applications, such as household robots and virtual assistants. We propose a neuro-symbolic procedural PLANNer (PLAN) with commonsense-infuns infused prompts elicited from the external knowledge base to solve the procedural planning problem in a zero-shot manner without human annotated exemplars. Experi- ments show the effectiveness of our proposed PLAN under both origin and counterfactual settings, indicating the capability of mitigating spurious correlation by injecting external knowledge in LLMs. Though, procedural planning over long-ho morizon and composite tasks remains challenging. And exploring multimodal learning and developing human- evaluation metrics are promising future directions in this area.

---

Published as a conference paper at ICLR 2023
7 ETHICAL STATEMENT
Given the limited diversified cultural background of the dataset we are using from RobotHow and WikiHow, assume results may be biased toward single cultural background. For instance, given the task "make breakfeast", should take multi-culture into consideration to generate the procedural plans.
REPRODUCIBILITY STATEMENT
We provide more data samples and qualitative samples in supplemental materials. In addition, we provide our code implementation at reproduce experiments. The Preprocess folder provides the utils construct the data. The Eva Pivaluatin i on folder provides the code for automatic and human evaluation tools. The P Plannin Panning folder contains the main code for our approach and reproduced planners for procedural planning. The Aisualizati i on folder provides the code we use to visualize in the environment.
4open. www.oos.ooe...ccinneeeeeeeeeeeee science/r/LLANNER
to
ACKNOWLEDGMENTS
The research was sponsored by the U.S. Army Research Office and was accomplished under Contract Number W91 19-111---0000 for the Institute for Collaborative Biotechnologies. This work was also supported by the National Science Foundation award #20481 122. 22. We thank the Robert N.Noyce Trust for their generous gift the University of California via the Noyce initiative. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Gov- ernment is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169) 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerberttVosss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information P Processiig Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpss:/prrcceeigggs... cc 22//9ppr/222///De//4466666662 Btt.://T..........11111111112.2.
Boxi Cao, Hongyu Lin, Xianpei Han, Fangchao Liu, and Le Sun. Can prompt probe pretrained language models? understanding the invisible risks from causal view. ACL, 2022.
Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure planning in instructional videos. In Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI, pp. 334-350, Berlin, Heidelberg, 2020. Springer- Verlag ISBN 978-3-030-58620-11. 20. URL http://d...ogr.//////oo owg. 10. 1007
doi: 10. 00.10079778-3-00-5882222
Wenqing Chen, Jidong Tian, Liqiang Xiao, Hao He, and Yaohui Jin. Exploring logically dependent multi-task learning with causal inference. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2213 2213-2225, 2020.
Xiang Chen, Ningyu Zhang, Xin Xie. Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. Knowprompt: Knowledge-awee aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference 2022, pp. 2778-2782 2022.
10

---

Published as a conference paper at ICLR 2023
Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. Causalm: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333 - 2021.
Fernand Gobet, Peter C.R. Lane, Steve Croker, Peter C-H. Cheng, Gary Jones, lain Oliver, and Julian M. Pine. Chunking mechanisms in human learning. Trends in Cognitive Sciences, 5(6):236 243, 2001. ISSN 1364-6613. doi: http://00.0.20000000006066166600
URL https
7 www www.:/.cieneeiieeeoo rect com
Rakesh Gupta, Mykel Kochenderfer, Deborah Meguinness, and George Ferguson. Common sense data acquisition for indoor mobile robots. In AAAI, pp. 605-610, 2004.
Zhiting Hu and Li Erran Li. A causal lens for controllable text generation. Advances in Neural Information Processing Systems, 34, 2021.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint vrXiv2221.00700) 2022.
Vladimir Ilievski, Claudiu Cristian Musat, Andreea Hossmann, and Michael Baeriswyl. Goal-oriented chatbot dialog management bootstrapping with transfer learning. In IJCAI, 2018.
Peter Jansen. Visually-grounded planning without vision: Language models infer detailed plans from high-level instructions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4412-4417, Online, November 2020. Association for Computational Lin- guistics. doi: 10.18653/v 10t.://11222222/1 ht20://iii.n....ons.o.//e condinng-emnll.395 URL https: https//////aaaaoooooyyooo org/ 2020. findings-emmn 395
Katherine Keith, David Jensen, and Brendan O'Connor. Text and causal inference: A review of using text remove confounding from causal estimates. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5332-5344, 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2022. URL https://aaxxx/ wwwpxxiwwwggaa////ooe..o.a /205
11916
Mahnaz Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset. ArXiv, abs/1810.09305, 2018
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 957-966, ml www.s/rsssvv3
Lille, France, 07-09 Jul 2015. PMLR. URL htttps:/pproceed:: kusnerb15. html
Teven Le Scao and Alexander M Rush. How many data points is prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2627-2636, 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising
pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880 Online, July 2020. Association for Computational Linguistics. doi: 18.18653/v a22000/0//222.almmmmmm 703. URL https://ellattiooooy.oooo www.tthhoooo...r/22000 act. 703
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL wttpt:tp///.m ac antho wow.ooogy....oo ogy. -04- 1013
Qing Lyu, Li Zhang, and Chris Callison-Burch. Goal-oriented script construction. In Proceedings of the 14th International Conference Natural Language Generation, pp. 280, 2021.
11

---

Published as a conference paper at ICLR 2023
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro- Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In International Conference Learning Representations, 2019. 2019. URL https:://ppereeviee
net/fooumm r= hRctm
George A. Miller. The magical number seven plus minus two: some limits on capacity for processing information. Psychological review, 63 2:81-97, 1956.
Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. Improving coherence and consistency in neural sequence models with dusl-system, neuro-symbolie reasoning. Advances in Neural Information Processing Systems, 34, 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073033.007133... URL 1040)
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669-688, 1995.
Judea Pearl. Causality. Cambridge university press, 2009.
Douglas J. Pearson. Learning procedural planning knowledge in complex environments. In AAA//IAAI Vol. 2, 1996.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. Proceedings the Computer Vision and Pattern Recognition, pp. 8494-8502, 2018
the IEFE Conference
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. URL htttps://githuu com/ code ucas/
newspaper
Michaela Regneri, Alexander Koller, and Manfred Pinkal. Learning script knowledge with web experiments. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 979-988, 2010.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3982 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19. 1410. URL
Bernhard Schölkopf. Causality for machine learning. In Probabilistic and Causal Inference: The Works of Judea Pearl, pp.
2022.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. 2020 JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10737-10746. 2020.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'1,, pp. 4444 4451. AAAI Press, 2017.
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li. Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou On transferability of prompt tuning for natural language processing. Anmual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2022.
12

---

Published as a conference paper at ICLR 2023
Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In International Confer- ence on Learning Representations, 2020. URL https:://ppenrevieeetee net forum?in umdid
BkXUvnEYDH
Stefanie Tellex. Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp. 1507-1514, 2011.
Tyler VanderWeele. Explanation in causal inference: methods for mediation and interaction. Oxford University Press, 2015.
Lilian DA Wanzare, Alessandra Zarcone, Stefan Thater, and Manfred Pinkal. A crowdsourced database of event sequence descriptions for the acquisition of high-quality script knowledge. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'I6) 16), pp. 3494-344444 3501, 2016.
Noah Weber, Rachel Rudinger, and Benjamin Van Durme. Causal inference of script knowledge. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7583-7596, 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou.
Chain of thought prompting elicits reasoning in large language models. 2022. URL http
//arxiv.ore abs/ /aabb//2200 2201.119000
Zach Wood-Doughts Ilya Shpitser, and Mark Dredze. Challenges of using text classifiers for causal inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2018, pp. 4586. NIH Public Access, 2018
Te-Lin Wu, Alexander Spangher, Pegah Alipoormolabashi, Marjorie Freedman, Ralph M. Weischedel, and Nanyun Peng. Understanding multimodal procedural knowledge by sequencing multimodal instructional manuals. ACL, 2022.
Wenda Xu, Michael Saxon, Misha Sra, and William Yang Wang. Self-supervised knowledge assimilation for expert-layman text style transfer. In AAAI, 2021.
Shiquan Yang, Rui Zhang, Sarah Monazam Erfani, and Jey Han Lau. An interpretable neuro-symbolic reasoning framework for task-oriented dialogue generation. 2022.
Hongbin Ye, Ningyu Zhang, Shumin Deng, Xiang Chen, Hui Chen, Feiyu Xiong, Xi Chen, and
Huajun Chen. Ontology-enhanced prompt-tuning for few-shot learning. WWW, 2022.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B. Tenenbaum. Neural-symbolie vqa: Disentangling reasoning from vision and language understanding. In NeurlPS, 2018
Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Interventional few-shot learning. Advances in neural information processing systems, 33:2734-221 2746, 2020.
Li Zhang, Qing Lyu, and Chris Callison- Burch. Intent detection with wikihow. In Proceedings of the 1st Conference of the Asia-Pacify Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 328-333, 2020a.
Li Zhang. Qing Lyu, and Chris Callsoon-Burchh Reasoning about goals, steps, and temporal ordering with wikihow. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4630-4639, 2020b.
Tianyi Zhang*, Varsha Kishore* Felix Wu Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL http:htttt//peereeeeeee
net. tttt:/ooormmmi-sseettcoc
13

---

Published as a conference paper at ICLR 2023
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, and Jure Leskovec. Greaselm: Graph reasoning enhanced language models for question answering, 2022. URL
abs/ 2201. 08860
Zhongheng Zhang, Cheng Zheng, Chanmin Kim, Sven Van Poucke, Su Lin, and Peng Lan. Causal mediation analysis in the context of clinical research. Anmals of Translational Medicine, 4(21), 2016. ISSN 2305-5847. URL hRttps/://aaat amegroups. http//attleeeee//336d
Zelin Zhao, Karan Samel, Binghong Chen, and Le Song. Proto: Boogram-guided transformer for program-guided tasks, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. CVPR, 2022.
14

---

Published as a conference paper at ICLR 2023
Appendix
Table of Contents
A
SCM Theoretical Details
16
A. Causal Preliminaries
16
A.2
The Backdoor Adjustment
16
A.3
The Front-door Adjustment
17
B
Implementation Details
B.1
Original Dataset Details
19
B.2
Counterfactual Dataset and Experiment Details
................................
20
B3
Method Details
................................
20
B,4
Hyperparameter Search and Configuration Deicision
21
B.S
Computation and Resources
21
Evaluation Details
21
C.I
Crowdsoureing Human Evaluation
21
More Results
24
D Qualitative Examples
29
D.I Intermediate Output
29
D.2 Predicted Procedural Plans
29
E
Discussion
34
F.1
Limitations
34
1.2
Failure Analysis
34
E,3 Ethical considerations
34
15

---

Published as a conference paper at ICLR 2023
A SCM THEORETICAL DETAILS
A.1 CAUSAL PRELIMINARIES
The Structural Causal Model (SCM) is directed acyelic graph (DAG) to describe the causal relationships within a system Pearl 2009). In this paper, we refer to the unrolled SCM along the time dimension as the full temporal causal graph, while the rolled-up version is also called the causal summary graph Peters et al. (0017) In an SCM, if the variable D is a cause of both T and S, then it called a confounder. A confounder opens up a backdoor path and causes a spurious correlation between 7 and S, The backdoor path is defined the remaining path between 7 and S; when all the arrows pointing out of T are removed. Therefore, T <- D + S: is backdoor path. For our SCM with mediator P; shown in Figure 4c (same as Figure 2b from the main paper, there is no backdoor path between T' and ( S because only D + is left after removing outgoing arrows of T. On the other hand, there is backdoor path between P. and S,, Le. P. t- T t- D - S; so that P; indirectly affects the observation of S; through {7. - and D. The mediator is the variable added between treatment variable (the cause 7 and S in our case) and treatment variable (the effect S; in our case), and thus blocks all directed path from the cause to effect ( (Zhang et al.) 2016)). The spurious correlations happens when two variables are statistically related but causally related because of a third variable influences these two variables at the same time or the correlation is coineidental.
To identify the true causal effect between X and Y. we aim to estimate the conditional TY [{]]do(X) (X)( after intervention with the do-operator. The do-operator to break the backdoor path by setting to fixed value independent of Z. Then the path Z - X can be removed to eliminate the backdoor paths. In practice, the backdoor adjustment and front-door adjustment are two fundamental methods to implement interventions and obtain the conditional +T ()))))))
Clarity the Definition As language prompt, P; inherits the content from P:-1 and thus can be detached from steps before - for simplicity.
Causal Intervention There are two types of operation to control the confounding bias: the backdoor adjustment and the front-door adjustment (Pearl 2009). The backdoor adjustment is intractable in our case because it requires the prior distribution the confounding variables. On the other hand, can construct input prompt mediator P. for T -> S, and S - S.. Then the front-door adjustment applies a two-step do-operation to mitigate bias by investigating P - S (Pearl 2009). Specifically, we construct the prompt mediator P, using techniques illustrated in Section 2.2
The pre-trained knowledge (D) LLMs confounds language models to make biased decisions toward unreasonable action. Since the confounder is unobservable, intervention techniques such as back-doo (definition in Appendix A.2) adjustment (Hu & Lij 2021 Weber et al.) 2020)) Yue al.) 2020) are not applicable in our SCM. Instead, commonsense-infuseed prompt. Through the mediator, we can identify causal effects among goals and steps by investigating the indirect effect from the goals, which is essentially the front-door adjustment (definition in Appendix A.3) in causality (Pear 2009).
build a mediator and implement it
A.2 THE BACKDOOR ADJUSTMENT
The backdoor adjustment is one way to realize the intervention do(7 = () by considering the conditional probability over the existing data distribution with observed confounder D.. Let n; denote (-1P;-1) that represent the probability density function conditioned on P It calculates the average causal effects by considering all stratums of the dataset:
7. T((S(do(T)) = Σ n. ==
(5)
However, for LLMs, the pretraining data is usually unobservable and has been transformed as knowledge incorporated into the hidden space. Therefore, we are not able to directly apply the backdoor adjustment.
16

---

Published as a conference paper at ICLR 2023
D
D
St
T
Pi
S
S
P
S
(b) The SCM and three backdoor paths at timestep i >
(c) Equivalent SCM at timestop
(a) SCM at timestep =
i after eliminating P;-1
Figure 4: The front-door Adjustment for Causal Procedural Planner. (a) the structural causal model at timestamp = 1. 'I' denotes the task name and S₁ denotes the step at timestep 1. D is the unobservable confounding variable introduced by the pre-training data. P1 denotes the mediating variables we construct to mitigate the spurious correlation at timestep 1. (b) D opens up backdoor paths for T 5,, P:-1 > S; and 5;-1 path 2 share the same path D T. Intervention on I blocks D - 'I' and the backdoor path 2. Intervention on S blocks D + 1 and the backdoor path (c) the structural causal model at timestamp > 1 after simplification based Equation 12 16
which can blocked by introducing P. path and
A.3 THE FRONT-DOOR ADJUSTMENT
The front-door adjustment is another technique to apply intervention by introducing a mediator P. when the confounder is unobservable. As is explained in Section 1.0.oom22.mmmmmm... from the main paper, the front-door adjustment equivalent to two consecutive do-operations on task and prompt P.. We first investigate the generation of S1 and then expand it to St.
Timestep i ==1 As is shown in Figure 4aj since there is no preceding steps, the first step generation involves D and P1 only. Similar to the proof in Section 2.2 from the main paper, have:
(((i(do(T)) = Σ T (Siidoo(P = p))
(6)
= Σ m<(plT) Σ (Spp. (T=;(==))
By adding intervention to T, we make the value of do(T 1) independent of the confounder D at eliminated as a result.
the beginning. The backdoor path through D
Timestep i As is shown in Figure 2a from the main paper, model the mediator Pı as effect of three variables, T, P and S 1- The first step of our front-door adjustment is to apply the do-operator the three variables and observe the change in P. explained in Section 2.2 from the main paper. Since there are no backdoor paths between P. and these variables, we have the probability after intervention equal the conditional probability without intervention:
Ti P: pl)) (P: pl7)
(PP = (P_o)())))) T; = P3-1-)
m; (P: = pldo(S,-1)) Ti(P: =p(S:-1
(9)
The second step is to apply do-operator on P. and then identify the causal effect as:
क (S.(d((()))
Σ tp
(S. P:-1 =p', 1,,11-1 =
(10)
TT = -
Combining Equatior and Equation 10 we have the front-door adjustment. Note that there are three backdoor paths from each of the variables T, P:-1, and Si-1, 1, is shown in Figure 4b (drawn
17

---

Published as a conference paper at ICLR 2023
D
(T
D
T
Pi
S1
Task-relevant sampling
Ti (P1 |do (T)) π, (P11T)
D
do (P1)
T
P1
S1
T
P1
S
Adaption & Symbolic Structuring
Σ
({
π. (S
(a) SCM at timestep 1
D
D
(T)
P
Si
Task-relevant sampling
S
IT do())))))),o(((--))) = (P.ITT,S
D
T
Pi
Si
do (P:)
T
Adaption & Symbolic Structuring
Pr
S
Si-1
S
77 (S,Ido
n.
tp
(b) The SCM timestep
Figure 5: The Causal Graph after do-operation. (a) the causal graph transition of Structural Causal
Model at timestamp 1. (b) the causal graph transition of Structural Causal Model at timestamp i 1.
18

---

Published as a conference paper at ICLR 2023
in blue, red and purple). More importantly, the one through T, i.. P. P D S. (the blue path in Figure 4b) and the one through P,-1, i.c. P, < P.-1 + + - St (the red path in Figure (4b) shares the same subpath. The intervention on the task T breaks the backdoor paths for both 7' and P Therefore, we have our front-door adjustment as
T. (S())do(Ss PP P--1), PP-1)),d(TT)
(11)
Σ (c(S: -)
(12)
Σ (T( do (P. p) 1)
(13)
Σ (T( do(P: (p)
(14)
Σ T. (p|s T)
Σ -, s, s,T=t)
(15)
P
8,t
= ny
(T)))
(16)
We have Equation 13 because of the intervention on T and Rule 2 (Pearl 1995). Equation 14 because
of Rule 1 (Pearl 1995). After simplification based on Equation 12 16 we get the SCM at timestep
é >> 1 in Figure 4c This is an equivalent SCM after eliminating P. 1 Figure 4b The reason could eliminate P. is as follows. We follow a common method of constructing temporally-extendee prompt, which is to append the prediction at previous timesteps the prompt current timestep. In our case, the Pa.: is the same as Pa.i-1, thus P; inherit part of the content from P:-1, the change only depend on the S/-1- Thus P:-1 and S are fixed, and there no need to predict P.- at timestep i again. In this way, we simplify the causal graph in Figure to the one in Figure In summary, we define and simplify the causal graph based on the temporal-extended property of our prompt construction (P; inherit the content from P;-1). We end up with Equation 14 16 which is shown as Equation 3 in Section 2.2 from the main paper.
B IMPLEMENTATION DETAILS
B. ORIGINAL DATASET DETAILS
RobotHow This dataset is mons License. We evaluate the inference of 150 tasks by random selection from the dataset. Each program contains the task name, task description and steps. We use the task name and sequence of steps as our input and output references.Eas Each step is a composition of and [Number]] J. For example, the sequence of steps of the task "Watch TV" are: 1. [Walk] <TELEVISION>> (1) 2. [SwitchOn] <TELEVISION> (1) 3. [Walk] <SOFA> (1) 4. [Sit] <OFA>> (1) 5. [Watch] < TELEVISION> (1).
4.0 International Creative Com-
[Object
WikiHow This da under an Attribution Noncommercial- -Sare Alike 3.0 Creative Commons License. And the text content is free modify, republish and share. We evaluate the inference of 1000 tasks by random selection from the dataset. The admissible action space and interaction object space are more complex than the programs in RobotHow. And there is no fixed "[Action] ;Object; (Number)" form of each step. For each article, it contains the title, the bold headlines and text. We utilize the title and headlines our task name and steps respectively.
External Knowledge Base For the external knowledge base, we utilize ConceptNet to leverage commonsense reasoning ability to help ground language generation in goal-guided procedural text generation. ConceptNet (Speer et al.) 2017) captures commonsense knowledge explicitly with triplets of (head node, relation, end node). contains 799, 273 nodes and 2, 487, 810 edges that represent both symmetric and asymmetric relations. Specifically, the core relations we utilized are Synonym, AtLocation, CapableOf. Causes, CausesDesire, HasPrerequisite, HasSubevent, and UsedFor. Since we are looking the commonsense knowledge in housc-holding tasks, so filter out the relations (I//Distinttrrom, /t/DerivedFrom, /r/SymbolOf, fiSSymbollO.///Etymooooollllll h/tAyyyooogiclllylllaeeelllll InEtymoolgiclllyyeeveeFooon that are related to the linguistic.
https: : /ww. wi wwt.:.wwkihw..oo..oo how.com
19

---

Published as a conference paper at ICLR 2023
B.2 COUNTERFACTUAL DATASET AND EXPERIMENT DETAILS
Category
Description
Examplc
Initial Configuration
Constrain the environment configuration, e.g. location
Watch TV in bedroom
Intermediate Step
Constrain the way to finish the task
Work (Find Computer)
Final Goal
Change the final effect of the task by composition
Watch youtube and Put away jackets
Table 5: Three Types of Counterfactual Procedural Planning, Three types of methods, including initial configuration, intermediate step, and final goal applied to intervene the original procedural data.
Category
Original Program
Counterfactual Program
Task: Watch TV
Task: Watch TV bodroom
Step I: Find remote control,Step 2: Grab remote control.
Step I: Walk to bedmom Step 2: Find remote control.
Step Find lelevision Step Switch on television.
Step Grab remole cuntrol.Step Step Find television.
Initial Conliguration
Step 5: Turn to television. 6. Watch television.
Step S: Switch on television Step 6: Turn to television.
Step Switch off television.
Step Watch television.Step 8: Switch television.
Step 8: Put back remote control
Step 9: Put back remote control
Task: Work
Task: Wark (Find Computer)
Step Walk to home office office.Step 2: Walk to chair.
Step Walk home oflice.Step Walk to chair.
Intermediate Step
Step Find chair.Step 4; Sit on chair.
Step Find Stair 4: Sit on chair.
Step S: Find Sompoter.Step 6; Switch on computer.
Step 5: Find computer.Step 6 Switch on computer.
Step Turn computer.Step Look computer
Step Tum to Computer.tepp Look at computer
Task1: Turn light off
Task: Turn light off and Clean
Step 1: Walk to bedroom Step 2: Walk to light
Step Walk badroom Step Walk light
Step Switch off light
Step 3:
Switch off light
Task2: Clean
Step Walk home office
Final Goal
Step Walk to home office Step 2: Walk to rag
Step 5: Walk to mag Step 6: Find rag
Step Find rag Step Grab rag
Step S: Walk to desk Step 6: Find computer
Step 7: Grah rag Step 8: Walk to desk
Step Wipe computer Step 8: Wipe
Step Find computer Stop 10: Wipe computer
Step 9: Put back rag
Step II: Wipe desk Step 12: Put back rag
Table 6: Comparison between Standard and Counterfactual Procedural Planning, Three types of methods, including initial configuration, intermediate step, and final goal are applied intervene the original procedural data.
show the examples that compare the original program and the counterfactual program of each intervention method are also provided. Specifically, for Initial Configuration, we randomly append the location to given task name to constrain the location of completing the task. The steps are prepended with the initial step "walk to [Locationg". For Intermediate Step, we randomly sampled a step from the task-specific program and append it to the task name constrain the way to implement a given task. For Final Goal, we randomly combine two tasks by combining both the task names and the programs construct set of long- nonghoorzzon composite tasks.
We conduct counterfactual experiments by applying randomly selected intervention methods over RobotHow. And we only apply the Intermediate Step intervention method over WikiHow due the loose configuration requirement and the long text of the WikiHow contents. Note that the performance gain of PLAN under the counterfactual setting mainly comes from the additional guidance of the task introduced from the Intermediate Step intervention method. However, the baselines mostly experience performance drops due the limited annotated exemplars. PLAN consistently outperforms baselines by a large margin, indicating its superiority under the counterfactual setting.
B.3 METHOD DETAILS
The existing formalization of the procedural planning task can mainly categorized as 1) sequential choice making (Lyu al. 2021: Wu et al. 2022, Zhang et al.] 2020a b), which reasons about the next step from the options given, the task, and previous steps; 2) conditioned generation (Huang et al.
2022 Ahn et al. 2022). which generates the temporally extended plans to implement the task. We
study the procedural planning task as the conditioned generation problem (Huang et al., 2022 Ahn
et al. 2022) since it resembles real-world scenarios.
Baselines LLMaP propose a procedure to extract temporally extended plans from large pre-trained language models. Chain explores manually creating exemplars that mimic the reasoning process
20

---

Published as a conference paper at ICLR 2023
and uses them to prompt large language models for reasoning tasks. To compare with Chain on the procedural planning task, we manually generate exemplars that contain the chain of thought for 1% of the inference task programs. Note that for the BART language model, we use BART-large version. And we use the 1.5 billion parameter GPT-2 (aka gpt2-xl). For the translation model LMT, we use contence-rransfommer (RoBERRaalargge. All these models released by HuggingFace. In addition, our experiments with GPT3 (davinci) use OpenAI API (May, 2022).
External Knowledge Graph Conceptnet5 define a of 34 relations Within the relations consider in the procedural planning task, the averaged sampling time of subgraph sampling is 0.03576 milliseconds per task program.
B.4 HYPERPARAMETER SEARCH AND CONFIGURATION DEICISION
We perform a hyperparameter search for all evaluated methods for the following hyperparameters.
The confidence threshold 0, which terminate the generation when below it, is searched in {0, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8).
The steps horizon, which constrains the maximal number of procedural planning steps, is searched in {10, (10,20,40 40).
The number of hops for retrieving the subgraph from the external knowledge base searched in (1,
The ratio of maximal concepts to the length of the task name is searched in {1, 3).
The cosine similarity threshold for keeping the task-specific concept is searched in {0.4, 0.6, 0.8}.
The edge weight threshold de is searched in 00.1,0.2,0.33 0.4, 0.5, 0.6, 0.7, 0.8). The top-k task-specific nodes value searched in (1,5, 10, 15, 20, 25, 50, 100).
The configurations used in the experiments are: 0=0.7, 20 step horizon, hops, ratio of concepts to task length, cosine similarity threshold 0.4, 0.=0.6 and k=10.
We empirically choose the hop number as 3 considering both the input length limit of the LLMs and the fact that 3-hop contains reasonable relevant information in practice (Zhang et al. (022)
B.5 COMPUTATION AND RESOURCES
We use one single NVIDIA A100 GPU Server for all the experiments. Since there is no training in zero-shot settings, the computation is only used for the inference stage of the experiments.
C EVALUATION DETAILS
C.1 CROWDSOURCING HUMAN EVALUATION
We conduct all the human evaluations (rating and win-lose comparison) on Amazon Mechanical Turk platform. Each example is rated by 3 annotators. We ask Amazon Mechanical Turk workers, for every assignment, to evaluate the quality of the provided low-level steps given the high-level task description. For the Win-Lose Comparison, they were asked choose one from the two provided model generated results by 1:the first one is better, 2:equal and 3:the second one is better. For the Human Ratings, they were asked score each sample with 5-point Likert scale. This process does not involve collecting any personal information. And we manually check no offensive content is produced by the models.
The assignment layout templates for workers are shown in Figure 7 and Figure Specifically, evaluate randomly selected task examples from each dataset (RobotHow and WikiHow) under all the settings (standard and counterfactual). We only collect the examples that the workers read the instructions carefully by checking whether they give score for the empty program as a sanity check. The hourly wage paid to participants is estimated $9. And the total amount spent participant
https: httts///..tth...oommmmmmmeeee//s ki
21

---

Published as a conference paper at ICLR 2023
compensation is $1296. The details of the Human Intelligence Tasks process described in the following sections.
Entar Propertiea
2 Design Layout
Previoew and Finish
Requesar:
Ranaarhhe
Revard: $6.13 per bak.
Takks ava Aible
Quratees: Minuta
Quiiitatirs Required:
Instuciions
Shorloado
Wieutt sogaono urvera rear and anrononea to poo laggt, lunk?
Select an option
given and the secuence stepe, determine which sequanne covers more stsps that necassary camplete the tar task. Jonore sequential ordar stops.
Tle
Sequencce
$(task1_tr) txt)
Sequence 1: S/stsppsl_,ttt
Sequence 2: S(stepe2_bd)
Scaliia Required:
Instmaciions
Siorttouts
Whith seguence of steps can butter complese in to glvan toak?
Select option
1 Sequence better
Read the given task and the sequence of stops, determine which set of stops san better complete target tazk.
2 Tie
In other words, can the task be decomposed into these stepss
3 Sequence better
S(teak1 txl)
Sequence 1: S/steps1 [_44]
Sequence 2: 3/stpp22
Figure 6: Amazon Mechanical Turk Platform. Questions Layout for Human Raters for Comparison.
C.1.1 WIN-LOSE COMPARISON
During the process of Human Intelligence Tasks, the workers are shown the following instructions: Read the given task and the sequence of steps, determine which set of steps better complete the target task. In other words, can the task be decomposed into these steps? Please consider the sequential order of the steps.
Then the program to be evaluated is provided as:
Question Task: Study
Sequence 1:: Step Walk to textbook Step 2: Read book Step 3: Walk to book
Sequence 2:: Step 1: Walk home office Step 2: Find desk
Finally, the workers are asked to score the program by following the instructions below: Select an option: / Sequence I better; 2 Tie; 3 SSequence 2 is better
22

---

Published as a conference paper at ICLR 2023
Propartics
Design Layout
Proviem and Fintah
Fiza qailly asscuaico siig
Teaks
For queetion below. determine whethhe the completed reasonable scenario using the provided (Please the
You stepe? Ignove seop decompood
steps.)
quastion. Please just see your first intuition determine the ardwarrs. 12) If you' re not sure what standard follow, you mey scroll through the queetions first. Once you ve set your standarde, pleese abice by them all the qu essionns for the purpose fair comparisons.
(1) correct
Question
$/task1
Bequence of Steps: $(slepa1 txl)
Use the slider below to indicate how much you agree with the following siatement (1 Strongly disagree, 5 Strongly agree), If "sequence steps" are blank, please direct chocse (lowest score).
be completed in any mesonable acanario using provided steps.
Enter Properties
Dasign Layout
Previow and Finish
Hesartee
Hewert: 30
Teato avaliible
Darater:: tuk
Gualtiredinns Reepired: New
For every question below, determine whether the task somplcted any reasonable scenarto using the provided stops (Please consider the senuectial order of the store.) You could direcdly give loweet score (1) for the emply steps. othar words, can tha fack be decompased Into these staps? (Plasse corsidar the sequential arder of the stops.)
Additional Notes:
There correct anewer each question. Please you're
first intuition to deternine the snewers. standard follow, may through purpose fair comparisons.
standards,
Question
$taakk1. txt)
Sequence of Steps: $isteps
Use siider below to indicate how much you agroe with the following statemont (1 Strongly disagres, - Strongly agros). "sequence of stepss are blank, plesse directly choose 1 (lowest score).
The task can be completed in any rassonable scanario using the provided stops.
Figure 7: Amazon Mechanical Turk Platform. Questions Layout for Human Raters for Point Likert Scale.
The above example is to evaluate the order metric, for the coverage metric. the same process are conducted, except for the instructions are: Read the given task and the sequence steps, and determine which sequence covers more steps that are necessary to complete the target task. Please ignore the sequential order the steps.
23

---

Published as a conference paper at ICLR 2023
1 Orter Propetion
Dwoggg Leost
Preniie and Finin
For every quatton balon, determaks stutter th task amn conmieeed anannaable aomnado tou rusudded slens aocnssoosssso.oooooooooooooo basa stash stanna Press seep that The prssedum pana connct sointy commoneante cor the sonr h........... "ear tolk" socepfat it, soore chis bwirdd
bigher
Addiliond Netwe: (1) la sornett euch quentoon Phease jast usw your mustonn deterrine answans stamdard
atondarcts, abile tham one
Question
Btaok1 (sl]
Sequeroce Staps:
bekow dizzal how hollwcing stmmesed Strungyy
dirccly cheose RECM anvicement...
sucoms cor sirted provided srcedded staouaatle
Figure 8: Amazon Mechanical Turk Platform. Questions Layout for Human Raters for Point Likert Scale on Success Rate.
C.1.2 HUMAN RATINGS
Similar as the Win-Lose Comparison Human Intelligence Tasks, the workers are shown the following instructions: For every question below, determine whether the task can be completed in any reasonable scenario using the provided steps (Please consider the sequential order of the steps.). You could directly give the lowest score for the empty steps. In other words, can the task be decomposed into these steps? (Please consider the sequential order of the steps.)
Then the program to be evaluated is provided as:
Question Task: Write an email
Sequence of Steps: Step 1: Walk to home office Step 2: Walk to computer Step 3: Find computer
Step 4: Turn to computer Step 5: Look at computer Step 6: Walk to computer Step 7: Find chair Step 8: Sit on chair Step 9: Find keyboard Step 10: Grab keyboard Step 11: Find mouse Step 12: Grab mouse Step 13: Type on keyboard
Finally, the workers asked to score the program by following the instructions below: Use the slider below to indicate how much you agree with the following statemens (1 = Strongly disagree, Strongly agree). If "sequence of steps" are blank, please directly choose (lowest score). The task can be completed in any reasonable scenario using the provided steps. [SLIDER PROVIDED HERE]
The above example to evaluate the order metric, for the coverage metric, the same process is conducted, except for the instructions For every question below, determine whether the lask can be completed in any reasonable scenario using the provided steps (Please ignore the sequential order of the steps.). You could directly give the lowest score (1) for the empty steps. In other words, can the task be decomposed into these steps? (Please ignore the sequential order of the steps.)
C.2 MORE RESULTS
Significance Test We provide paired-t test (p;0.05) statistics results for Table On RobotHow, PLAN significantly outperforms all baselines
and Counterfactual GPP2)) On WikiHow, our PLAN significantly outperforms all baselines on Original- CorR,, GPT2), Counterfactual-Covenn Cove-oageeerrrAR GPT2), and Counterfactuall-Oddr(AAAAAA Order(BART)) For the coverage metric under the counterfactual setting, the numan--rovidee program is not signifi- cantly better than our PLAN.
We also conduct the paired-t test 0.:0.5 statistics results over the variant "w/o Adaption" and "w/o Symbolie". Compared with the full model PLAN,, the variants experienced statistically significant
24

---

Published as a conference paper at ICLR 2023
Model
RohotHow
Step Backet
S-BLFU
WMD
BERT-11
ROLIGE-
Covengge
Order
Step Avg.
Time Cost (ms)
BART + Chain
Wei al. 2022
(10,20)
(0,10)
D,073
0.499
0.909
0.915
0.857
0.863
0,432
0,442
2,921
2.947
2.825
2.760
13.714
6,600
3,820
3.330
[0,10)
0.867
0,450
2,973
BART + LLMaP Houng et
(10,20)
0.028
0.931
0.853
0.393
3.095
2,889
13.714
3.366
BART + PLAN
(0.10)
0,099
0.955
0.894
0,532
3.187
3,013
6,600
3.272
(10,20)
3.079
4.02
GPT2 Chain
Wes ct 2022
(10,20)
[0,10]
0.776
0.87
0.877
0.856
0.845
0.359
0.395
2,453
2.365
2.147
2,476
13.714
3,804
GPT2 LLMaP
Huung 2022
[0.10)
0.112
0.912
0.894
0.186
3.147
2,987
6,600
3.212
(10.20)
0.859
GPT2 PLAN
(10,20)
[0.10]
0.167
0.101
0.940
0.924
0.901
0.882
0.554
0.480
3.173
2.984
2,813
3.063
13.714
6,600
3,954
3.344
GPT3 Chain
Wes
2022
(10,20)
(0.10]
0.086
0.920
0.881
0.878
0,445
3.562
3.568
3,459
6.838
3.215
GPT3 + M.M Huang at
2022
(10,20)
[0.10)
0.139
0.132
0.951
0.939
0.911
0.894
0.502
0.541
3.811
3.531
3,625
3,486
13.688
6.838
3.144
3.964
GPT3 PLAN
[0.10)
0.171
0.961
0.918
0.574
3,459
3,568
6.838
3.379
(10.20)
3.750
Table 7: Evaluation results on the Original RobotHow by separating test set into several Step Bucket.
WikiHow
Model
Step Backet
S-BLEU
WMD
BERT-F1
ROUGGE-F1
Covernge
Order
Step Avg.
Time Cost (ms)
BART Chain Wei al. 2022
(10.20)
(0,10)
0,033
0.032
0.921
0.919
0.784
0.789
0.356
0.294
2.644
2,969
3,311
3,521
14.467
6.156
7.349
8.233
BART + LLMaP Houng et
2023
(10,20)
[0.10]
0.088
0.02
0.944
0.924
0.814
0.794
0.353
0.293
2,600
2,802
3,488
3.178
14.467
6.156
8.289
7.487
BART PLAN
(10,20)
[0.10)
0.108
0.939
0.834
0.812
0,431
0.372
2.978
3,083
3,594
3.244
14.467
6.156
8.341
GPT3 Chain
Wes ct
2023
(10,20)
[0,10)
0.107
0.077
0.933
0.817
0.812
0.353
0.328
2.733
3.0R1
3,422
11,167
7.585
366
GPT3 MaP
Huange
2022
(10.20)
[0.10)
0.111
0.916
0.831
0.342
0.36
3.292
2,978
3.625
6.156
8.218
GPT3 PLAN
(0.10)
0.136
0.961
0.856
0,416
3.645
6.677
8
(10,20)
0.127
0.961
0.868
0.458
3.68
3.2
13.6
7.632
GPT3 Chain
Wei
(10,20)
(0,10)
0.125
0.954
0.837
0.432
0,465
3/421
3.655
3.517
6.0
8.424
GPT3 MaP
Huang et al. 2022
(0.10)
0,146
0.956
0.865
0.514
3.652
3.739
6.565
7.953
(10,20)
0.099
0.951
0.849
0.452
3.375
3,312
15.0
7.247
GPT3 PLAN
(0,10)
0.203
0.969
0.506
3,31
3,643
5.81
8.101
(10.20)
0.855
3.714
Table 8: Evaluation results on the Original WikiHow by separating test set into several Step Bucket.
performance drop. Especially on BERTScore-fl, the p-value is 8.884e - 13 and 1.4e 8 respectively. This further confirms the importance of the modules.
Results on GPT-3 In addition, we conduct experiments with GPT-3 (davinci version) using OpenAl API. We showcase the comparison in Table 9 and Table 10
25

---

Published as a conference paper at ICLR 2023
Model
Program
RobotHow Task: Write an Email
Step I: Walk to home office. Step 2: Walk to computer. Step 3: Find computer. Step 4: Turn to computer.
Step 5: Look at computer. Step 6: Walk to computer. Step 7: Find chair. Step 8: Sit on chair.
Human
Step 9:: Find keyboard. Step 10: Grab keybourd. Step 11: Find mouse. Step 12: Grab mouse.
Step 13:
Type on keyboard
Chain
Empty plan prediction due to low confidence for the first step.
LLMaP
Step I: Switch on computer.
PLAN
Step I: point at mail. Step 2: envelope on printer.Step 3: put notes on printer.
RobotHow Task: Turn on light
Human
Step Walk home office. Step 2: Walk floor lamp. Step 3: Switch on floor lamp
Chain
Step Switch on light switch.
LLMaP
Step 1: Switch on light switch.
PLAN
Step I: Switch on light. Step 2: Switch on lamp. Step 3: Turn to light switch. Step 4: Switch on light switch.
RobotHow Task: Drink
Human
Step Find water glass. Step 2: Grab water glass. Step 3: Drink water glass.
Chain
Step Find coffee cup.
LLMaP
Step
Find cup. Step 2: Drink cup. Step 3: Pour cup into glass.
PLAN
Step 1: Grab gluss. Step 2: Pour water into drinking glass. Step 3: Drink water.
RobotHow Task: Play games
Step 1: Walk to bedroom. Step 2: Walk to video game console. Step 3: Find video gume console.
Step 4: Walk to video game console. Step S: Switch on video game console. Step 6: Find television.
Human
Step 7: Switch on television. Step 8: Find video game controller. Step 9: Grab video game controller.
Step 10: Find chair. Step 11: Walk to chair. Step 12: Sit on chair
Chain
Emply plan prediction due to low confidence for the first step.
LLMaP
Step I:
Switch on console. Step 2: Put console on dvd player.
Step 1: Find video game console. Step 2: Grab video game controller. Step 3: Put console boardgame.
PLAN
Step 4: Look at video game console.
RobotHow Task: Put groceries in Fridge
Step I: Walk to dining room. Step 2: Walk to food food. Step 3: Find food food.
Human
Step 4: Grab food food. Step 5: Find freezer. Step 6: Open freezer.
Step 7: Drop food food
Step 1: Walk to fridge. Step 2: Find groceries. Step 3: Find fridge.
Chain
Step 4: Open fridge. Step 5: Put groceries in fridge. Step 6: Close fridge.
LLMaP
Step 1: Open fridge. Step 2: Pull groceries. Step 3: Put groceries on blender. Step 4: Close fridge.
PLAN
Step I: Walk to fridge. Step 2: Open fridge. Step 3: Put groceries in fridge. Step Close fridge.
RobotHow Task: Admire art
Step Walk home office. Step 2: Walk to drawing. Step Turn to drawing. Step Look drawing. Step 5: Find drawing. Step Turn to drawing. Step Look drawing.
Human
Chain
Step Walk drawing.
LLMaP
Step Look painting.
Step Look at centerpicce. Step Put centerpiece on music stand. Step 3: Point at painting. Step Look drawing. Step S: Touch centerpicce. Step Look at painting.
PLAN
Step Put centerpiece on love
How Task: Greet guests
Step Walk dining room.Step Walk Shild..tep 3: Find child. Step Grect child.Step Find woman.Step Grect woman.
Human
Step Find chair.Step Sit chair
Chain
Empty plan prediction due low confidence for the first step.
LLMaP
Step
Open door. Step 2: Close door:
PLAN
Step
Walk to dining room. Step 2: Walk to entrance hall. Step 3: Greet woman.
Table 9: Showcases of procedural steps predicted by different models with GPT3 as the base LLM on RobotHow.
26

---

Published as a conference paper at ICLR 2023
Model
Program
WikiHow Tusk: How to Become an Art Investor
Step I: Start with some experience or interest in art.
Step 2: Understand the difference between collectors, art investors and speculators.
Step 3: Figure out what you willing to pay for art, before going to auction house.
Step 4: Pay atlention to whal schools of art are selling well, and which are down.
Step 5: Focus urt investrennts on fine art paintings, ruther than decorative art.
Humun
Step 6: Reach out to trusted auction houses and dealers when you are looking to buy art.
Step 7: Buy your investment art when you feel confident its worth, price and its ability grow in value.
Step 8: Study how art is properly stored.
Step 9: Have your art investments appraised occasionally.Son Step 10: Consider renting out your art investments.
Step 11: Understand that selling an art investment can take time.
Chain
Step 1: Buy your investment art when you feel confident of its worth, its price and its ability to grow in value.
LLMaP
Step 1: Reach out to trusted auction houses and dealers when you are looking to buy art.
Step 1: Figure out what are willing to pay for art, before going to auction house.
PLAN
Step 2: Consider renting out your art investments.
Step 3: Bay your investment art when you feel confident its worth, its price and its ahility to grow in value.
WikiHow Task: How to Be an Organized Artist
Step I: Make sure you know what is expected of you. Step 2: Stick to your topic.
Step 3: Don't try to be to funny unless the scenario calls for
Step 4: Act naturally
the situation; talk, act and Step 5: Participate. Step Don't react what others say yourself. stay churacter.
your character would usually do the circumstances.
Human
Step 7: Don't make anything violent or too crazy. Step 8: Relax and enjoy yourself.
Step 9: Be your character. Step 10: Play games that allow you to practice improvisation.
Step 1: First, you will need to make sure you have all of the materials listed below.
Chain
Step 2: Set schedule, Step 3: Create a comfortable space.
Step 4: Take notes in journal or sketchbo. Step Keep neat and tidy. Step 6: Take break.
LLMaP
Step Make plans.
Step 1:
Start with some experience or interest in art. Step 2: Put together schedule and chart.
Step 3:
Prepare create your neopoprealist mural. Step 4: Organize your computer-hasee materials. lave clear plan.
PLAN
Step 5:
Step 6: Buy your investment art when you feel confident of its worth, its price and its ability to grow in value.
Step 7:
Work on being the best you.
WikiHow Task: How to Be Good at Improvisation
Step 1: Keep related supplies in the same area.
Step 2: Make an elfort to clean a dedicated workspace after every session.
Step 3: Place loose supplies in large, clearly visible containers.
Step 4: Use clotheslines and clips to hang sketches, photos, and reference material.
Human
Step 5: Use every inch of the roum for storage, especially verticul space.
Step 6: Use chalkbourd paint to make space for drafting ideas right on the walls.
Step 7: Purchase a label maker to make your organization strategy semi- memi-pemmment.
Step 8: Make a habit of throwing out old, exeess, or useless stuff each month.
Chain
Step 1: Play games that allow you to practice improvisatio.
LLMaP
Step 1: Don't Io overlo it.
Step 1: Try the spontancous approach.
PLAN
Step 2: Express yourself creatively. Step 3: Play games that allow you to practice improvisatio.
Step 4: extracurricular activitie.
WikiHow Task: How to Train a Parrot to Say Something
Step 1: Decide what you want your parrot to say, but make basic.
Step 2: If you want, you can make say simple but funny things.
Human
Step 3: You should go nice and quiel
Step 4: To start teaching it, repeut what you want to say many times.
Step 5: If you DO get your parrot to say it correctly, then you's ve sucoceded!
Chain
Step 1: Decide what you want your parrot to say, but make it basic.
LLMaP
Step 1: If you do gel your parrot to say it correctly, then you'se 've succeeded.
Step 1: Decide what you want your parrot say, but make it basic.
PLAN
Step 2: If you do get your parrot to say it correctly, then you'le 'e succoeded.
Table 10: Showcases of procedural steps predicted by different models with GPT3 as the base LLM on WikiHow.
27

---

Published as a conference paper at ICLR 2023
Model
Origiial-Fxeeutabilll
Chenterfactual-Feeetttt
Exeembbllyy
Counterfactul--xxrrttta
Chain Wes et 202
3.16
3,50
3.32
3,58
Ma Triming 2022
3,50
3.88
1.42
3.74
PLAN (0B3)
3,4
3,90
4,02
3,84
Table 11: Averaged 5-point Likert scale human evaluations on Success Rate aspect with GPT3 language model architecture.
RobotHow
WikiHow
Model
S-BLEU
WMD
BERT-FI
ROUGE-FI
S-BLEU
WMD
BERT-FI
ROUGE-F1
GPT3 PLAN (Ours)
0.155
0.939
0.902
0,561
0.155
0.961
0.849
0.433
w/o Adaption
0.139
0.923
0.887
0.517
0.144
0.955
0.830
0.420
w/o Symbolic
0.135
0.933
0,898
0.536
0.843
0.414
w/o First Translation Model
0.126
0.932
0.894
0,534
0.146
0,948
0.836
0,417
Table 12: Automatic evaluation results for additional ablation on the Original RobotHow and
WikiHow. Metrics are computed between the annotated programs and the predictions.
Motivation of Evaluation Metrics Since the nature of the procedural planning task can be open- domain in that the golden plans may not unique. This leads to the challenge that common automatic metrics proposed in natural language task not perfect to evaluate procedural planning. The same observations of such challenge to directly judge the system using automatic metrics are discussed in LLMaP(Huang al., (022) well. We assume that the human evaluation Cover rage and Order can reflect how well the procedural plans are close to human annotated program, because the human annotators are required to determine whether the task be completed in any reasonable scenario using the procedural plans explicitly. Thus we provide both the automatic evaluation and human evaluation on two aspects Coverage and Order, with description in the Metrics paragraph in Section 4.1
Evaluation on Success Rate Metric To make human evaluations more intuitive, we provide an additional Success Rate metric to show whether the procedural plans can successfully implement the task, which focus more on the success rate instead of the coverage or the order of the plans. We show the Success Rate evaluations on the baselines and method in Table I1 The assignment layout template for workers is shown in Figure g
To verify the contribution of the first translation language model LMr that trans- lates the knowledge prompt Pa into admissible one Pc. we conduct an additional ablation experiment by simply removing the first LMr and replacing P& with Pg to prompt the LLM for procedural planning. We provide results with comparisons to other ablations in Table 12
More Ablation
Results on Counterfactual Task Samples RobotHow in Table B
We show automatic evaluation results on counterfactual
Configuration
Step
S-BLLEU
WMD
BUKE111
ROUGEE-T1
S-BLEZU
W.D
BUKEIT
ROUGBE-1
S-BLEU
WXD
BBKE11
RUUGL-11
Choin (oo 20127
0.125
0.905
0,375
0.518
0.136
0.926
0.892
0,550
0,063
0.918
0.857
0.467
LiMall d 2023
0.148
0.929
0.367
0.566
0.141
0.886
0.002
0.547
0.070
0.928
0.968
0.400
PLAN (001))
0L860
0.934
0,897
0.570
0.183
0.953
0.913
0.590
0,082
0.934
0,873
0,493
Table 13: Automatic evaluation results the Counterfactual RobotHow with language model GPT2.
28

---

Published as a conference paper at ICLR 2023
D QUALITATIVE EXAMPLES
D.1 INTERMEDIATE OUTPUT
We provide running examples with intermediate output for each module in the following paragraph. First, we show the intermediate output of input task T, the subgraph G, depicted in the tuple of the start node, relation type, tail node and edge weight, the knowledge prompt Pa and the translated one Pg as below:
Input task T: Take shower.
Human-annotater Plan Reference: Step I: Walk to bathroom. Step 2: Walk to clothes dress. Step 3: Find clothes dress. Step 4: Put off clothes dress. Step 5: Find shower. Step 6: Enter shower. Step 7: Find soap. Step 8: Grab soap. Step 9: Scrub soap. Step 10: Put back soap. Step 11: Leave shower. Step 12: Find towel. Step 13: Grab towel. Step 14: Wipe towel.
Step 15: Find clothes dress. Step 16: Put on clothes dress.
Task-relevant subgraph G, (NNend,, Re: Ntait, Em:): (take shower, HasLastSubevent dry off, 6.0); (bathe, Husseettueevent, dry off, 6.0); (take a shower, HasPrerequisite, take out your clothes, 4.47); (take shower, HasSubevent, get clean, 4.47); (take a shower, HasPrerequisite, take your clothes off, 3.46); (go to a party, HasPrerequisite, take shower, 2.82); (play lacrosse, Hasl astuubeette take shower, 2.82); (get clean, HasPrerequisite, take shower, 2.82); (take a shower, MotivatedByGoos sports, HasbestSueevent, take a shower, 2.82); (go to the hairdresser, HasPrerequisite, take a shower, 2.82); (take a shower, HasPrerequisite, turn on the water, 2.0); (have bath, HasLastSubevent, dry off, 2.0); (get wet, HasSubevent, dry off, 2.0): (become more clean, HasLastSubevent, dry off, 2.0); (take shower, HasSubevent, wash your hair, 2.0); (take shower, Hasl asttuubeete, turn off the water, 2.0); (become more clean, Hasl asttubeeee,, dry off, 2.0); take a shower, Hasl LassLaaSSueeeet put your clothes on, 1.0); (take a shower, HasSubevent, use shampoo, 1.0); (take a shower, HasSubevent, wash behind your ears, 1.0); (take shower. HasSubevent, wash your body, 1.0); (take shower, Has HasPrerequisite, go to the bathroom, 1.0); (take a shower, HasPrerequisite, go to the bathroom and undress, 1.0); (take a shower, HasPrerequisite. step into the shower, 1.0); (take shower, HasPrerequisite, soap up, 1.0); (take a shower, HasPrerequisite, bring some soap. 1.0); (take a shower, HasPrerequisite, bring some shampoo, 1.0); (take a shower, Has aatSubveeee towel yourself off, 1.0); (take a shower, HasPrerequisite, remember to take your towel, 1.0);
wash your hair, 2.82); (play
Knowledge prompt Pc: Step: take out your clothes. Step: set clean clothes. Step: go to the bathroom. Step: go to the bathroom and undress. Step: take your clothes off. Step: turn on the water. Step: bring some soap. Step: bring some shampoo. Step: remember to take your towel. Step: get clean. Step: wash your hair. Step: use shampoo. Step: wash behind your ears. Step: wash your body. Step: turn off the water. Step: dry off. Step: become more clean. Step: put your clothes on.
Translated knowledge prompt Pa: Step: find clothes underwear. Step: wash shower. Step: walk to bathroom. Step: look at shower. Step: take off clothes underwear. Step: wash shower. Step: find soap. Step: grab soap. Step: scrub shower. Step: find shampoo. Step: grab shampoo. Step: find towel. Step: wipe shower. Step: wash hair. Step: wipe shampoo. Step: scrub shower. Step: wash body. Step: switch off washing machine. Step: scrub shower. Step: wipe shower. Step: put on clothes underwear.
Generated plan Sy: Step 1: Find clothes underwear. Step 2: Walk to Bathroom. Step 3: Take off clothes shirt. Step Scrub shower. Step 5: Pour shampoo into hair. Step 6: Wipe shampoo. Step 7: Wipe hair. Step 8: Wash body. Step 9: Find Towel. Step 10: Put on clothes underwear.
D.2 PREDICTED PROCEDURAL PLANS
More qualitative examples of final predicted procedural plans that are randomly selected are provided. Table [4] show random samples on the original dataset. Table [5] show random samples on the counterfactual datasets with the Intermediate Step intervention method. And Table shoss random
29

---

Published as a conference paper at ICLR 2023
samples the counterfactual RobotHow with the Initial Configuration and Final Goal intervention methods.
30

---

Published as a conference paper at ICLR 2023
Model
Program
RobotHow Task: Play Games
Step I: Walk to hedroom.Stepp Step 2: Walk to video game console.
Step 3: Find video game console.Step 4: Walk to video game console.
Step 5: Switch on video game console.Step
Step 6: Find television.
Human
Step 7: Switch on television.Ste Step 8: Find video game controller.
Step 9: Grab video game Sontroller.Step 10: Find chair.
Step 11:
Walk to chair. 12: Sit on chair
Chain
Step
chef knife on water glass Step 2: Find sink.
Step I:
Walk to bedroom Step 2: Walk to video game console.
LLMaP
Step 3: Find video game console
Step I:
Walk to video game controller.Step 2: Find video game controller.
PLAN
Step 3: Switch video game controller.Step 4: Find tv.Step 5: Switch
WikiHow Task: How to Be an Organized Artist
Step 1:
Keep related supplies in the same area...
Step 2: Make an effort to clean dedicated workspace after every session..
Step 3: Place loose supplics in large, clearly visible containers...
Step 4: Use clotheslines and clips to hang sketches, photos, and reference material...
Human
Step 5: Use every inch of the room for storage, especially vertical space..
Step 6: Use chalkboard paint make space for drafting ideas right on the walls...
Step 7: Purchase label maker to make your organization strategy semi cemi-permament...
Step 8: Make habit of throwing out old, excess, useless stuff each month.
Step 1: Feel
dense, kinky coat.
Chain
Step 2: Look for rounded cars with crimped inner hairs.
Step 1: Make sure that all money has been covered on both end.
Step 2: Check your bill and banking statements regularly.to ensure that your bills are getting paid as they should be, you should check all statements that you receive.
LLMaP
Step 3: Keep track of your expenses.Step 4: Keep track of your income.
Step 1: Prioritize your goals
workshup. 2: Prepare to create your neopoprealist mural. Step 3: Organize your computer- materials.Stup 4: Have clear plan.
Step 5: Work on being the best you.
Step 6: Paint patterns with brushes of suitable size for the particular portion of work you are painting.
PLAN
Step 7: Put together a schedule and chart.
Step 8: Buy your investment art when you feel confident its worth, its price and its ability to grow
in value.
Step 9: Make a clean brea.
Step 10: First, you will need to make sure you have all of the materials listed below.
WikiHow Task: How to Use Saddle Soap
Step 1:
Read the manufacturer's directions., Step 2: Avoid lye heavy products., Step 3: Make your own saddle soap.
Human
Step 1: Give her safe space.Step 2: Let him exercise.
Chain
Step 3: Be kind to him.Step 4: Offer him treat.
Step 1: Dip damp cloth the saddle soap.
Step Dampen several cloths and work them into lather with the saddle snap.
Step 3: Dip a cloth into the mixture.Step
Rub the product on with cloth. Step Clean the grime off with the Steth Dampen a cloth with water.
LLMaP
Step Rub damp microfiber cloth over the surface Step 8: Use damp cloth to wipe once again.
Step 9:
Dampen towel with lukewarm Sater. Wipe the area clean with damp towel. Step 11: Soak a towel in warm water.
Step 1: Brush your htrse.Step 2: Clean the saddle.
Step 3: Start by using the first cloth to rub down the leather, making sure you remove any built
grass and dirt.
PLAN
Step 4: Rub the leather down so you remove most of the oil and wax.
Step 5: Wipe off excess oil. 6: Rinse the leather.
Step 7: Clean
other leather tack items (martingale, breastplate. draw reins, cruppers. etc.
Table 14: Showcases of procedural steps predicted by different models with GPT2 the base LLM.
31

---

Published as a conference paper at ICLR 2023
Model
Program
RobotHow Intermediate Step
Turking (Find computer.).
Step 1: Walk to home office.
Step 2: Walk to desk.
Step 3: Find chair.
Human
Step 4: Sit on chair.
Step Find computer.
Step 6: Switch on computer
Step 1: Switch on computer.
Chain
Step 2: Switch off computer.
LLMaP
Step 1: find computer.
Step 1: Walk to chair.
Step 2:
Turn computer.
PLAN
Step 3:
Look at computer. Switch off computer.
Step 4:
Step Switch on computer.
WikiHow Intermediate Step
Task: How to Become an Art Investor (Figure out what you are willing to pay for art, before going to an auction house.).
Step 1: Start with some experience or interest in art...
Step 2: Understand the difference between collectors, investors and art speculators...
Step 3: Figure out what you are willing to pay for art, before going to an auction house...
Step 4: Pay attention what schools
are selling well, which are down...
Step 5: Focus art investments on fine art paintings, rather than decorative art...
Human
Step 6: Reach out trusted auction houses and dealers when you are looking to buy art...
Step 7: Buy your investment when feel contident its worth, price and its ability grow in value,,.
Step 8: Study how art is properly stored...
Step 9: Have your investments appraised occasionally...
Step 10: Consider renting your art investments...
Step 11: Understand that selling an art investment can take time.
Step 1: See being sold directly by artists by clicking "direct from artist" link.
Step 2: Look at the price.
Chain
Step 3: Look at the writer's website.
Step 4: Create portfolio of your work.
Step 5: Look at price aguin.
Step 1:
Create sketch the neopoprealist manner future mural on small piece of paper 8"x10" using the black ink pen.
Step 2: Prepare to create your neopoprealist mural.
Step Prepare your paint.
Step 4: Regin your project with design.
Step 5: Produce scaled down version of your finished mural.
Step 6:: Prepare the wall be painted.
LLMaP
Step 7: Allow the background and base coats to dry.
Step Draw
the appeared section different repetitive patterns (examine the images above).
Step 9: Paint patterns with brushes of suitable size for the particular portion of work you are painting.
Step 10: Clean up the lines and shapes as needed.
Step 11: Seal the mural needed.
Step 12: Be inspired and it will help you succee.
Step This where you get
most creative.
Step 1: Figure out what you willing pay for art, before going to auction house.
Step 2: Sell your art online.
Step Go the auction
Step 4: Buy it.
first floor..reecat steps 1-4 to add more floor.
Step 5: Install
PLAN
Step 6: Enter in the second round values.
Step 7: Get the carat diamond.
Step 8: Know the four es of diamonds.
Step 9: Find a reputable diamond appraiser your area.
Table 15: Showcases of procedural steps predicted by different models with GPT2 as the base LLM under counterfactual setting (Intermediate Step).
32

---

Published as a conference paper at ICLR 2023
Model
Program
RobotHow Initial Configuration
Task: Play games in dining room.
Step I:
Walk to dining room.Step 2: Walk to video game console.
Step Find video game console.Step 4: Walk to video game console.
Step 5: Switch on vidco game console.Step Step 6: Find television.
Step 7: Switch television 8: Find video game controller.
Human
Step 9: Grab video game controller.Step 10: Find chair.
Step 11: Walk chair.Step 12: Sit chairTask: Play games in dining room.
Step Walk to video game controller.Step Put video game controller on diningtable.
Step 3: Put boardgame on kitchen table.Step 4: Put boardgame on diningtable.
Step 1: Walk to dining room.Step 2: Walk to sauce pan.
Chain
Step Pour bottle water into dish bowl.Step 4: Walk water.
Step 5:
Walk to carrot.Step 6: Walk to food salt.
LLMaP
Step 1: Walk to Kitchen.Step 2: Walk to food food. 3: Walk to dining room.
Step I:
walk to dining room.Step 2: find diningtable.Step 3: walk to board game.
Step 4: put video game controller on diningtable.Step 5: put food snack diningtable.
Step 6: put board game on diningtable.
PLAN
Step 7: put boardgame
diningtable.Ste Step 8: put board game kitchen table. Step 9: find video game console.Ster 10: put glass on board game.
Step II: grab video game controller.Step 12: put glass on boardgame.
RobotHow Final Goal
Task: Turn light and clean
Step Walk to bedroom.
Walk to light.Step Switch off light. Step 1: Walk to home office.Step 2: Walk to rag.Step 3: Find rag.
Human
Step Grab rag. Step 5: Walk desk.Step 6: Find computer.
Step 7: Wipe computer.Step 8: Wipe desk.Step 9: Put back rag.
Step 1: Walk to kitchen.Step 2: Walk to cooking pot.Step 3: Walk to water.
Chain
Step 4: Walk to dishwasher.
LLMaP
Step 1: Put light bulb on bowl.Step 2: Switch off light bulb.Step 3: Switch on light.
Step 1: plug out lighting. Step 2: put cleaning solution desk.
PLAN
Step 3: find dish stap.Step 4: scrub light switch.
Step 5: wipe lighting.
Table 16: Showcases procedural steps predicted by different models with GPT2 as the base LLM under counterfactual setting (Initial Configuration, Final Goal).
33

---

Published as a conference paper at ICLR 2023
E DISCUSSION
E LIMITATIONS
Though pointing out a direction to prompt out actionable knowledge in large-scale pre-trained language models with external commonsense knowledge, the limitations of reasoning long-horizon procedural plan still exist. Existing datasets for procedural planning like WikiHow and RobotHow are all monolingual supporting only English goals and plans. In the future, it is important to expand these datasets or having novel datasets that support multiple languages used across the world. The inherent difference between these languages may also result in different planning strategies in granularity or abstraction levels, which is potentially challenging. In addition, the long- long-horizon and complex composite tasks still remain challenging for the existing procedural planners.
Above limitations are discussed mainly based on the challenges of procedural planning task. In addition, there limitations of our implementation that are guided by causal analysis. First, the coverage of the leveraged external resources is limited, which is common in a knowledge-enhance system. This may result in the wrong understanding of the task and produce not reasonable procedural plans. For example, the knowledge of the word "Turking", which refers to "Thh act or process of performing small tasks using the Amazon Mechanical Turk service." according Wiktionary, not covered in the external resources (e.g.. ConceptNet). Since our proposed system does not assume specific external resources.
plausible in the future if we utilize more powerful external resources (e.g., Wiktionary). Second, the hop number and the threshold of the multi-hop retrieval in task- relevant subgraph sampling is currently a configured hyperparameter. This may result in not ideally constructed prompt. The future work could instead make these hyperparameters learnable on each task domain, and also explore the pros and cons between end-to-end commonsense-infused prompt versus neuro-symbolic constructed prompt.
E.2 FAILURE ANALYSIS
We discuss detailed failure modes and examples with analyses below. For example, the predicted procedural plan on task "Turking", which refers to "The act or process of performing small tasks using the Amazon Mechanical Turk service." according Wiktionary. We compare the predicted procedural plan on this task among baselines and our method: (1) The ground truth plan is "Task:
Turking. Step 1: Walk home office.Step Walk to desk. 3: Find chair.Step Sit on chair.Step 5: Find computer.Step 6: Switch on computer" (2) The plan predicted by Chain baseline is empty. (3)
The plan predicted by LLMaP baseline is "Task: Turking. Step 1: Put teddybear on oven." " (4) Our prediction is "Task: Turking. Step 1: Eat food turkey. Step 2: Drink water. Step 3: Sleep." We can see that for the "out-of- -ittoo--knowledee" task, our method also lead failure planning. We assume this is mainly due to the limited knowledge in external resources, discussed in the Appendix E..]] and this main failure mode can be avoided by introducing larger external resources (e.g, Wiktionary), similar as other enowledge-enriched methods.
E.3 ETHICAL CONSIDERATIONS
We hope to de-bias the procedural planning to avoid misleading either humans robots with daily life instructions, which may result unsafe situations. The cultural bias behind these datasets can be critical issue for future work. As the ground truth planning steps usually reflect the culture shared by the English-speaking group. other cultures may have a completely different practical consideration that leads to different orders of these steps or even novel steps that are not proposed by the LLMs we utilized in this paper. In the future, we will consider cultural bias as a proxy variable so that we could adjust the implicit knowledge from LLM commonsense from external sources according to the different needs of cultural backgrounds.
34


Reviews:
[{"review_id": "wr4E4tUUXs", "reviewer": "Reviewer_i21n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work shows that by adding a dash symbolic reasoning to a neural model, it shows better performance w.r.t consistency and generalization. It is unclear to me how much engineering efforts are required to add these symbolic reasonings. It appears their approach is general, using a common External Knowledge Base, but the writing is confusing and I cannot tease out the details.", "summary_of_the_review": "Overall, the paper is difficult to read, and as a result I cannot judge it properly.", "strength_and_weaknesses": "strength : the proposed approach works. they conduct a user study where they ask crowd workers to rate which agent, one with symbolic reasoning and one without, performed better on a task, and the crowd workers preferred the agent with symbolic reasoning. this result is solid and shows evidence of the proposed approach.\n\nweakness : \n\nthe proposed method is may not be entirely novel. people have been adding symbolic reasoning to neural models for awhile, and the finding has always been : \"If we can successfully 'hack' the underlying DSL that represented the set of tasks, adding symbolic reasoning would perform well\". For instance, these works tend to follow the steps of: 1) identify a set of tasks that would be easily represented with symbolic execution, and 2) devote significant engineering efforts to construct the DSL and a symbolic interpreter to help the neural/llm model make better inferences/plans. \n\nthis work would be of significant contribution if it can show that steps 1) and 2) can be avoided by using a generic external knowledge base (as shown in figure 3). however the writing is too confusing I cannot be sure if that is the case or not.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g4aAedSQ036", "reviewer": "Reviewer_W99u", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a principled algorithm for incorporating symbolic information related to text generation using an LLM. The idea is justified using a causal analysis, so the algorithm is motivated by the front door criteria. The external information is extracted by entity extraction and extracting related information from ConceptNet. The results are concatenated to the prompt and interpreted as conditioning. The human evaluation and metrics indicate the method leads to improvement in three different LLMs.\n", "summary_of_the_review": "The paper provides a causality-justified algorithm for retrieving information about a task. Although some points seem to break the causal argument, I suspect this is indeed the explanation for better performance. Some aspects of the causal analysis are hard to follow.", "strength_and_weaknesses": "Strengths\n- Principled motivation using causality\n- Concrete technical solutions challenges for integrating with the selected source of information\n- Ablation study shows the impact is not trivial.\n- Idea is novel as far as I know.\n\t- For instance, this survey appeared after the submission. It does not mention work as specific as this submission: Feder, Amir, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, et al. “Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond.” Transactions of the Association for Computational Linguistics 10 (October 2022): 1138–58. https://doi.org/10.1162/tacl_a_00511.\n\nWeaknesses\n- It's not clear which is the current causal model used at each point. (See question below)\n- The paper might be assuming P_i is the only path, but the existence of other paths to be blocked it's not discussed. \n- The argument loses a bit in the adaptation plus other decisions on what to retrieve.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aDMUKHJVdX", "reviewer": "Reviewer_ai1E", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary:\n- Existing large language models (LLMs)  require manual exemplars to acquire procedural planning knowledge in the zero-shot setting. \n- The paper proposed a neuro-symbolic procedural PLANner (PLAN) with commonsense-infused prompts elicited from an external knowledge base (i.e., ConceptNet) to solve the pure-language-based procedural planning problem in a zero-shot manner. \n- Human and automatic evaluations on WikiHow and RobotHow show the superiority of the proposed PLAN over the prior methods on procedural planning. \n", "summary_of_the_review": "The paper proposed to generate commonsense-infused prompts elicited from an external knowledge base (i.e., ConceptNet) to allow LLMs to solve the pure-language-based procedural planning problem in a zero-shot manner. The key idea of this paper is novel and solid. The proposed method has a strong performance.", "strength_and_weaknesses": "Strength:\n1. Solid and reasonable idea: Based on the observation that due to potentially biased pre-training data, pre-trained knowledge in LLMs may confound the model to make wrong decisions when asking the model to generate a procedural plan of a task, the authors proposed to apply the frontdoor adjustment in causality to build a mediator and implemented it as a commonsense-infused prompt. The prompt obtained from their neuro-symbolic-based method allows the LLM to attend to the causal entities instead of the highly correlated ones for the next step generation. \n2. Strong performance: the proposed method PLAN outperformed two recent SOTA methods LLMaP (“Language models as zero- shot planners”) and Chain (“Chain of thought prompting”) statistically significantly. \n3. Quite thorough experimental study and analysis: for the experiments, the authors utilized several evaluation methods including human evaluations, two datasets, and several pre-trained LLMs. \n4. The paper is well written and organized.\n\nWeaknesses:\n1. Discussion on the failure cases is currently missing. In addition, the generated procedural plan of the proposed method was shown, but it would be interesting and useful for readers to see the exact intermediate outputs of the proposed framework given an actual task from the evaluation dataset, e.g., $G_s$, $P_G$, $\\hat{P}_G$, etc. In this way, readers may have a better understanding of the current capabilities of each module of the proposed framework.\n\n2. Some minor issues, e.g.:\n(1) “Note that in the path T →D→Si ←Pi, Si is a collider…” (second line of the “Stage1” paragraph), should it be “T←D”?\n(2) It would be good to refer readers to the appendix for definitions of backdoor path, frontdoor adjustment, etc.\n(3) How is the Symbolic Rule Set $R$ obtained? \n(4) One more ablation experiment: what if removing the first Translation $LM_T$ and replacing $\\hat{P}_G$ with $P_G$?\n(5) Suggesting one relevant work: https://arxiv.org/abs/2205.11916 (manual exemplars not required).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "UIj62BWhtr", "reviewer": "Reviewer_UGe5", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a procedure planning model which enforces the logical order. The authors claim that they can parse the goal to subgraphs and translate the knowledge into admissible knowledge. The proposed model also learns casual relationships via an SCM for procedural planning. Experimental results show the proposed model surpasses the original baselines by a large margin.", "summary_of_the_review": "Addressing an important task with promising novel technics. Writing needs to be improved, and more insights should be given. \n-------------------------------------\nAfter reading the responses, I think most of my concerns are addressed. Therefore I raised my score.", "strength_and_weaknesses": "## Strengths:\n1. The procedural planning task is interesting and fundamental. Through procedural planning, we can understand how the neural networks understand the task structure and generate the corresponding plans. The counteract cases are also interesting and touch the core of AI.\n2. The source code is provided, so this work might be reproducible.\n3. The idea of using causal models to prevent relational biases and keeping order in LLMs is very important and is introduced with good examples.\n4. The improvements in the experiments seem to be large. I like the illustration of the results introduced in Table 5. Sadly, some results are omitted in Table 5. On the contrary, I suggest the paper should have a large table showing the concrete procedural results to help understand each part of the model. Try always to be concrete and precise.\n\n## Weaknesses:\nI am confused by this paper's methodology a lot of times. In general, this paper could be challenged in the following positions:\n\n1. Probably, the proposed model is not technically sound. In fact, many of the components are not well-defined, and the system looks too complicated. The overall model is not ablated well and is full of tricks.\n2. The proposed model seems to borrow a lot of big concepts, such as neural-symbolic or casual models. However, the specific technical contributions are not very clear, and many problems remain unaddressed. In general, the system looks too complicated. This is not a good paper. A good paper should stick to one major point and show the advantage of this point over previous baselines.\n3. Human evaluations are not very objective and cannot be easily reproduced. I fail to see a clear motivation for using expensive human evaluations. Instead, showing a lot of generated procedures is far more beneficial than showing a lot of numbers. \n\nThe specific comments are:\n\n1. In the task definition, what are the actions, and what are the object sets? I think in real natural language, the set of actions (described in natural language) or objects can be extremely large. How can the formulation handle this? I did not see the specific usage of these variables. Probably the problem is over-formulated.\n2.  The authors use the same variable D for task D_T and confounding variable D. This is hard to decode.\n3.  I can hardly understand the implementation of the SCM. What are the input and output of the SCM? Is there any replacement (e.g. standard transformers) for this module? What is the sample output of the SCM? Is there any reference to the SCM? Take Figure 2 as an example; how does the SCM forward the reasoning path? How does the SCM encode the action and object? How does the SCM produce the output? These problems are still unclear to me.\n4. Why does procedural planning need so many steps (five-stage pipelines)? I see that motivation is to fuse different motivations (task definition, previous steps, and external knowledge). If so, why not encode this information in parallel so that the fused (e.g. concatenation) information can be used in downstream tasks?\n5. Algorithm 1 is a waste of space, and it is better to replace it with a figure (or make figure 3 better).\n6. Many concrete designs, such as the symbolic executors, are missing. It seems that the symbolic executors are trivial and can be easily learned. In other words, I do not think the \"neural-symbolic\" stuff in the title plays an important role in this system. In fact, what is neural symbolic in this line of work? Why is it important? How to connect the big concept with the experimental results with the LLMs? These questions are far more important to show than the ad-hoc model designs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MUX63gdA7Rl", "reviewer": "Reviewer_UWNS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an approach for procedural planning with LLMs and casual models. The approach first builds a commonsense casual model from an external knowledge base with adjustments. Then, given a query, it builds a task-relevant subgraph, which is provided as a procedural prompt. Finally, this is translated to admissible actions for the LLM to plan. This is demonstrated on two large planning tasks, RobotHow and WIkiHow, and shown to outperform baseline planners. ", "summary_of_the_review": "The paper presents and interesting and novel approach to an impactful area of research. PLAN also shows performance on two large and varied environments. However, the paper is quite dense and could be improved in clarity. The results show PLAN outperforms baselines, but by a small margin given the additional complexity, and the results do not include success rate metrics that may be most crucial for understanding the gains in performance.", "strength_and_weaknesses": "\nThe paper is interesting and the topic is very relevant. Planning from LLMs is promising and grounding them in common sense as well as admissible scenes is a core challenge in this area.\nThe approach of building a large knowledge base and leveraging within an LLM is well founded.\nThe two large planning datasets are large and good environments to test in. The results show improvements across the board compared to baselines.\n \nThe paper however has a few areas for improvement. \n\n(1) The clarity could be improved. Some sections like Section 3 are quite dense and difficult to parse, particularly Section 3.1. Perhaps an earlier overview could help clarify or explicit running examples. Section 3.1 could use it’s own algorithm block and potentially a zoomed in figure of the computation. Figure 3, though a nice overview, is very dense.\n\n(2) Though the performance of PLAN is stronger than baselines, it is a smaller improvement than I would think given the additional complexity and also somewhat difficult to judge how large of an improvement it is. In many of the metrics it seems PLAN outperforms by a few percentage points or in voting it wins 50% of the time. While I acknowledge that this shows that PLAN is performing better, it isn’t clear from these results that it is worth the vast additional complexity compared to baselines. The authors should add an additional metric similar to executability in Huang et al., showing the actual success rate of these plans, as this is the ultimate metric we care about. A few related questions:\n* I would be interested to learn more about what the main failure modes are.\n* Some results, such as Table 1, seem to outperform baselines with larger models. I’m surprised by this as I would think your approach would be particularly important to add structure when LLM’s are more inaccurate. Do you have any intuition on why this might be?\n\nMinor notes:\n* “The Termination Condition is either reaching the max step t or the matching score is below threshold θ. “ Instead of thresholding, one can compare to an end of statement token’s probability.\n* Mention in the intro where the common sense external knowledge comes from (though I know it is in Figure 1).\n* How do you extract entity names from the task name?\n* “show that pre-trained knowledge (D) in LLMs confounds” What is D here?\n* “We describe the implementation of such frontdoor adjustment in Section 3.1.” but section 3.1 is 2 pages long, be more specific.\n* Table 3 bolds 0.433 in the button right, though GPT3 + Chain outperforms it with 0.471\n* “PLAN surpasses powerful baselines (Chain of Thoughts (Wei et al., 2022) and Zero-shot Planner (Huang et al.)) by large margins on both the original and the counterfactual samples.” Are these baselines powerful? They don’t have external information except their prompts.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%