PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning
Authors: Faeze Brahman
Chandra Bhagavatula
Valentina Pyatkin
Jena D. Hwang
Xiang Lorraine Li
Hirona Jacqueline Arai
Soumya Sanyal
Keisuke Sakaguchi
Xiang Ren
Yejin Choi
Venue: ICLR 2024 poster (2024)
Keywords: language-based planning
procedural/script knowledge
distillation
large language models
decoding-time algorithm
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a novel two-pronged approach to endow small language models with procedural knowledge and (constrained) language-based planning capabilities through knowledge distillation and an inference-time algorithm.

Abstract: Procedural planning, which entails decomposing a high-level goal into a sequence of temporally ordered steps, is an important yet intricate task for machines. It involves integrating common-sense knowledge to reason about complex and often contextualized situations, e.g. ``scheduling a doctor's appointment without a phone''. While current approaches show encouraging results using large language models (LLMs), they are hindered by drawbacks such as costly API calls and reproducibility issues. In this paper, we advocate planning using smaller language models. We present PlaSma, a novel two-pronged approach to endow small language models with procedural knowledge and (constrained) language-based planning capabilities. More concretely, we develop *symbolic procedural knowledge distillation* to enhance the commonsense knowledge in small language models and an *inference-time algorithm* to facilitate more structured and accurate reasoning. In addition, we introduce a new related task, *Replanning*, that requires a revision of a plan to cope with a constrained situation. In both the planning and replanning settings, we show that orders-of-magnitude smaller models (770M-11B parameters) can compete and often surpass their larger teacher models' capabilities. Finally, we showcase successful application of PlaSma in an embodied environment, VirtualHome.

Published as a conference paper at ICLR 2024
PLASMA PROCEDURAL KNOWLEDGE MODELS FOR LANGUAGE-BASER PLANNING AND RE-PLANNING
Faeze Brahman 12 Chandra Bhagavatula
Valentina Pyatkin 1f Jena D. Hwang 1t
Xiang Lorraine Li 15
Hirona J. Arai
Soumya Sanyal Keisuke Sakaguchi Allen Institute for Artificial Intelligence
Xiang Ren
Yejin Choi 12
University of Washington
UUniversity Southern California *Tohoku University
University of Pittsburg
Faezeb@allenai.co
ABSTRACT
Procedural planning, which entails decomposing a high-level goal into sequence of temporally ordered steps, is important yet intricate task for machines. It involves integrating common-sense knowledge to reason about complex and often contextualized situations, e.g. "scheduling doctor's appointment without phone". While current approaches show encouraging results using large language models (LLMs), they are hindered by drawbacks such costly API calls and reproducibil- ity issues. In this paper, we advocate planning using smaller language models. We present PLASMA, novel two-pronged approach to endow small language models with procedural knowledge and (constrained) language planning capabilities. More coneretely, we develop symbolic procedural knowledge distillation enhance the commonsense knowledge in small language models and an inference-time algorithm facilitate more structured and accurate reasoning. In addition, we introduce new related task, Replanning, that requires a revision of plan to cope with a constrained situation. In both the planning and replanning settings, we show that orders-of -fgmnnii smaller models (770M-1 1BB parameters) and often surpass their larger teacher models' capabilities. Finally, we showcase successful application of PLASMA in an embodied environment, VirtualHome."
compete
INTRODUCTION
Powered by massive scale, large language models (LLMs) excel on many downstream tasks that require commonsense. One such task is procedural planning (Schank & Abelson, 1975b; Pearson & Laird, 2005), a task that involves decomposing a high-level goal into a sequence of coherent, logical, and goal-oriented steps (plan) (c.g. "see movie" "Look up movic showings", "Choose movie" ...). Recent approaches model this task as a conditional language generation problem using LLMs (Madaan al., 2022; Huang et al., 2022: Ahn et al., 2022; Zhao al., 2023). Despite their reasonable performance on the task, their steep computational cost and inaccessibility to models' parameters hinder the wider adoption of LLMs (OpenAI, 2023) for procedural planning.
We present PLASMA (PLAn with SMAll models), a novel framework and model to impart procedural knowledge and language-based planning abilities in small LMs.² In the first phase of the framework, we enhance the implicit commonsense knowledge in small LMs through symbolic procedural knowledge distillation (West al., 2022: Bhagavatula et al., 2023) illustrated in Figure 1. We formulate it in two stages: (i) Knowledge verbalization to generate procedural knowledge from an LLM, and (ii) Knowledge distillation transfer LLM --geerrrtd knowledge to a smaller LMM
For the knowledge distillation stage, we introduce two constrained settings: Constrained planning and Counterfuctual replanning addition to the standard language planning task. These tasks enable
1 Authors contributed equally.
Our data and code is publicly available at: https: : hightu....... com/ ena / Hereafter, will use 'planning' refer 'language-based planning' for brevity.

---

Published as a conference paper at ICLR 2024
Procodural Knowledge Vorbalizaaion
Procedural Knowledge Distillation
Tlanning (P)
BOAL
RIAA
LLM
Provide steps
lpeal] sou movie
Lolkk drocinge and
Templates:
goal plan
Oonutrattedd Flamning
Distilled PLASMA
2,Choese movi
dioul jlaci conditiee
[goal] conditinalyy
4.Gtt terra from the concsssion stand
[candition] see more
Welk together toooede theater
St down the ass geidgnddd seats 7,5e a move..
Couskerfactual planting
Supervised
Rewrtte stepi:
COUNTERSACTUNA PLAN:
[geal] she movie
[plan] 1, Lo up movi showings and
Choose
time; Choose
2.Chou
up viewing dexice a comforaaile and safe place
Goals
Plans
Conditions
Countartsctual Plans
Muttrakknn
36 and enjoy the mevis.
CoPLAN Datasel
JP
DR
Figure 1: Symbolic Procedural Knowledge Distillation.
more realistic setting by requiring models to reason about contextually constrained situations in real-world applications. Specifically, the model generates or revises plan based on a given goal (e.g., "see a movie") while adhering to an additional condition (e.g.. "at home"). Our knowledge verbalization process results in a large dataset for (i) language-based planning, (ii) eaguuae--based planning under constraints, and (iii) language-based re-planning of existing plans under constraints. Our dataset, COPLAN, is then used to train smaller models, PLASMA, using both task-specific and multi-task distillation.
For the second phase of PLASMA, enable structured, tree-based reasoning via novel inference- time decoding algorithm (Figure 2). We observe that the standard next-token prediction objective in auto-regressive LMs (applied during distillation) does not equip them with sufficient causal and temporal reasoning abilities to generate high-quality plans, or a mechanism to rectify their mistakes develop verifier-guided step-wise beam search to
in earlier steps. To address this challenge, better leverage the multi-step structure of plans (resulting in PLASMA+). Concretely, we incorporate a step-wise verifier in tree-based decoding algorithm to guide PLASMA+ to generate more semantically coherent and temporally accurate plans.
Experimental results show that our approach is effective endowing smaller LMs with planning abilities. For the standard planning task, smaller student models (of varying sizes) achieve 17.57% relative improvements, on average, over their teacher. The best student model comparable even to GPT-3, a model 16 times the student's 's size. For the first time, we distill constrained and counterfactual planning abilities in small-size models, achieving 93% and 86% validity rates according to human evaluation. Interestingly, in the VirtualHome environment (Puig al., 2018), our model significantly outperforms previous work based on GPT-3 (Huang et al., 2022) on executability (absolute 17%) and correctness (absolute 25%). Our framework including symbolic procedural distillation, decoding-tine timme algorithm, and the proposed tasks and the accompanying COPLAN dataset provide valuable resource and direction for advancing research in the field of procedural language-
annuage--bseed based planning.
SMALL LANGUAGE MODELS AS PROCEDURAL KNOWLEDGE MODELS
In this section, we discuss how endow small student models with procedural knowledge for (con- strained and counterfactual) planning capabilities. We first describe our knowledge verbalization and distillation framework which we collectively refer to as Symbolic Procedural Knowledge Distillation 1, $2.2). We then propose a strategy to enhance the reasoning capabilities of small students via novel verifier-guided step-wise decoding algorithm ($2.3).
2.1 COPLAN: PROCEDURAL KNOWLEDGE VERBALIZATION FROM LARGE TEACHERS
Large language model can perform new tasks by adapting few in-context examples (Brown et al., 2020). We thus leverage this emergent reasoning capabilities of LLM to circumvent the challenge of crowdsourcing supervised datasets at scale. We collect data targeting the following three tasks:
2

---

Published as a conference paper at ICLR 2024
1. Goal-based Planning (pl.), decomposing a high-level goal g into a sequence of temporally
extended steps = {8:] ]
2. Constrained Planning (cp.), decomposing high-level goal g into sequence of temporally extended steps {8t) ){_ while satisfying given condition c.
3. Counterfactual Replanning (er.), rewriting initial plan y given goal g into new plan
y' in order to satisfy given condition c.
Our knowledge verbalization pipeline shown in the left side of Figure 1 a two-stage process: 1) instance generation through few-shot prompting, and 2) automatic data curation using critic to filter out the low quality data. The process results in COPLAN, a quality dataset containing goals, plans, conditions, and counterfactual plans.
Step 1. Data Generation We start by generating large pool of goals with a diverse range of topics in bootstrapping fashion. Concretely, we start with 5 manually written goals and expand them through prompting GPT-3, We then manually filter out low-quality (in terms of acceptability'cciivv ability) ones and repeat this expansion/filtering for several iterations until we obtain seed goal pool with 1 10 goals. We subsequently use this goal pool for randomly selecting few-shot examples for prompting and generating large number of goals in our final dataset.
For each generated goal g E 9, we few-shot prompt teacher model .M to generate a set of ordered steps, as plan achieve the goal. The input to M, including instruction and few-shot examples, takes the format shown in Appendix Figure 7. Since LLMs can be sensitive to instruction, and/or few-shot examples (Perez et al., 2021; Lu al., 2022b), we randomize the prompt by (i) manually creating a set of semantically similar instructions and each time randomly sample from the instruction set, and (ii) using different set of in-context examples for each input. We use a subset of the existing P ProScript (Sakaguchi et al., 2021) and DeScript (Wanzare et al., 2016) datasets as our seed source form in-context examples, = {(9). y5) 35)
y: M(yslg. P)
The result is a pool of 140k pairs of goal and plans, (g. v), generated from the teacher model.
For the constrained and counterfactual (re)planning tasks, we also obtain conditions and modified plans y/ from a teacher model M through few-shot prompting- We manually design our prompts P to collect natural language conditions concerning the environment the task is performed in such as Locat io ('the store is closed"), Equipment ("you don't 't have a sharp tool"), Safety ("the car breaks down") or user's specifications such as Physico cal Condition and P Preference ("you have an injury"). For a given goal gi and plan yi, we sample conditions:
C M(c.lgs Vr. P))
Next, few-shot prompt M to rewrite an initial plan y for given goal g such that it satisfies the requirement of a condition c:
v M(yils Vc, c, P)
The prompting templates and examples of conditions shown in Appendix Figure 8 and Table 6.
Step 2. Automatic Data Curation To retain high-quality data for (re)planning under the original and constrained settings, we filter out generated samples from Step i.c. generated plans, conditions and counterfactuals, that are invalid or of low quality. A plan y is considered invalid it contains an illogical order of steps, is off-topic (w.r. the goal) incomplete. Whereas modified plan y' should not only satisfies these general criteria but should also adhere to the condition.
To this end, we train separate supervised critic models to judge the quality of generated samples of different types. We collect 13K human annotations of valid vs. invalid samples on Amazon Mechanical Turk to train RoBERTa-Lagee (Liu al., 2019a) as our critic models (see Appendix B.1 and B.2 for more details on annotation instruction and hyper-paramett tuning). All critics are binary classifiers which identify whether tuple of either (goal, plan), (goal, plan, condition) or (goal, plan, condition, modified plan) is valid.
Naturally, there trade-off between dataset size and precision. Following West et al. (West et al., 2022), we test several confidence thresholds at which the critic rejects pair and choose the best values (0.65, 0.76, 0.82)² according to precision-recall curves. After filtering out low quality
'These values are for plan, condition and counterfactual plans, respectively.
3

---

Published as a conference paper at ICLR 2024
Buy new car
2 3D0
Doet freed stepl logicaly follows the to leve
Plan
Researcn rch vehicle and features
Logiclity
Comp
Achendo
[bext step]
Verifier
Contact seller
54
Research oehicle
F'll out registaaiion
41
Go deei ership
75
Make the purchase
28
24
20
Creck sales
69
Negotiate best
Get the cays
.56
Figure 2: Verifier-guided Step-wise Beam Search. For brevity, we only showcase with N = 5 and K for the first step and N 4 and K 2 for the second step. The scores are for illustration.
data, our final COPLAN dataset consists of main subsets including 57,794 (goal, plan) for the original goal-based planning task (DPL),, and 43,690 (goal, plan, condition, modified plan) for the constrained and counterfactual settings, (D and D ). On the original planning task, COPLAN x 11 larger in scale than existing datasets (Sakaguchi et al., 2021; Wanzare al., 2016) while keeping the precision at 74%. On the proposed constrained and counterfactual settings, our dataset is to the best of our knowledge the first large-scale constrained procedural (re)planning dataset with free-form (open vocabulary) conditions. Analyses show that the COPLAN includes a diverse array of topics covered by goals ($A.1) 1) and conditions ($A.2).
2.2 PLASMA: PROCEDURAL KNOWLEDGE DISTILLATION INTO SMALL STUDENTS
After obtaining our procedural planning data COPLAN, we use it to fine-tune student models on the three different task settings described in $2.1. We consider both task-specific and multi-task distillation objectives to transfer generated procedural knowledge into the student models:
Task-specific Distillation. Following the common practice, we use the standard autoregressive language modeling objective (Radford et al., 2018) to fine-tune separate student models for each task:
(()) ((r
log (o (x)) for tskk
(1)
where T(x) is task-specific template for each tpsk-s input (see right side of Figure 1).
Multi-task Distillation. We aim to improve the generalization of the student by exploiting the knowledge found in the three related tasks as inductive bias (Raffel et al., 2020; Wei et al., 2022). We thus minimize the joint loss including all three task settings. We name this student PLASMA-MUl
2.3 PLASMA+: ADVANCING STUDENT WITH VERIFIER-GUIE GUIIEE DECODING
During inference, the student may generate logically and/or temporally ill-formed sequence of steps = {st) as it is only trained to maximize the next-token probability. For example, in Figure 2, it may generate "write a check" step with relatively high confidence due a spurious correlation between "sales price" and "check". We mitigate this issue via step-wise guided decoding, Rather than generating plans greedily, we instead generate step-by-step by sampling several candidate next steps and searching for those with high log- peg--rrbbablity under both the distilled student and a verifier. The verifier tasked to check for sequential ordering and semantic completeness. an embodied setting, the verifier could be taken over by any affordance or safety module (Ahn et al., 2022) that determines the executability of an action given environment.
Step Verifier.
We introduce verifier, which is trained to check the validity of plan steps and encourage PLASMA to produce more temporally and causally valid plans. The verifier takes as input goal, the flan-so-far and candidate next step and outputs continuous validity score Pveritier (1 s E [0, 1].
We implement the verifier by fine-tuning RoBERTa model (Liu et al., 2019b) to classify candidate step as valid or invalid. For training, we reuse only 3K human written plans from existing datasets (Sakaguchi et al., 2021) 1) to form positive examples (valid next steps). However, since no negative
4

---

Published as a conference paper at ICLR 2024
examples are readily available, automatically create set of invalid steps pseudo-negative examples. Inspired by the common model errors, we design perturbations over ground- plans to target sequential ordering semantic completeness, topicality, and fluency See Appendix B.3 for details on perturbation strategies. Our verifier achieves an Fl score of 78% on a held out test set.
verifier-guided decoding in Figure 2. The procedure generates a plan y = (81,...,(T)) by sequentially sampling and pruning the next step candidate St. Concretely, at each iteration, it selects and expands size-K beam of plan-so-far, Y - {
Veriffer-guided Step-wise
Beam Search. We illustrate
< k=1 and generates N next-step candidates,
K
2e ((1
where || is concatenation, is task-specific input, and q is decoding algorithm. We encourage exploration at each step, by generating candidates using multiple decoding methods such as beam search, and nucleus sampling with temperature 1.0.
To select the top-K scoring next-step candidates S7, we use a value function v(sg() + I which returns the weighted sum of normalized sequence bog-liieldhood from the student model and the verifier validity score,
S arg K
er, (() ((gg)) a (og + (1-ay) a) log (s S<t)
with a controlling the impact of the distilled student and the verifier. The search ends when the beam contains K completed plans. We return the highest-scored plan as the final output. Our step-wise beam search strategy maintains diverse set of candidate plans during the decoding process, allowing the model to explore multiple plausible paths before converging on most promising one.
EXPERIMENTS
Implementation Details. While any model with few-shot capabilities could be used, we choose teacher model M to be GPT-3 text-curii-001 (Brown et al., 2020) for collecting the goals and initial plans, and GPT-3 text-davinci We sample data points from GPT-3 using nucleus sampling (p- = 0.98) and temperature of T 0.9. For our student models, we try range of model sizes in T5 family (Raffel et al., 2020), such as T5- T5-large, T5-3B, and T5-11B. Student models are trained using Huggingface Transformers (Wolf al., 2020). Main experiments can be done on GPUs with 48GB of memory.
for collecting conditions and counterfactual plans.
During inference, we use beam K = 5 for regular beam search, and N = 10 (next-step candidates), beam K 5, p 0.9, and a = 0.5 for our geiide step-wise decoding (see $2.3).
Baselines. For each task, we compare our distilled students with their corresponding teacher, zero-shot and few-shot variants of GPT-3 (Brown et al., 2020), COCOGEN (Madaan et al., 2022) and human performance (when available). CoCoGEN frames the planning task as a code generation task and use a pre-trained code LM (code-daving) -ete-ddvviii-0000000
few-shot setting.
Next, we present the experimental setup for each task, along with their results.
3.1 GOAL-BASED PLANNING
In this section, we aim to study two key research questions through experiments. Firstly, we seek to investigate the extent to which scale impacts the distillation of procedural knowledge. Secondly, we aim to examine whether the scale gap can be bridged through the use of multitasking and/or novel decoding algorithm. In essence, we seek to determine whether small language models can perform procedural planning tasks with the same level of proficiency as large language models.
Evaluation Set. For the original planning task, we use human- written plans from the test set of ProScript (Sakaguchi et al., 202 dataset as our evaluation data.
'In total, we automatically create 47K +/- pairs of (plan-so-far, next-step) using 3K human-writter plans. II our preliminary experiment, found text-davi vivinniii nci-003 (the strongest GPT-3 version the time) be helpful for the more challenging counterfactual data collection.
5

---

Published as a conference paper at ICLR 2024
Setup. We compare several student models of varying scales (770M-11B) with the teacher model, text-curie-000 and extremely large scale models (175B). For all student models, we decode using both regular beam search (PLASMA) and our verifier-guided step-wise beam search (PLASMA+).
Metrics. Since there may exist many equally valid plans to a goal, we conduct human eval- uations for the main results and report automatic metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2020) in Appendix Table 7. We ask human annotators on the Amazon Mechanical Turk (AMT) platform to rate the generated plans for 250 randomly sampled goals on three aspects: 1) Order: how well-ordered the plan is (captures se- quential correctness), 2) Coverage: how well the plan covers the necessary steps to accomplish the goal (captures semantic completeness), and 3) Overall quality: overall quality and cor- rectness of the plan. Details of the human evaluation can be found in Appendix D.3 Figure 10.
Table and Figure summarize the human evaluation for the original planning task.
Table I: Averaged S-point Likert scale human eval- uation for the goal-based planning. Small students paired with our decoding algorithm consistently out- perform their teacher (text - -curie-oro0 and are competitive with order of magnitude larger models in zero/few-shot settings. *CoCoGen (Madaan et al., 2022) is a 16-shot baseline using code LLM.
Does scale matter? Larger models perform relatively better across all aspects.
Does multi-task distillation help bridge the scale gap? As we observe, multi-task distilla- tion almost always wins over its task-specific counterpart with the exception of the small- est student, PLASMA (770M). We posit that very small student models might not have enough capacity to leverage the related tasks efficiently during multitasking.
Overall
Modelatae
Coverage
Order
Quality
PLASMA
3.18
3.64
3.17
PLASMA*
4.25
4.55
4.28
Distilled 770M
PLASMA
2.85
PLASMA-Mul+
4.16
4.48
4,23
Does verifier-guided decoding help bridge the scale gap? Pairing models with our verifier-guided step-wise decoding substan- tially improves performance across students of varying sizes over all aspects. Specifically, compared with regular beam search, our pro- posed decoding results in 7%488% relative im- provements in overall quality across different student sizes. The improvements achieved by the proposed decoding is larger for smaller students. We showcase the comparisons with qualitative examples in Table 8.
PLASMA
3,78
4.07
3.83
Distilled 3B
PLASMA-Mul
3.96
4.35
4.03
PLASMA-Mul+
4.29
4.62
4.33
PLASMA
4.01
4.33
4.03
PLASMA+
4.33
4.60
4.39
Distilled 11B
PLASMA-Mul
PLASMA-Mul+
4,53
4.77
4.58
Curie (Teacher)
few-shot (5)
3.75
4.27
3.75
zero-shot
4.84
Davinci (175B)
few-shot (5)
4.88
4.90
4.90
CoCoGEN (175B)
few shot (16)
4.48
4.70
4.55
The best distilled students with 770M, 3B, and 11B parameters achieved respectively 14.13% 16%, and 22.59% relative improve- ments over their teacher model (texx-tcuiieO00) Mul+) performs equally well as human and is competitive with orders-of-magiitude larger models (175B). These results support our claim that smaller model can, in fact, be as powerful as larger models when augmented with smarter decoding-tinn -ime techniques. Figure visualizes how we bridge the scale gap using our multi-task distillation and verifier- guided decoding. Since the initial submis- sion, we conduct an additional comparison with GPT-4 (see Table 14), indicating similar trends.
Human
Finally, our best distilled model (11B PLASMA-
Effect of symbolic distillation. In this experiment, we investigate the utility of CoPlan that is obtained through symbolic distillation in the presence of manually curated ProScrip pt dataset (Sakaguchi et al., 2021). We thus compare T5-1 IB distilled model trained on CoPlan with a IB model trained only on ProScript, and the mix both. Due to potential distribution shifts, we evaluated them on both their in- and out-of-domain test sets. We generate plans using our proposed verifier-ge guided decoding for randomly sampled 150 goals from P troScript and COPLAN. We use the same human evaluation setup before. Table 2 shows that training our LLM generated
Pairwise annotator agreements (i.e., how often do two annotators agree the answer) are 0.78, 0.84, and 0.80 for coverage, order and overall quality, respectively.

---

Published as a conference paper at ICLR 2024
PlaSma Teacher
PlaSma-Mull CoCoGen
PlaSma+
PlaSma-Mul++
Table 2: Effect of symbolic knowledge distilla- tion. The model trained on our CoPLAN dataset transfers better other dataset, ProScript.
Davinci
on
ProScript
Overall Quality
Overall
Scale
Train on
Coverage Order
Coverage
Quality
ProSeript
4.47
4.68
4.51
4.51
4.81
4.58
3.0
COPLAN
4.58
4.78
4.73
4.72
4.86
4.73
770M
3B
11B
1758
Mix
4.82
4.83
4.83
4.77
4.88
4.78
Molde Size
Figure 3: Bridging the scale gap in goal-based planning. Smaller models are able to achieve comparable performance and sometimes surpass larger models via multi-tasking and guided decoding.
COPLAN dataset, consistently transfers better to human-written dataset, ProScripp pt: across all dimensions. Training on the mix of both datasets, however, achieves the best performance.
3.2 CONSTRAINED AND COUNTERFACTUAL (RE)PLANNING
Here, we seek to benchmark language models' planning abilities under constrained (contextually grounded) situations. This task goes beyond the original planning task, requiring models to produce novel linguistic alternatives to unseen situations.
Evaluation Set. We created the test set of CoPLAN by generating conditions and counterfactual plans for the human-written (goal, plan) in the ProScripe iitrript..... Additionally, instead of using trained critic filter out low-quality samples, used human annotators verify them. We only used human-verified tuples of (goal, plan, condition, cf. plan) as the test set of COPLAN.
Setup. We compare 3B and 1IB student models with GPT-3 Curie and text davinci-003 the 175B teacher, in zero/few-shot settings. During inference, we use our verifier-guided step-wise decoding with 0.75 outweigh student model's probability over the verifier validity score."
Metric. We conduct human evaluation on the AMT. We generate (counterfactual) plans for 300 randomly sampled examples using each model. We ask workers to if each generated plan contains the necessary steps to make the goal achievable while satisfying the condition. We provide 3 answer options: A: The plan contains all the necessary steps to meet the requirements of the condition on the goal, B: The plan addresses the condition, but it is trivial and lacks thoughtfulness? and C: The plan does NOT address the condition or does so very poorly. We take the majority vote for the final results. Details crowd-sourcing human evaluation can be found in Appendix Figure 12.
Results. Figure depicts the results. Large students perform better on both tasks. In constrained planning, our 11B PLASMA-Mul+ demonstrates 93.33% success rate in producing high-quality plans while adhering to the given condition, which is comparable to the performance the 175B parameter Davinci model in zero-shot setting. Furthermore, plans, only 7 as opposed to 12 by havinci. While multi-tasking seems to be somewhat helpful in constrained planning, this is not always the case for replanning. We hypothesize that the reason for this could be that the original and constrained planning tasks, which do not involve modifying an existing plan, may negatively impact the replanning task. The best performance for the counterfactual replanning is achieved by Davinci (90%) followed by PLASMA+ (86.33%). Nonetheless, statistical T-test of our best models for constrained and counterfactual (re (re)nnnng tasks indicate that they are statistically par with the much larger Davinci G (175B). Human-annotated error types are reported in Appendix Table 11, showing "missing necessary steps" the most prevalent mistake."
model generates slightly fewer low-quality
"ee performed hyperparameter search over a = (0.5, 0.75., 0.8).
*Example: addressing the condition "you have money" with adding a step "find money" in the plan. "Piirwise annotator agreements are 0.96 and 0.94 for constrained and counterfactual (re (ra)pnann. Resultss with 95% confidence intervals reported the Appendix Table 13 and Figure 9.
7

---

Published as a conference paper at ICLR 2024
Con strained Planning
Counterfactual Replanning
Glodd
Inval
Bed
Good
Trxnal
100.%
13004
20
!12
19
40
36
141
122
40%
40%
20%
70%
264
264
274
280
287
266
Plotiex (00)
PasMa-Stal (38)
PlaSman (318)
Pledna Mide (118)
Carin z-bhot Curle Astiat
Dariect z sott (1 15th Davina fotud (1750
Pletiox (38)
Preslee Mul 198)
PlaSon (11B)
PleSan Made (718)
Curle psahe
Carte Fahoo
Deries pisked (175h)
Dartect dokud (1 (1758)
Figure 4: Human evaluation of 300 generations. PLASMA+ models (in left and right plots) are trained on the constrained and counterfactual (re)planning subsets of COPLAN. Statistical 7-test indicates that our best models for constrained and counterfactual (re plannnnng are statistically par with the much larger Davinei (175B) and are able to generate good plans 93.33% and 86.33% of the times.
We provide qualitative examples of model generations across all three tasks in Table 4. More examples of (good and bad) generations according to human annotators provided in Appendix Tables 9, 10.
3.3 APPLICATION TO EMBODIED AGENTS
As extrinsic evaluation, investigate the application of PLASMA a domain with hard executability conditions. We evaluate PLASMA on the task of planning in the VirtualHome (Puig al.. 2018) environment. In this envi- ronment, agents can perform household activ- ities, e.g. "paint ceiling". through programs, the form of supported actions (42 in total) and arguments. For evaluation, we use their test set consisting of 88 goals (and corresponding gold programs). compare our best student PLASMA-Mul (1IB) with the best sess-performing model on VirtualHome environment accord- ing to Huang al. (2022). Specifically, we compare with Planner, a --shot GPT-3 (175B)
Table 3: Human-evaluated correctness along with (automatic) executability and Longest-common sub- sequence (LCS) scores on VirtualHome (Puig et al., 2018). Steps generated by our model are more exe- cutable and correct for accomplishing the task.
Executability LCS Correctness
model
(%)
(%)
(%)
Planner (175B) (Tuang all, 2022)
77.17
19.10
18.33
ASAA- Mul PT (1IB)
28.36
41.38
PLASMA-Mul+ (118)
94.18
31.93
43.68
N/A
model with several inference-time strategies designed to ensure executability in embodied environ- ments. Following their setup, we translate generated steps from natural language to steps executable in the environment. To apply our model VirtualHome, we finetune PLASMA-MUl on 4K human labeled examples and also finetune the verifier on the same data using the method described in Section 2.3. This labeled data, obtained from previous work (Huang et al., 2022), consists of pairs of (task, natural language plan). The same data was used to finetune supervised models including GPT-3 13B, resulting in performance inferior to that of Planner (Huang et al., 2022).
We evaluate models on: (i) Executability: measures whether a generated action is executable within the environment, (ii) Correctness: measures whether the generated actions accomplish the task, and (iii) Longest common subsequence (LCS): measures the action-level similarity between generated program and a that are significantly more executable (according to automatic metric) and also more correct/complete (according to human judges). This suggests successful application of PLASMA in embodied setting. More experimental details can be found in Appendix F.
one. We show, in Table 3, that PLASMA generates steps
8

---

Published as a conference paper at ICLR 2024
Table PLASMA generations for Constrained planning and Counterfactual replanning tasks, More examples are provided in Appendix Table 15.
Goul
Condition
Initial Plan
Generated (Counlerfactual) Plan
Empty lint filter
Find lint filter > Remove lint filter Clean lint filter Replace lint filter
If lint trap is too hot to touch
Unplug dryer Open dryer Locate lini trap > Use resistant glove or mitt to pick up lint trap > Pull lint trap out of dryer > Empty trap
Lrad
dryer Locate trap Move
Load clothes into dryer Locate trap Use gloved hann to move hand toward lint trap Grip lint trap with fingers Use other hand to lift lint from lint trap > Remove lint trap from dryer
hand towards lint trap Grip lint trap with fin- gers > Pull lint trap out of dryer > Use other hand lift lint from lint trap Empty lint filter
RELATED WORKS
Procedural Planning The problem of planning to accomplish goal via sub-steps is widely studied in two contexts. One is script knowledge generation, which is a long-standing NL.P problem (Schank & Abelson, 1975a). Collecting script knowledge requires either human annotation (Wanzare al., 2016), unsupervised canture-baser extraction (Chambers & Jurafsky, 2008), or, more recently, methods that utilize spec-ipe fine-tuned LLMs (Sakaguchi al., 2021) and pipeline- baaeed approaches (Sancheti & Rudinger, 2022). In addition, there is line of procedural planning that involves planning with executable actions that can be executed by robots real-life environments (Huang et al., 2022; Ahn et al., 2022; Wu et al., 2022; Jansen, 2020; Guan et al., 2023). Recent approaches view planning as conditional text generation problem using L.LMs (Madaan et al., 2022; Huang et al., 2022; Ahn et al., 2022; Lu et al., 2023). Despite showing strong performance, their success heavily relies on scale.
Symbolic Knowledge Distillation Crowd-sourcing human- written datasets at scale is both chal- lenging and costly, leading to a growing interest in using LLM-generated data to train smaller models which falls under the conceptual framework of symbolic knowledge distillation (West et al., 2022). In concurrent work, Yuan et al. (Yuan et al., 2023) proposed similar approach to distill script knowledge from LLMs for constrained planning task. However unlike our conditions which allows nuanced and free-form format, their constraints are limited to specific types by extending an orig- inal goal with modifier. Relatedly, Collins et al. (2022) benchmarked LLMs' planning abilities (Valmeekam al., 2023) under 28 manually constructed constrained goals. We instead investigate a broader range of constraints in a larger-scale CoPLAN and distill this knowledge into smaller models.
Decoding-time Algorithm Decoding-time algorithm is an emerging approach for adapting language models' output for task-specific characteristics. Works in this line often focus on incorporating explicit lexical constraints (Lu et al., 2021: 2022a; Hokamp & Liu, 2017; Pascual et al., 2020). Besides discrete lexical constraints, applying continuous optimization functions, e.g. KI. loss, has been found to effective (Qin et al., 2020; 2022; Kumar et al., 2021; Hoang et al., 2017). Perhaps approach is most similar to function- -uidded decoding methods. Krause et al. (Krause et al., 2021) and Yang et al. (Yang & Klein, 2021) fuse next-token probability with desired attributes' probabilities at inference using discriminator model. These and related token-level beam search variants assume access to per-token logits and gradient updates. Our decoding method however only relies on model log- oogppbbbaiiiites and verifier to facilitate semantic and temporal constraints step level.
CONCLUSIONS AND FUTURE WORK
In this paper, we focus on procedural planning, a challenging task that involves decomposing high- level goals into ordered steps. We introduce PLASMA as an effective approach that uses smaller and more accessible models. By leveraging symbolic procedural knowledge distillation and an inference-time algorithm, we have endowed smaller models with enhanced procedural knowledge and planning capabilities. Furthermore, we introduced the task of Counterfactual Planning, which involves generating/revising plans to accommodate realistic counterfactual scenarios. Our results demonstrate that significantly smaller models can effectively compete with and often outperform their larger teacher models in both original and counterfactual settings. We hope our work sheds light on new directions towards developing smaller yet powerful multi-modas models for (counterfactual) procedural planning and reasoning.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was funded in part by the DARPA MCS program through NIWC Pacific (N66001-19-2- 4031), and the Allen Institute for AI. We thank the Beaker Team the Allen Institute for AI for helping with the compute infrastructure, OpenAl for providing access to the GPT-3 API, and the anonymous reviewers for the helpful discussions.
ETHICS STATEMENT
IRB AND ANNOTATION ETHICS
We obtained IRB exemption for our data collection and evaluation from our institution's internal review board. In full compliance to exemption clauses as published the code of federal regulations (45 CFR 46. (24())(,.3)), we did not collect any deanomyzing information, and we do not publish our dataset with worker specific information such as the MTurk's worker id. Based on our exempted status, according to our internal regulation,, does not require for to use consent forms with our crowdsourcing.
Additionally, data collection and evaluation efforts only involve human judgments about world knowledge relating to general real-world goals and plans. We have no reason to believe that our crowdsourcing posed harm discomfort beyond the minimal risk defined by 45 CFR 46. 102().
LIMITATIONS
One potential limitation of our work is that the verbalization component of our framework involves open text generation from large-scale language models (GPTs). Works such as Bender et al. (Bender et al., 2021) have argued that generations from LLMs can be prone to harmful biases stemming from the massive language data they are trained on. the process of constructing the dataset, we have not directly observed levels of biases to cause alarm. We believe harmful and discriminatory generations are largely mitigated by the very nature of the goals and scripts we obtain: our data primarily composed of low-level everyday situations such education, self-care, and mundane chores like vacuuming the floor or cooking meal (see $A.1, A.2). This said, we acknowledge that prejudices like gender roles, for example, do also surface in the most mundane scenarios.
related limitation is that LLMs have been trained on primarily English pretraining data, likely sourced from texts that reflect North American or European culture norms. Consequently, note that the goals in CoPLAN may reflect the goals that are most culturally expected or appropriate to the cultures of English--peaking countries. This is also expected of the plans that may include culturally limited processes and procedures. This should be consideration that any follow-up studies using our data and model should attend to. Extending our study to include more cecoocculurrlly inclusive goals and plans is a compelling direction for our future research.
BROADER IMPACTS
Related to the concerns discussed the Limitations section above, it is important for any downstream application be aware that our data may have limited representation of the goals and procedures of dominant cultures of English-speaking countries.
REPRODUCIBILITY STATEMENT
We include all experimental details for reproducing the distillation and decoding algorithm in the beginning of $3, Appendix B, and E. Additionally, instruction for collecting COPLAN and human evaluations are provided in $2.1 and Appendix D.3.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang- Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can and not as say: Grounding language robotic affordances. In arXiv preprint arXiv:2204.0169), 2022.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Mayj Major, and Shmargaret Shmitchell, On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, FAccT 221, pp. 610-623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 9781450383097. doi: 3445922.
/ttt:::://.i.oggg/// 00.14453342288.4445522 URL https: //oi.iii.o/ orge:i..rr/11 /00
Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Lianhui Qin, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, and Yejin Choi. 12d2: Inductive knowledge distillation with neurologic and self-imitation 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: httts::/procceeee proceed igs. neurips. cc/paper/ (eaaper20220/ het//ileella// 11t11/445766660c05c 967 674116 Bb Bbfbaacc 42 2e.f6644PPapeee .dff
Nathanael Chambers and Dan Jurafsky. Unsupervised learning of narrative event chains. In Proceed- ings of ACL-08: HLT, pp. 789-797, Columbus, Ohio, June 2008. Association for Computational Linguistics. URL https:::p / aeclanthoole ogy- org/ P08-1090.
Katherine M Collins, Catherine Wong, Jiahai Feng, Megan Wei, and Josh Tenenbaum. Structured, flexible, and robust: benchmarking and improving large language models towards more human- behavior in -ittt--diitriittion reasoning tasks. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, 2022.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, An- uoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adrians Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao. Vikas Raunak, Juan Diego Rodriguez, Sashank San- thanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. The GEM benchmark: Natural language generation, its evaluation and metrics. In Antoine Bosselut, Esin Durmus, Varun Prashant Gangal, Sebastian Gehrmann, Yacine Jernite, Laura Beltr-beltrachini, Samira Shaikh, and Wei Xu (eds.), Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metries (GEM 202 ), pp. 96-120, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v11 20/55//112021.p// 1e02111.em-m.111 URL ht https: / aa anthologyy oogy org/ 2021 gem- 10.
Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre- trained large language models to construct and utilize world models for model-based task planning, 2023.
11

---

Published as a conference paper at ICLR 2024
Geoffrey E. Hinton, Oriol Vinyals, and Jeftrey Dean. Distilling the knowledge in neural network. ArXiv, abs/1153..2231,,200
Cong Duy Vu Hoang, Gholamreza Haffari, and Trevor Cohn. Towards decoding as continuous optimisation in neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 146-156, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653/11/D17--0041 URL https: / //aclanthllogy httpe/logyyorgg///oo// wtt/D17-10111
Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Pupers), pp. 1535-1546, Vancouver, Canada, July 2017, Association for Computational Linguistics. doi: 10. 16555/ URL https: http::///caaaatoorr/.. org/p17-1141.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118-9147. PMLR, 2022. URL https: https://procee: http:///oceeeeeig/// mlr press / v1 62/huang22a html.
Peter Jansen.
planning without vision: Language models infer detailed plans from high-level instructions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4412-4417, Online, November 2020. Association for Computational Lin- guistics. doi: 011/86633/112220000 htwp:////so222...onnnnnmnoomm audings--emnl.399 URL htt.tttts//aclaattllllo/o/ wor.l/ogggooooooo/ooooooooo 2020. findings-nnniii emn 395.
Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan. Symbols as lingua franca for bridging human-ai chasm for explainable and advisable ai systems, 2021.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4929-4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 18.136/3/v 1/2021 1 findings-emmlp.424. URL https: /ttps////aaaaaataaaaaaaac ogy. www.g..org.. orgg/0211
findings- emplp.. 424.
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Tsvetkov, Controlled text generation as continuous optimization with multiple constraints. In Neural Information Processing Systems, 2021.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: / ac weeennnnhoooyyy org. 1004-1133
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.1 1692, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arviv: 1907. 1692, 2019b.
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. NeuroLogic decoding: (un)supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4288-4299 Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 .naacl- main.339. URL https: // el antho www.ooggy//ooooooooooo.ooo..oo.o org/ otg//021/11 naacl -main.an...mmmmmm 339.
12

---

Published as a conference paper at ICLR 2024
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. NeuroLogic a essque decoding: Constrained text generation with lookahead heuristies. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Seattle, United States, July 2022a. Association for Computational Linguistics. doi: 10.18653/v htt///11/2200 naccl maaac-- URL https: /t///aalaattttt /tt///aaaaantoooooooeaoee org/ 2022. nwaclaaac-am....a ma....m.....
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi: 11111663/1122222acllloo/ 556.
URL hwwpsttpps... http:///aaaaaaaaae
202g200222222222 acl-long
Yujie Lu, Weixi Feng, Wanrong Zhu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. Neuro-symbolic procedural planning with commonsense prompting. In The Fleventh International Conference Learning Representations, 2023. URL https: /tt:://ppee /toppppeevvveeeeeeee ew. net/ forumm?rrm d= 5c5700
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang. and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1384, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: htt////laaaahooo.......... org/2022 . emnlp-mmee moa-mmiin... 90.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023.
OpenAL. Openai api pricing. https: http:///pppaaaa.ccoocccciiigg/// 2023.
Accessed: 2023-05-15.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 00.3115/1073033.0731333 URL https: /a//accaannhhoo
Damian Pascual, Béni Egressy, Florian Bolli, and Roger Wattenhofer. Directed beam search: Plug- and-play lexically constrained language generation. ArXiv, abs/2002.15416 2020.
Douglas Pearson and John Laird. Incremental learning of procedural planning knowledge in challeng- ing environments. Computational Intelligence, 21:414-439, 11 2005. doi: 10.1111/j.146788400 2005.00280.x.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot lcarning with language models. NeurlPS, 2021. URL https: http://axvvvv..//e org/ 2ts./11............11...........
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8494-8502, 2018
Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D. Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. Back to the future: Unsupervised backprop- decoding for counterfactual and abductive commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 794-805, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v 1/t0//0eem//ommmaiii.. main.58. URL https: ac wacchannthoogyyy org/oo//////2 wwtp///..emmnmmmmmmmmaa.....aaaa emnlp-ma ww...aain... 58.
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 2022.
13

---

Published as a conference paper at ICLR 2024
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1 1-6, 2020. URL http:// /ttt:://mmmrt wwt./oorgppppppeee v/2 /0 074.html.
Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, and Yejin Choi. proSeript: Partially ordered scripts generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2138-2149, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 1/26211 findings-emnlp..ee URL atttss://act antho 1 ooyy org. 2021. fi fi inddnns-eemn 1 p. 1..884 84.
Abhilasha Sancheti and Rachel Rudinger. What do large language models learn about scripts? In Proceedings of the 11th Joint Conference on Lexical and Compulational Semantics, pp. 1-11, Seattle, Washington, July 2022. Association for Computational Linguisties. doi: 10.18653/v 2022.starsem- 1.1. URL ht ttspp: /ttpaaaaaannnnnnaaaaaaaaaaaaaaae org/ 2//g//222 2022. starsem-1.1.
Roger C. Schank and Robert P. Abelson. Scripts, plans and knowledge. In International Joint Conference on Artificial Intelligence, 1975a.
Roger C. Schank and Robert P. Abelson.
Scripts, plans and knowledge. PN Johnson- Laird and PC Wason (eds.), Thinking: Readings in Cognitive Science, Proceedingy of the Fourth International Joins Conference on Artificial Intelligence, pp. 151-157. Tbilisi, USSR, 1975b.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't 't plan (a benchmark for IIms on planning and reasoning about change), 2023.
Lilian D. A. Wanzare, Alessandra Zarcone, Stefan Thater, and Manfred Pinkal. A crowdsourced database of event sequence descriptions for the acquisition of high-quality script knowledge. In Pro- ceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' pp. 3494 3501, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA).
URL https::///// / / www. a acl web. org wwt.ooo///tttooooogoy//////////e anthooogy/h1htt5 1-556.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL / //pp/////p/e/eeee/e// net forum?
d.d=EEZ=Goozdq..
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4602-4625, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2022.naacl- mat.clm-mi..34.... URL httph:tt///o httttp:://acaaathhooogoooooo
org/2022 nwacnnaalaaaaaaaa mai-mmain... 341.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao. Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 10.188/3/v 11220000mmll--dmoms. URL
org/ .2002.eemmlp.... www.eeelleee....... petpp--dems.... 6.
Te-Lin Wu, Alex Spangher, Pegah Alipoormolabashi, Marjoric Freedman, Ralph Weischedel, and Nanyun Peng. Understanding multimodal procedural knowledge by sequencing multi- modal instructional manuals. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 4525-4542, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v hwtp//222222221o..332233 URL https: / //cc anthology org//022.22....... acll--ccl--- llng. 310.
14

---

Published as a conference paper at ICLR 2024
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguisties: Human Language Technologies. Association for Computational Linguisties, 2021. doi: 10. 10.1863//1/2021..00 watp:/2022..aacmmmaaa222222 mainn-a
URL ht.pss: /ooi .org/ CC. 1863.
Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Jankowski, Yanghua Xiao, and Deqing Yang. Distilling script knowledge from large language models for constrained language planning. In Proceedings of the 61 Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers), pp. 2300-4325, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https::tpp........tt...ppp / aaaaaaaaaa ////a/aaanntooooooooooooo loyy org/ 2023 a acl- -lngg 2.3
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:// openreview.noe net/ forum? d=
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large-scale task planning, 2023.
15

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIAL
A COPLAN ANALYSIS DETAILS
A.1 GOAL DIVERSITY
liness 3%
Credit
Dae resturro
Social 5%
Education 21%
Firencall
anoughh money
Caroer 9%
ahert sbirey
possess
Otherr 20%
anooe safery proteca
Service 14%
ab
Smstution
tome resmunce
ROCeSS
Roonl mese
Figure 5: Goal diversity COPLAN
Figure 6: Condition diversity in COPLAN
In this section, we demonstrate that the goals in our CoPLAN dataset broadly cover a diverse set of everyday, real-world human activities.
For this analysis, we first define seven goal-relevant categories based on categories defined by the US Bureau of Labor Statistics" (1) career and work related activities; (2) education and professional growth; (3) financial and commercial activities; (4) fitness and health; (5) service and civic activities; (6) social activities and relationships; and (7) self-improvement and leisure.
Next, using the seven categories, we manually annotate 200 most frequent verb unigrams, 300 most frequent noun unigrams, and 300 most frequent nominal (nouns + adjectives) bigrams extracted from the goals statement. Only when the unigram (e.g. "make"" or the bigram (e.g. "new word"") indicates one of the seven categories (e.g.. "close friend" for relationship or "college university" for education) the instance is annotated with the category. Otherwise, it is annotated with eight category, other. For each goal in COPLAN, each (verb, noun) unigram or (nominal) bigram casts a category as vote if found in the annotated data. If not found, then it casts other vote. Majority vote is taken the category of the larger goal statement.
Figure shows the distribution of the activity types in CoPLAN. Education is the largest category ("join an online course to learn new language") followed by self-improvement ("develop my creative writing skills"). Service ("cooking meals for homeless shelter"), career ("get interview for new job"), and financial ("upgrade to a new car") are the next largest categories. The other category includes miscellaneous activies like chores and events like "vaccuum the livingroom floor".
A.2 CONDITION DIVERSITY
We assess the diversity of the conditions in CoPLAN by analyzing the verbal use and nominal trigrams employed in the statements.
We manually analyze 20 most frequent verbs and phrasal verbs (e.g., "have access")) appearing in the condition statements. The verbs are grouped into semantic categories: (1) want (to want, to desire, etc); (2) possess (to have, to possess, etc); (3) access (to obtain, to get, to procure etc); (4)
"https: : www. bls. gov..oov//eo gov/news. rel ease att...u.s........... 1... htm defines 11 categories to cover common everyday civilian activities. We cluster these categories into five.
16

---

Published as a conference paper at ICLR 2024
able (to be able to, be capable of, etc); and (5) trust (to be safe, to rely, etc). Note that each of these categories include conditions of both polarity; for example, for pussess, includes both the condition imposed by having ("have enough money") and by lacking ('not have enough money"). A sixth category, other, was included for the verhs not included the above categories. For each condition in COPLAN,, the first trigram made up of verbs, adjectives, and nouns appearing after the main verb (e.g.. "If you want to lapply to an online program]" -> main verb: want, trigram: apply online program) were extracted. Trigrams were then associated with each of the 5 semantic categories based on the main verb.
Figure 6 shows the most frequent unique trigrams in each category. The graph includes the 20 most frequent trigrams for each category. The displayed trigrams were manually clustered when appropriate for readability purposes (e.g., "take course online" clustered with "take online course").
We find a wide variety of real-world constraints that pose varying levels of restriction such as preference and desire ("want take an online course") and hindrances posed by the state of having or not having something ("not having enough money" or "having a disability").
B ADDITIONAL EXPERIMENTAL DETAILS
B. CRITIC MODELS: COLLECTING HUMAN ANNOTATIONS
We gather human annotations of valid vs. invalid teacher generations. Annotations are crowdsourced through the Amazon Mechanical Turk (AMT) platform. We qualify 263 best performing workers through a paid qualification round. Additionally, we chose annotators among those who were located in US, GB and CA, and had 98% approval rate for at least 10,000 previous annotations. Crowdworker compensation for qualification and annotation HITs is maintained at average of $15 per hour.
Plans. For plans, the crowdworkers were presented with randomly-sampled 13K generated (goal, plan) pairs, and were asked to evaluate the plans along three dimensions: topicality topic the plan is relevant and appropriate for the goal, ordering -he steps in the plan are appropriately ordered, and completeness -the plan provides complete and informative steps to achieve the goal. We asked separate (fourth) dimension. Each dimension was
the workers to evaluate the goal's I achievability rated on a 5-point likert scale with three valid labels (Definitely, Mostly, and Somewhat; numeric value 1) and two invalid labels (Hardly, Not at all; numeric value 0). Each (goal, plan) pairs were annotated by three crowdworkers. The template used is shown in Figure 10.
We determine the validity of (goal, plan) pair in the following manner. We then calculate the mean score (over the three annotator responses) for each of the dimensions. A (goal, plan) pair is considered valid only if: (1) it receives score greater than 0.25 for each of the achievablility, topicality, or ordering dimensions, and (2) receives a scores greater or equal to 0.65 on the completeness dimension. Failing that, pair is considered invalid.
Conditions. For conditions, we collect human judgements on whether the condition makes the goal more specific or harder to achieve (but not impossible) on a candomly-sapple set of 6100 generated tuples of (goal, plan, condition). We include screenshot of our annotation template in Figure 11.
Counterfactual Plans. And finally, for counterfactual plans, we collect 10.5K human judgements on whether the modified plan contain all the necessary steps to make the goal achievable while adhering to the condition. We include screenshot of our annotation template in Figure 12.
batch size
learning rate
Plan Critic
16
le-6
Condition Critic
32
le-5
Counterfactual Critic
32
Ie-6
Table 5: Hyper-parameter values for training different critic models.
B.2 CRITIC MODELS: TRAINING DETAILS
We train 3 binary classifiers (critics) for filtering out low quality teacher generations in $2.1 using pre-trained RoBERTa-Lard Large Liu al. (2019a). We conduct small grid search on validation loss for batch size bs = {16, 32, 64] and learning rate = {le- - 4, 1e - 5, - 6, 5e - -6). We report the effective hyper-parameters for each critic in Table 5. We use early stopping validation loss.
17

---

Published as a conference paper at ICLR 2024
B.3 TRAINING THE VERIFIER
Constructing Pseudo-negative Examples. For training the step verifier, we use the human-writte plans Sakaguchi et al. (2021) construct positive examples of (plan-so-far, next-step) pairs and devise three main perturbation strategies to automatically construct negative examples as explained below:
Reordered Steps: in a plan. Thus, we apply both near and distant reordering by randomly reordering two consecutive and two distant steps.
Conflicting logical order results from inaccurate causal or temporal dependencies
Repetitive Steps: Degeneration i.e., generating repetitive text commonly observed in language models. Similarly, we include both near and distant repetition by repeating the immediate previous step and distant previous step as a pseudo-negative next-step.
Another common mistake made by language models is missing necessary steps, leading to incoherent plans. To simulate this behaviour, we randomly select non-immediate step as a pseudo-negative next-step.
Missing Steps:
We collect training set of 47k positive and negative pairs of (plan-so-far, next-step) using only 3k human-written plans.
Training Details. We fine-tune RoBERTa Large Liu al. (2019a) as a binary classifier identifying the validity of a candidate next-step. We train for 10 epochs with early stopping on validation accuracy using batch size of 32 and learning rate of le
Category
Goal
Condition
Purchase gardening supplics
there are no local gardening stores nearby
Location
Sing lyrics
you want to sing the lyrics in recording studio you want to use a laptop or computer
Studying for the exam
Equipment
Practice pottery techniques
you don't 't have the right tools clay
Take out several plates
the plates are heavy fragile the breaks down runs out of gas
Safety
Transport materials home
User's condition/
Practice playing the instrument
you unable read sheet music you need size specific equipment
specification
Rent rock climbing equipment
Table 6: Examples for different categories of conditions in CoPLAN dataset.
C OUT-OF-DOMAIN EVALUATION
Collins et al. (2022) proposed two out-of-distribution reasoning tasks to evaluate LLMs, one of which involved constrained planning. For given goal and one or more conditions, the task is to gener- ate a plan. We evaluate PLASMA the 28 constrained goals provided by the paper. We compare our generations to the GPT-generated plans provided by the paper and text-davinch -002 prompted in a zero shot manner. To evaluate the generations we perform human evalua- tion, as described in $D.3.
Model
% good
PLASMA
GPT-3 (from (Collins et al., 2022))
36
GPT-3 zero shot
64
Table 12: Percent of generated counter- factual plans which have been rated as good by annotators.
The human evaluation results in Table 12 show that PLASMA outperforms the other baselines in this out-of- domain subset of counterfactual planning task.
D EVALUATION DETAILS
D.1 AUTOMATIC EVALUATION
We report automatic evaluation of models for the original planning task in Table Note that human- written plans not the only possible plans, hence these automatic metrics may not provide
18

---

Published as a conference paper at ICLR 2024
modelgise
BLEU
ROUGE-2
ROUGE-L
BERT-I
PLASMA
12.97
14.02
PLASMA
14.26
16.31
31.02
85.30
Distilled 770M
PLASMA-Mul
14.47
14.43
27.99
84.02
PLASMex-Mul+
31.49
PLASMA
12.89
14.39
28.57
84.70
PLASMA
13.92
15.56
30.83
85.19
Distilled
PLASMA-Mul
13.62
15.42
29.31
84,80
PLASMA-Mull Mui+
14.96
16.80
31,97
85.28
PLASMA
12.64
13:93
28.14
84.56
PLASMA
14.65
15.84
31,04
85.33
Distilled 11B
PLASMA-Mul
13.61
15.67
29.99
85.10
Curie (Teacher)
few-shot (3-5)
7.13
9.24
22.78
83.08
zero-shot
4.98
7.81
21.38
81.20
Davinei (175B)
lew-shot (3-5)
10.27
10:27
24.57
83.48
CoCoGen (175B)
few-shot (16)
18.07
21.01
33.99
86.41
Table Automatic Evaluation Results for the Planning task. Note that the Human--riiteen plans are not the only possible plans, hence these automatic metrics may not provide an informative measure of performance.
informative measure performance. To further verify this, we computed the correlation between the most commonly used BLEU score and human scores. We find that BLEU has very weak correlations to human scores of coverage, ordering an overall quality, with a Pearson correlation of 7.7%, 5.9%, and 5.6%.
D.2 CONFIDENCE INTERVALS
We provide the 95% confidence intervals for our main results on goal-based (Table 13) and constrained and counterfactual (re)planning (Figure 9).
D.3 HUMAN EVALUATION ON AMT
All human evaluations were conducted on the Amazon Mechanical Turk (AMT). We sourced our annotators from the same pool qualified workers (see B. Throughout the entirety project, we estimation of time needed to complete
maintained average of $15/hour pay rate based the task. Each examples were annotated by 3 workers and majority vote was taken for the reported results. The layout templates for evaluating plans and counterfactual plans are shown in Figures 10 and 12, respectively.
E EXPERIMENTAL DETAILS OF VIRTUALHOME EVALUATION
We follow the same experimental setup and metrics for evaluation Planner Huang al. (2022). The test set consists of 88 high-level goals. To translate generated natural language step into an executable step, we follow Huang et al. (2022) and find an executable action closest in embedding space to the generated step. To compute these embeddings, model. Executability and LCS are computed identical to Huang et al. (2022). Due to challenges with reproducibility of GPT-3 outputs, evaluation results of GPT-3 do not exactly match between works.
use the rebbrrobeta-aae -argg
Note that the previous method (Huang al., 2022) involves extracting the most relevant (in-domain) human-annotated examples by matching goals (Make breakfast (-) Make toast) as well as sampling L.Mss several times for each step and ranking them.
19

---

Published as a conference paper at ICLR 2024
Model
Plan
Goal: put coffee in coffee machine
Curie (Teacher)
Step I: find coffee machine Step 2: unlock coffee machine Step 3: put coffee coffee
machine Step 4: close coffee machine Step 5: wait a few minutes Step 6: coffee should be ready
PLASMA-Mul (1IB)
put coffee in coffee machine Step 3: turn on coffee machine Step 4: wait for coffee to brew Step 5: drink coffee
Step I: get coffee Step 2:
PLASMA-Mu++111) Step I: buy coffee beans Step 2: grind the coffee beans Step 3: put the ground coffee
into the coffee filter Step 4: put the filter into the machine Step 5: turn the machine on Step 6: wait for the coffee to brew Step 7: enjoy your coffee
Goal: put on baseball jersey
Curie (Teacher)
Step 1: get basehall jersey Step 2: put baschall jersey on Step take baseball jersey off
PLASMA (1IB)
Step I: find a baseball jersey Step 2: put on baseball jersey Step 3: take off baseball jersey Step 4: put another baseball jersey Step 5: take off sccond baseball jersey Step 6: put on third baseball jersey Step 7: put on fourth bascball jersey
PLASMA+ (1IB)
Step I: walk to closet Step 2: open closet Step 3: look for baseball jersey Step 4: find
baseball jersey Step 5: take baseball jersey out of closet Step 6: put baseball jersey
Goal: sit at the table with dinner date
Curie (Teacher)
Step text or call the person to let them know that you will be Step 2: arrive at the restaurant on time Step 3: take seat at the table Step 4: wait for the other person to arrive Step 5: cat dinner Step 6: say goodbyc and thank you
PLASMA (3B)
Step 1: find dinner date Step go to dinner date's house Step 3: the table Step 4: talk to dinner date Step 5: eat dinner Step 6: leave dinner date's house
PLASMA+ (3B)
Step I: choose restaurant Step make reservation Step 3: drive the restaurant
Step 4: park the car Step 5: enter the restaurant Step 6: find seat Step 7: sit at the table with dinner date
Goal: brush all teeth
Curie (Teacher)
Step I: get in the Step 2: drive to the dental office Step 3: park the car Step 4: go inside the office Step 5: pay the bill Step get receipt Step brush all tecth Step 8: leave office
PLASMA (770M)
Get toothpaste Step 3: Get toothbrush Step 4: Get toothbrush Step 5: Get toothbrush Step 6: Get toothbrush Step 7: Brush all teeth
Step 1:
Get toothbrush Step 2:
PLASMA+ (770M)
Step 1: Get toothbrush Step 2: Get toothpaste Step 3: Put toothpaste toothbrush Step
4: Grab toothbrush Step 5: Hold toothbrush up mouth Step 6: Brush all teeth
Table 8: Qualitative examples generated by different models for the goal-based planning task.
F COMPARISON WITH GPT-4
It is noteworthy to mention that the planning subset of COPLAN is collected from the smaller GPT model (as teacher),
which is significantly less powerful that the most recent GPT-4 model. Nonetheless, conduct comparison of our best goal-based PLASMA model (11B PLASMA-Mul+) with its teacher, GPT-312 and GPT-4 in few-shot setting on 50 instances (total of 200). Results shown in Table 14. As we observe, the trend remains the same as in Table 1, with GPT-4 slightly surpassing its predecessor (text - -davinci- (iinniiii)))))))- only in the ordering dimension.
""eettddaii nci -003
20

---

Published as a conference paper at ICLR 2024
Type
Conditional Plan
Goal:
cat icc cream if you have an ice cream maker
Step 1:
Decide eat ice cream Step Locate ice cream maker
good
Step 3: Turn on the ice cream maker
Step 4: Add ingredients to the ice cream maker
Step 5: Follow the instructions churn the ice cream
Step 6: Remove the
from the ice maker ice cream bowl
Step
Step 8: Eat ice cream
Goal:
brush all teeth you out of toothpaste
Step I: Decided to brush all teeth
Step 2: Locate store that sells toothpaste
Step 3: Purchase toothpaste
good
Step 4:
store Step 5: Purchase toothpaste
Step 6: Drive back home
Step 7: Unpack the toothpaste
Step 8: Open the toothpaste
Step 9: Brush teeth
Goal:
write names the cards if you have shaky hand
Step find a pen or pencil with a large tip
Step 2: hold the pen or pencil in one hand pencil in the other hand pen or pencil to write the names the cards are Step 6: close the drawer or file
Step 3: hold the
Step 4:
Step once the
cards, place them in drawer or file
Step 7: write names on the cards
Goal:
walk into town if you have disability that limits your mobility
Step 1: Decide to walk into town
Step 2: Put shoes
Step 3: Get
other mobility aid if necessary
bad
Step
the cane or other mobility aid in backpack
Step 5: Put the backpack on
Step 6: Leave the house
Step 7: Walk to the bus stop
Step 8: Take the bus into
Step 9:
Walk into town
Table 9:
Qualitative examples of the constrained planning task.
21

---

Published as a conference paper at ICLR 2024
Type
Original Plan
Counterfactual Plan
Goal: empty lint filter if the lint trap is too hot to touch
Goal:
empty lint filter
Step I: load clothes into dryer
Step I: Load clothes into dryer
Step 2: locate lint trap
Step Locate
trap
Step 3: move hand towards lint trap
Step 3: Move hand towards lint trap
good
Step 4: grip lint trap with fingers
Step 4: Use heat-resistant glove to grip
Step pull lint trap out dryer
lint trap with fingers
Step 6: use other hand to lift lint
Step 5: Pull lint trap out of dryer
from lint trap
Step Use other hand to lint from lint trap
Step 7: empty lint filter
Step 7: Empty lint filter
Goal: take shower after work you want
Goal:
take shower after work
take shower the gym
Step I: decided take shower after work
Step 1: Decide to take a shower after work
Step 2: drive home from work
Step 2: Drive to gym
Step 3: go bath room
Step 3: Go to locker room
good
Step 4: take off clothes
Step 4: Remove clothes
Step start water to get heat
Step Enter shower room
Step 6: take hot shower
Step 6: Start water to get heat
Step 7: take a shower after work
Step 7: Take hot shower
Step 8: Take shower after work
Goal: empty old left overs from the fridge
if you have difficulty bending down
Goal: empty old left overs from the fridge
Step 1: go through the house
Step 1: go through the house
Step 2: walk into the kitchen
Step 2: walk into the kitchen
Step 3: locate the refridgerator
Step 3: locate the refridgerator
Step 4: walk to the refridgerator
bad
Step 4: walk to the refridgerator
Step 5: grab a chair or stool
Step 5: grab the door handle
Step 6: place the chair stool in front of the refrigerator
Step 6: open the reffrigeraorr door
Step empty old left overs from the fridge
Step use the chair stool to open the refrigerator door Step 8: empty old left overs from the fridge watch your payment information compromised
Goal: buy
Goal: buy a new watch
Step I: decided to buy new watch
Step 1: decide to buy a new watch
Step 2: search the internet for watch retailers
Step 2: search the internet for watch retailers
Step pick a reliable retailer
Step 3: pick a reliable retailer
Step 4: search the retailer site for watches
bad
Step 4: search the retailer site for watches
Step 5; add watch to cart
Step 5: add watch cart
Step 6: click check out
Step 6: click check out
Step 7: add payment information
Step 7: add payment information
Step 8: add address information
Step 8: verify payment information
Step 9: buy new watch
Step 9:
buy new watch
Table 10: Qualitative examples of the counterfactual re-planning task.
22

---

Published as a conference paper at ICLR 2024
Constrained Planning
Counterfactual Replanning
Edits
Missing
Unnecessary
Edits
Missing
Unnecessary
Error Type
Required
steps
steps
Required
steps
steps
Plasma+ (3B)
4.66
8.33
3.66
13.33
19.33
6.00
Plasma-Mul+ (3B)
4.33
3.66
10.66
14.66
4.33
Plasma+ (11B)
3.66
5.00
3.33
4.66
10.00
3.33
Plasma-Mul+ (11B)
3.00
3.33
3.66
6.00
11.66
4.66
curic-001 scro
7,00
27.00
6.66
26.00
49.33
13.66
curie-001 few-shot
6,00
25.33
5.00
30.00
48.00
13.33
davinei-00) zero-shot
1.33
6.33
0.66
5.33
7.33
2.66
davinci-000 few-shot
1.33
0.66
4.33
2.66
Table 11: Percent of generated (counterfactual) plans with each error type. most common error types across all models.
"Missing Steps" is the
Model
Overall Quality
95% CT
PLASMA
3.17
[294:3.41]]
PLASMA+
[4.12;4.44]
Distilled 770M
PLASMA-Mul
(2.0033099)
PLASMA-Mul+
4.23
|4.05; 4:42]
PLASMA
3.83
13,69;
PLASMA+
4.35
[4.20; 4.50]
Distilled 3B
PLASMA-Mul
4.03
[3.85;4.20]
PLASMA-Mul+
4.33
[4.17:4.48]
[3.85; 4.20)
PLASMA+
4.39
4.544
Distilled
PLASMA-Mr Mul
4,28
[4.12;4.42]
PLASMA-Mul+
4.58
[4.46;
Curie ('eaacerr)
few-shot (5)
3.75
3.954;
zero-shot
4.84
[4.77:4.92]
Davinci (175B)
few-shot
4.90
[4.85
CoCoGEn (175B)
shos (16)
[4.43; 4.68]
Human
4.57
[4.46; 4.69]
Table 13: Averaged 5-point human "quality ratings' for original planning along with 95% Confidence Intervals.
G COMPLEXITY AND DIVERSITY ANALYSIS OF STUDIED DATASETS
We analyze the complexity of proScript and CoPlan from several dimensions:
Lexical diversity: We use generally accepted measures (Gehrmann et al.. 2021) to analyze the diversity of datasets. We compute 1/2/3-gram entropy and the mean segmented token type ratio (MSTTR). To establish comparison, we compute these values for three other datasets: XSUM (extreme summarization of news articles), DialogSum (real-life scenario dialogue summarization), and StoySto These have been specifically picked as they are stylistically different from our goal and script setup, and they often contain longer more natural sentences. In Table 16, observe that even though the goals and steps in dataset are shorter, the overall lexical diversity of proScript and Coplan are comparable with other datasets. XSUM displays higher MSTTR score, but this is likely attributed to
23

---

Published as a conference paper at ICLR 2024
Overall
Coverage
Order
Quality
PLASMA-Mul+ (1IB)
4.31
4.68
4.23
Curie (teacher)
3.53
4.37
3.58
GPT-3.5 (Davinci-003)
4.78
4.84
4.81
GPT-4
4.78
5.00
4.81
Table 14: Comparison of our best goal-based PI.ASMA model with its teacher, GPT-3.5 and GPT-4 in fewshot setting.
the characteristics of news and more formal text. We also note that the machine- -eaerneeeeeerteed CoPlan exhibits slightly higher lexical diversity than the Human-writted proScript.
Perplexity of an LM: We also report the perplexity of off-the-shelf language model which measures the degree of uncertainty (surprise) of an LM when it generates the next tokens. The higher the perplexity. the more surprised the LM is. we see from the last column, with the exception of XSUM, the remaining datasets exhibit comparable perplexity scores. The higher perplexity in the case of XSUM is, again, attributed its association with the news domain, a distinct characteristic compared to the other datasets, which predominantly encompass everyday scenarios. Also note that, LMSs generally have lower perplexity scores muchine-generated data (as seen for CoPlan vs proScript).
Overlapping of the plans. We additionally analyze the amount of overlap between the steps in the train and test set. To this end, we identify the direct noun object of a given goal (e.g.. "Carry [a plate] to the kitchen") and remove it from the goal (i.e., "Carry to the kitchen") as well as all the steps in the corresponding plan (e.g.. "Pick up the plate" -> "Pick up". We then concatenate each goal with individual steps (i.e., "Carry to the kitchen. Pick up.") ") and measure the maximum longest subsequence match of goal+step in the test set over all goal+step in the train set.
In the ProScript dataset, we find that only 4.3% of steps in the test set have exact overlap with steps in the train set. If we relax the overlap to 90% and 80% (as opposed to an exact 100% overlap), this number increases to 5.5% and 10%, respectively. If the same is computed for plan overlap (as opposed to the step level; i.e., the direct object removed goal and plan-not not individual steps-are concatenated), we observe 0% overlap.
In the COPLAN dataset, commuting overlap w.r.t full train set is still in progress (due to its large scale). However, on a randomly sampled 10K instances from the train set, the numbers are 1.8%, 2%. 3.3% for exact, 90% and 80% overlap, respectively. And 0% for plan overlap.
This suggests that differences in the object taken by the verb do not necessarily mean minimal changes to the plan. Intuitively speaking, this is sensical: changes in the direct object, in the real world, should affect the way we resolve goal. For example, what extra steps we take (e.g., stack plates, but not mugs), the final goal location of the item (e.g., kitchen sink for plates, but the fridge for apples), or the manner of carrying (e.g.. glasses vs. boxes) will affect the steps we take a plan even the goal is constant (i.e. "Carry X to the kitchen"). This is even more pronounced in CoPlan it contains broad set of everyday human goals (see Appendix A) which can lead to vastly distinct plans even when the event (verb) itself is the same. While learning is learning. the differences between the path taken to "Learn to play a violin" vs. "Learn to play Monopoly" vs. "Leam to speak Spanish fluently" non-trivially different.
H DISCUSSION
On the importance of Distillation. Finetuning LLMs requires updating models' parameters which not only costly but often inaccessible for the broader community. Reducing the scale and cost of strong models via teacher distillation is the key developing open-sourced LMs that accessible to all, facilitating men-ttning and seamless adaptation to various domains and custom use cases. Given the large-scale training dataset used for PlaSma, we hope it can serve foundation model that
24

---

Published as a conference paper at ICLR 2024
Goal
Condition
Initial Plan
Generated (Counterfactual) Plan
Buy new videogame
Find store that sells vidcogames Go to the store Find the videogume you want to buy Buy the videogame > Take the videugame home > Play the videogame
If the game is too far away
videogame Research online pame stores > Compare prices and reviows > Seloct the game to buy > Purchase the game online
Decide
the game to
delivered > Buy a new videogame
Decided buy new videogame > Decide game > Get into car > Drive to game store Walk into game store Find desired game Buy new vidcogame
Decided buy new videogame Decide on game Search for the gume online Compare prices online > Pur- chase the game online Wait for the game to delivered > Unbox the game Play the game Enjoy the game
Eal some ice cream from ice shop
Find ice cream shop > Enter ice cream shop Find ice Fat cream
cream Choose ice
If you have allergies dairy products
Decide to eal some ice cream from an ice cream shop > Find an ice cream shop > Walk
cream shop Ask for a duiry free ice cream > Pick out ice cream > Pay for cream Fat some cream from
cream shop
Decided to eat some ice cream from an ice cream shop > Decide to eat ice cream Gather the family Drive to ice cram parlor> > Get fam- ily out and lock car > Walk in ice cream shop and Eat some ice from cream shop
Decide to eal some ice cream from an ice cream shop > Research cream shops your area that offer dairy- free options Gather the family > Drive to ice cream parlor > Get family out and lock car Walk in ice cream shop and Ask the stalf about their duiry free options Selecct dairy-free option > Eat some ice cream from an ice cream shup
get extra exercise
Set a goal for how many hours week you want to exercise > Research diffcrent types of exereise Choose one type of exercise start with > Make plan to follow the exercise plan > Start exercising > Keep exercising
If the fitness activity presents a risk of injury
Decide what type of exercise you would like do Re- search the activity to make sure you understand the risks and henefits Consult with medical professional sure the activity is safe for you > Find place to do the activity > Set a goal for often you would like do activity Start doing the activity > Monitor your progress and adjust the activity as needed > Get extra exereise
Decided get exercise Open the calen- dar > Identily open time slots during the week Schedule fitness activity during free time > Set reminder for activity on phone > Attend weekly fitness activity Get extra exercise
Decide to get extra exereise > Open the calendar Ideutify open time slots during the week Research and select a finness activity that the fitness activity during free time > Set reminder for activity on phone Attend weekly finees activity Get extra exercise
low risk injury Schedule
chop the
Get chopping black Get a chopping knife > Chop the wond Clean the chopping block > Put the chopping block away
If you have chainsaw
Pick up the wood > Place the wood the ground Set up the chainsaw > Shhrpen the chainsaw > Start the chainsaw Cut the wood Chop the wood
Pick up axe > Pick up a piece of wood > Check Pick up the chainsaw > Pick up a piece of wood Check the condition of wood Make sure wood dry enough > Set the wood upright > Swing the axe > Chap the wood
the condition of wood Make wood enough > Set the wood upright > Start the chainsaw > Guide the chainsaw along the waod Chop wood
uprightt
Table 15: Additional PLASMA generations for constrained planning and counterfactual replanning tasks.
can be quickly adapted to specific domains with minimal additional annotation (like we demonstrate by adapting Plasma
to VirtualHome). Moreover, we could not augment most of the LLMs with our decoding-time algorithm due to limited access to the model's log probabilities.
25

---

Published as a conference paper at ICLR 2024
Perplexity
Dataset
HI
H2
H3
MSTTR
(GPT-med)
XSUM
10.45
13.49
17.15
0.80
18.55
DialogSum
9.04
14.38
16.73
0.66
13.20
TinyStories
8.65
14.25
17.80
0.61
7.53
ProScript
8.86
13.67
15.61
0.59
12.93
COPLAN
9.27
15.00
18.05
0.64
10.05
Table 16: Lexical Diversity of proScript and COPLAN.
"Symbolic" AT vs. "Symbolic" Knowledge Distillation. We would like to draw attention to the evolving use of the term "symbolic" within the contemporary Al community, particularly in the context of natural language. It is important to note that the term "symbolic" has acquired multiple connotations, and its modern usage may differ from its original application in symbolic AI (Kambhampati al., 2021).
In our case, "Symbolic" (in symbolic knowledge distillation) refers to numan-readable textual formats (West et al., 2022) rather than the transfer of obscure/soft model weights as in standard distillation (Hinton et al., 2015).
I EXTENDED RELATED WORKS
Building Smaller Models. There a recent line of work on building general-purpose small models for reasoning tasks such Orca (Mukherjee al., 2023). While work shares similar spirit with Orca, we find the key distinction in (1) our goal is to develop a specialized small model for procedural/counterfccuulll planning and replanning with potential application an embodied domain, and (2) Orca is focused on learning from GPT-4 explanations (Chain of Thought) to improve models capabilities. Nonetheless, building specialized models on top of them can be explored in future works as we only worked on models that were accessible at the time of submission.
26

---

Published as a conference paper at ICLR 2024
Example Template:
Given a goal write down a list of steps to achieve the goal:
Goal: take nap the bed
Step
the bed for little
Step pull back the blanket
Step 3: pull back the sheet
Step 4: fluff up the pillow
Step lay down the bed
Step fall asleep on bed
Step take nap on the bed
Goal: hire dog walker Step
Prompt Prefix Generator:
def generate_
ticterat__propp__pp___ wwwpptoooppppeeee. profptt)reeix():
["For a given goal", "Given a goal"] wist ["write down", "break down into", "put down" "jot down"]
_ist
w3_list ["stops", "subgoals", "a list of steps", "several steps", "several subgoals", some steps", "some small steps"]
w4 1ist "to achieve the goal",
"for achieving the goal", "to attain the goal")
w1 random, sample (w1_list, []]
random. sample (w2_list, []]
w3 random, sample (w3_list, 1) [0]
w4 random. sample (w4_list, 1) (0)
prompt_prefft ["(wl), (w2) (w3) (w4). AnAn" return prompt_prefix
Figure 7: Randomize prompt template for eliciting plans.
27

---

Published as a conference paper at ICLR 2024
Prompt Template (Conditions)
Prompt Template (Counterfactual Plan)
You want to use social madia. How can you do this in 7 steps? stop decided to use social media; stop 2: Grab the phone:
You want to learn how to swim. How can you do this in Stepe?
step 3: Open. Start phone;
4: Go app step 5: Dewnload Facebook from stare; step5: Open and facebook; stepß: social media What the hindrancn that might the plan above?
Step 1: Decided to learn how to swim; Step 2: Find swimming
instructor; Step 3: Travel to pool; Step4; Meet swimming
teacher; Step S: Practice swimming during clusses; Step 6: Review mistakes with teacher until right; Step 7: Learn how to swim.
your phone cracked.
You want to Iearn how to swim. How can you do this in several steps il you forost your swimsull?
You
plant. How you do this in
steps?
Step 1: Decided to learn how to swim, Step 2: Find swimminng
step 1: decideed plant tomato plant: step nursery, step 3: Purchase tomato seedling, step Purchase potting soil and pat.; step Return home.; step Plant seedling in soil and pot. step plant tomato plant What is a specification that might affect plan above? you want to use compost for soil.
instructor; Step 3: Travel to pool; Step4; Meet swimming
teacher, Step 5. If you have forgotten your swimsuit, ask the
instructor if is possible to barrow one or if there is place
where you can purchase one; Step 6: Practice swimming
during classes; Step 7: Review mistakes with teachar until
right: Step Learn how to swim
... 3
You to print the report. How you do this 7 steps? step type the edited draft; step save edited draft: step 3: open the menu in the word processor; step file menu: step select printer settings: step send document to the printer, step print the report the hindrance might affect the plan above?
You want to pick pen. How can you do this in steps? 1: look for step find pen; step 3: walk to pen; stop 4: extend hand per: 5: reach for per: step 6 pick up pen
select
You want pick up pen. you this several steps you
pick from high shelf?
Figure 8: Prompt templates for acquiring Conditions and Counterfactual Plans.
Counterfactua//Cond tional Plenning
Counterfactual Plan Revision
200
$1.00 A0.33
R0.3
81.33
53es
78.00
73.32
80-
GBO
Good (1)
80-
Good (96)
44:33
Madnas (B0)
PasMM Mair (98)
Racimer (210/
Pradim shate (218)
Caree reeet Carle Sahot
Passai D-shol
Davin Fabitt
Pladmax (38)
Paliiks-Mllt (RB
Prstnna (110+
HaStna Mal (218)
Carie zahot
Conie halost
Davind zndtot
Disied fubost
Figure 9: Human evaluation of constrained and counterfactual (re)planning tasks. We report the proportion of plans labeled as "Good" by annotators along with 95% confidence intervals. Applying statistical tests (t-test) indicates a significant difference between all PLASMA variants and Curie (p 0.01) as well statistical significance was found between variants w/ or wo/ multitasking as well as between our best PLASMA and x 16 larger soxx--dvvnnii--00
significant difference between students of different sizes (p < 0.05). No
28

---

Published as a conference paper at ICLR 2024
Intrrctioos jollee to expandicolapon
may
Thanks partidpating
In imaghne you in business teaching people go about actieving everyday lifeling poals gool plan you teach.
Your
criteri, plan contain
steps. toplc
ordered, complete
Goa!
pool to ach eve. Goel can simple "cooking nnerr to more aborrte ting Howwll"
failty
aoe ling every country country.
timessonsuming
"becoming
Plan
The propesed plan givan geal, A las tycical subgoals staps achieve the main goal
(e for Misit fowwlll buy ticket to Haawall ded what want to book lodging, pack, leave for
the air port
Grade with following rubric, define
midcle
the plan an-tppcct.
Dulventy Topic the plan relevent and appropriate lote to the gool.
Soneetut: Topic in the plan wanders bir from goal but it okay averall
hor Topic the plan is overall lirelevant to the goal.
the plan well orderad?:
Durvensty
just
could recrcering
stylist chango.
bad nonsensical.
the plan complete and informativet:
Dudeeidyy The plan provides complete and informetive preture of what's needed to echieve the goal
Buseetut: The steps are semewhar genaral but everall you get you need. You might noed few more minor decalls.
The pian y bland dominated by unnecrsurr reeeevant and repetitive steps, van- key steps
misinn
plan good?:
overall good. plan should be wellarderet edd and combe repetitive
Soneetur:
The steds somewhic genaral, but overall get nead.
The an realy bland and not with repetitive steps.
NOTES:
Steps
chere. Think start asking relevant one and taking relevent achieve the goalr
key
students solid
Note that on overall good plan should be topically relevant, ordened correctly, almost complete and contains no repeet tive unieccsssryy staps.
hover over fine-gran Ins-grained
doubt, choose go with your gut
incomplete thought
Bammpies (dick expandicolappe))
$igoal)
Sisteps him0
Hardly
plan on-topic?
O
O
Is plan well ordered?
0
0
0
plan homplate/iafrrmatiie
0
0
0
0
0
0
(Optionat) care les us know If anyyhing was undear you experfenced any iscue,, you have any fedback
Subriit
Figure 10: AMT human evaluation template for the original planning task. For validation round we substituted goal achievability (is the goal achievable with appropriate steps?) for overall question (is the plan overall good?).
29

---

Published as a conference paper at ICLR 2024
iapuctions (sick expanticssappe)
Therks for partopeting
this HIT shown pair goel and plan, fallowed ses conditions, Each candition mig make achioving the gool more complex or harder whichhh would require change of the plan. You will evaluatt the quality of each the consition for the given pair of goal and plan Addiliinlllyy Bl. you will evaluate the condition has an association to given tag.
gool, a
comesponcing toge: gont-denine that
Gow
ac ietivaable through plan
Harmei spend Sundlyys
beach, and so A sttep day- stop pr proposed actions to actiave the poal wiich cona ala of lai typion sutggoets or stece 1o achiere the main gool (n.g. for "viit Howall buy ticket to Hewall dedce whar you want cee. lodgina, pack,
puin
airport)
require changes goals,
fight
A
desemben the candian
add conatrennt
glvn plan In.g. q. the above conditions, novinngng cor ia associaned with equipment eis.)
Itoef' quality following
The condition should relevant to and realistic the given goal.
The concitio should requiee channg in the plan thet doea not makis ech eving the goal imposarble. in other worth, thene should be aitomaaive ways to achleve the godl. p. should not negate an existing steps plan for the goal) way that makes Et impossble gost.
goal given
For each sut of (goal, plan, condisoni, you will reed to arrwwrr wholter the conditon albers the plan achieiing the poal, or make inpossoble J.
achieve has some other issues.
Eeen plas (ckikk sopandiaalllppae
s(poa)
Aiplar_pertty
Conditlen: 1/sondtoon_ 1
impons
Conaliso oevatio gool inpowa mave
O Other sonditen ondor goo decssn make senso, condition irrewant to the gool, conditin sircl nepeting ek.)
Doea condtt on manociate with the S/teg_.
Nea
Conditien: //oondtton_
Covdltto rabe coedit actiicu
O Ganditze tint vil mew god impoos de sove
Opeer undor
conditonn 1D sandionnss nepaiing
Dowe die con un sescciate with tie leg 5/lvg.22
Yea
(Cptional) Please anything unclest: you exgerfences you any
Submit
Figure 11: AMT template for assessing validity of conditions for critic model training.
30

---

Published as a conference paper at ICLR 2024
Instraaiion (ditk to soce laappa
Thanks for partic partiiputing jouting
Please read carefully.
Trivaa modificaioon s istle trick please be sure look the examples carefully.
you working throe rughh these hin, you may are repeats in graiplen conns conditoon are different
you weshhww goal comesponding plan action, goo achieabble. Following definiiino
to gow
provided with moddfie plan.
meenn overtome findrance
Goal
goslddeeiie that be achirattee through (ea- Hawali, ail spend my Sundays the beach, and on)
Plan
step- byaatep up proposed actions schieve the gool which conseuts of typical subgoa's (nu. we
goot
you
book lodging.
Conditioo
handran fblocker or particular preference thet requires moditicaionns updates pla".
(Fig- above goals, having cart, "vait during Alohs Week', and and
Modiffed Plan
plan
that condition
modiited
modification of the original plan.
(e.. for the conditin hrving car" when the goal sioo 'Vist Howell: vallo moditicatio
would regu aoditonal stap "lock for shames caber, etc)
YOUR TASK is evaluatz the quality of the modifeed p anowering two questions.
antinealble
Yas The modified plan consains all the necessary stepe meet the requirements of the condition on goal
thoughtfulness
The modifiee plan does NOT address the consitiion does poorly. One more important stops are
What
with modified plan?
stops wrong care requirest?
there mizsing (edettional nont steps requiresy
Are there untra unreecssaee varry stace dellttes requiried?
enverything
Ekample of had modification:
Hindered
Q1: Dave the modified plau cancale eecessary steps with
Buy
hawe ansugh moriy
Vour luly
Sawo roed Bnd plas
Research
Research vahick festures
(peal on dos the roncon repertioss ees wheree
Teahdrioe ofyour cholen!
Eo deslentipp
Chock sale prize vamern vomantiic
Check salk price and warcenees
that opely!
Nogil aieo peire
some mmppo wrang jodtes neceerary.
some stepi welsiigg
senecemary
Your goal is to... sigeal)
But you hindered tyy.. sécondidion)
Your thal plan way... Spotpe, Himl)
Now, your modif ed plan it...
Simod steps,himi himl)
modified
necessary sceps
condition?
Teu allof the important steps present.
one more importent steps missing
Q2: What are the problems with the mudified plan? (volect all that apply! No problems: All imporian steps p present No missing or extra steps. necessary).
suepts) isfort wrong
Important stepti istere missing
There noere extira unnerensary stepp))
Figure 12: AMT human evaluation template for counterfactual re re-planning. We similar layout for counterfactual planning task only removing the initial plan.
31


Reviews:
[{"review_id": "oVMAwaIfbS", "reviewer": "Reviewer_drVQ", "summary": "This paper proposed a distillation procedure and an inference-time decoding algorithm to enable relative small language models for planning and replanning with performance close or surpassing its larger language teacher models.", "questions": "1. There are good ideas within the paper. The writing could be improved to make these good idea clear and stand-out. For example, how to train the step-verifier from human-written plans along with more formal analysis of impact of the step-verifier. \n\t2. Please define the loss functions formally with teacher-student distillation and verifier-training.\n\t3. For the step verifier, \"we design perturbations … ordering, semantic completeness, topicality and fluency\", please provide detailed analysis of these data-side steps regarding their intuition and formal properties if possible. How does a single verifier score reflect all these criteria? Any special design to achieve them with a simple RoBERTa based classifier?\n\t4. Regarding the evaluation metrics, please provide more details of the AMT human steps. Are coverage, order, over quality complete to evaluate a plan? Any comparison or correlation on the human evaluation metrics and the bleu numbers and the Emobided Environment's metrics? If not well-correlated, any proposal on automatic evaluating plans? Also how to relate and align human evaluation, bleu-style sequence matching metrics, embodied environment testable metrics and real-world execution measurable metrics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "• Proposed a paradigm to distill procedural planning knowledge from large language models to enable smaller languages to do planning, and it seems to be working. \n\t• Within the paradigm, a cost-effective human-in-loop LLM generated data curation procedure is also proposed to create the COPLAN dataset.\n\t• Proposed a  guided decoding procedure with a LLM-based (RoBERTa) step verifier to  guide the beam-search during planning steps decoding generation. The guidance help to further regulate the validity of the steps.", "weaknesses": "• The LLM-to-Planning-Model teacher-student paradigm for planning is not well motivated. Cost, performance (from specialization), controllable procedure, better-integration with downstream tasks (e.g. decoding/execution) and so on? It is more about better understanding of the key capabilities of existing techniques and combining them to solve the critical problems. For example, if it is more about specializing common knowledge embedded in LLMs to do planning, then smaller LLM might not be the right solution --- the same proposed paradigm can be combined with LLM of the same size or even larger LLMs for superior planning capabilities. What are the real problems and the corresponding means could be better sorted out? \n\t• The truth contribution and their relevancy might be hidden in the paper title and the current way of writing.  The proposal is composed of three parts (1) planning data generation from LLMs with human-in-loop curation, (2) teacher-student distillation training, (3)  language model decoding with step verifier.  There are less texts regarding teacher-student distillation. This might indicate that the teacher-distillation importance is over-estimated. With the planning data generation and verifier-guided decoding generation, there might other ways to enhance planning abilities, e.g. finetuning the original LLMs to specializing into planning domains. If the distillation step is an importance component in the ingredients, please detail it and discuss more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RZI9Gwzklz", "reviewer": "Reviewer_h13v", "summary": "The paper introduces a framework designed to improve the procedural knowledge and planning capabilities of small language models. This is achieved through symbolic procedural knowledge distillation and a verifier-guided step-wise beam search algorithm. The authors have conducted experiments to compare student models of varying sizes with their teacher model, and have utilized human evaluations to assess the generated plans in terms of sequence, completeness, and overall quality. The findings indicate that smaller models can reach or even surpass the performance of larger models by employing the PLASMA framework.", "questions": "- If smaller language models can be effectively paired with human input or external verifiers for improved planning, why is distillation from a larger model necessary? This question is particularly relevant given that the domains discussed in the paper appear to be amenable to human verification.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and well-structured.\n- Equipping small language models to come up with procedural knowledge at the same level as large language models is an important direction from an engineering perspective given the accessibility, carbon footprint, and cost of large language models.", "weaknesses": "- Although human evaluations were conducted, the executability conditions for the plans in the domains used in these experiments seem to be loose. It would be beneficial to evaluate the models in domains which have hard executability conditions (like the domains used in International Planning Competitions), where the correctness can be objectively determined, to more accurately gauge the language planning abilities of the proposed method.\n- A comparison with GPT-4, in addition to GPT-3, could provide additional insights into the effectiveness of the method.\n- The potential for increased bias due to the distillation from larger language models is mentioned in the limitations section but remains a concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GDfmZLE9Y2", "reviewer": "Reviewer_EJrs", "summary": "The paper considers the use LLMs for generating NL instructions for a given task, called procedural plans. The paper proposes that smaller LLMs trained specifically for generation of such plans can outperform the models used as teacher, and be on par of larger models.\n\nAlgorithmically, the contributions are three:\n- PlaSma: small model specialized in generating NL instructions.\n- PlaSma+: uses PlaSma and an additional model for biasing the output towards higher validity. The paper refers to this bias as a “constraint”.\n- CoPlan dataset\n\nThe key transversal issues are both data generation and evaluation.\n\nThe dataset (CoPlan) is generated using a combination of seed prompts, large models, and human validation.\n\nThe experimental setup is reasonable these days: use proprietary GPT as teacher and for generating data; use T5 variants as small models; BERT-variations for classification tasks.\n\nThe evaluation is more complicated. The paper reports good human evaluation results in one dataset as the plans cannot be tested. (The  appendix reports usual BLEU and ROUGE scores, perhaps for pacifying some reviewers, but for natural situations there are so many possible wordings that that might very misleading). For VirtualHome, they report an interesting success rate.\n\nThe key question is whether the smaller model is just mimicking the teacher’s behaviour. However, the paper reports that the student might outperform the teacher significantly, especially if it has enough capacity. The bias-towards verification model has a higher impact in models with lower capacity.", "questions": "- Except for PlaSma-Mul, what does PlaSma mean when measured in different tasks. Can you elaborate on how that manifests in the experiments?\n\t- For instance, in Table 1, the PlaSma model is trained in the planning task, so there are precisely 6 models there. Right?\n\t- Those models are completely different from the ones reported in Fig 4, correct?\n- Please describe the ProScript in-depth and discuss why this is a good dataset for studying this problem.\n- Please discuss what other datasets could have been used, and explain why some possibilities are inconvenient. \n\t- This should be added to the related work section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Interesting problem as instructions are a key possible application of LLM. Sensible to scale and cost.\n- Good description of the methodology in all aspects.\n\t- In particular the data generation vs curation.\n- Sensible complexity of the tasks: goal, conditions, verification.\n- A secondary model specialized in higher correctness is a good idea while focusing at lower capacity.", "weaknesses": "- The dataset ProScript is not well explained\n\t- It is hard to qualify the complexity of the instructions.\n\t- So, the results in Table 2 are hard to understand because we don’t know about the relative complexity of the task and the diversity of tasks.\n- I suggest reducing the tone of the phrase “we introduce the task of counterfactual planning”. A quick search in google scholar for “plan revision” reported, for instance, these papers: \n\t- Ow, P. S., Smith, S. F., & Thiriez, A. Reactive plan revision. AAAI 1998.\n\t- Williams, K., & Burdick, J. Multi-robot boundary coverage with plan revision. In Proceedings 2006 IEEE International Conference on Robotics and Automation.\n- Abuse of some terms\n\t- The “step verifier” is **not** verifying, but adding a bias. For instance, the paper mentions that in the case of embodied agents, that verification can be taken over by a safety module. In that scenario, with reasonable \\alpha that follows the LLM when the so-called verification is not saying anything relevant, the aggregation of Eq (3) cannot prevent cathastrophic errors that are considered very attractive by the LLM. Perhaps a better name would be “quality bias” or anything saying bias.\n\t- Same applies to the notion of “symbolic knowledge destilation”, but we are probably too late for this one. I find it problematic that in AI we associate NL with symbolic, as the word is overloaded with a huge body of work in AI ranging from logic to graphical models. It should be more clear to call it something like “instruction distillation”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XOoiXMT8gR", "reviewer": "Reviewer_bpqq", "summary": "This paper proposes to leverage knowledge distillation to train smaller LMs to replicate the procedural plan generation abilities of LLMs.This serves as a cost-effective alternative to achieve the same performance as LLMs using smaller LMs. The authors generate a novel dataset, called CoPlan which includes goal-based and constrained planning tasks, as well as counterfactual replanning tasks. When trained on the CoPlan dataset, the smaller (distilled) models showcased comparable performance to their LLM counterparts. Empirical and ablation experiments further demonstrate the same.", "questions": "1. How is CoPlan different from ProScript? Barring the size factor, is it the counterfactual and replanning subset that is novel? Or is there a difference in the diversity of data too?\n\n2. Can you explain the use of the term \"symbolic\" in \"symbolic procedural knowledge distillation\"? What is \"symbolic\" here?\n\n3. It would be interesting to see when distillation leads to overfitting. What factors (model size of small LM, amount of training data) does it depend on? This would help motivate the generation of a larger dataset (compared to existing ones like ProScript).\n\n4. Minor Comment: The use of the term \"task\" in Sec 2.2 is ambiguous. The authors should clarify upfront that the three tasks are the three different settings that they investigate.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the paper is well-written and has a smooth flow, which makes it easy to follow.\n\n2. The data collection process used to generate CoPlan is novel and would be useful for researchers to collect high-quality data with minimal human involvement.\n\n3. Some of the results shown in the paper were interesting, although not entirely surprising.", "weaknesses": "1. Novelty: The main contribution of this work -- to train a small LM to imitate an LLM by using the LLM as a teacher to train the small LM, is not novel. It has already been demonstrated in [A] (and has not been cited here) for a variety of tasks including complex reasoning If considered in the context of [A], the novelty here is limited to its extension to planning. The use of beam search-based planning is also very similar to [B], however, given its recency, I have discounted it in my evaluation. \n\n2. Missing Key Experiments: While the authors motivate the use of knowledge distillation and compare their distilled models with that of the teacher model, the comparisons with the original (undistilled) model seem to be missing. Without this, it is hard to gauge the performance enhancement from distillation.\n\n[A] Orca: Progressive Learning from Complex Explanation Traces of GPT-4, Mukherjee et al., 2023\n\n[B] SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledge, Hazra et al., 2023\n\n--------------------------------------------------------------\n\nEdit 1:\n\nI'm discarding the Novelty contention given that [A] is considered \"contemporaneous\" according to ICLR guidelines. The authors have also answered the aspect of the missing experiment. Hence, I'm raising my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)