Distilling Script Knowledge from Large Language Models for Constrained Language Planning
Year: 2023

Abstract:
In everyday life, humans often plan their actions by following step-by-step instructions in the form of goal-oriented scripts. Previous work has exploited language models (LMs) to plan for abstract goals of stereotypical activities (e.g., “make a cake”), but leaves more specific goals with multi-facet constraints understudied (e.g., “make a cake for diabetics”). In this paper, we define the task of constrained language planning for the first time. We propose an overgenerate-then-filter approach to improve large language models (LLMs) on this task, and use it to distill a novel constrained language planning dataset, CoScript, which consists of 55,000 scripts. Empirical results demonstrate that our method significantly improves the constrained language planning ability of LLMs, especially on constraint faithfulness. Furthermore, CoScript is demonstrated to be quite effective in endowing smaller LMs with constrained language planning ability. 1

heading: 1 Introduction
text: To accomplish everyday goals, humans usually plan their actions in accordance with step-by-step instructions. Such instructions are discovered as goal-oriented scripts (Schank and Abelson, 1975, 2013), involving a set of prototypical event sequences to achieve goals. For the example in Figure 1, to achieve the goal (make a cake), one usually has to follow certain steps of instructions, e.g., gather ingredients, preheat the oven, etc. The planning for such step-by-step scripts chains up reasoning toward complex goals (Abelson, 1976; Wei et al., 2022). Therefore, the automation of planning envisions more intelligent and reasonable AI
∗Corresponding authors. †Work done while at Brain Technologies, Inc.
1Resources of this paper can be found at https:// github.com/siyuyuan/coscript.
systems in various domains, such as executable robotic systems (Kovalchuk et al., 2021; Huang et al., 2022) and reasoning systems for problemsolving (Wei et al., 2022; Wang et al., 2022).
Recent studies have identified that language models (LMs) can be used to plan scripts (Sancheti and Rudinger, 2022). Previous work (Huang et al., 2022) has shown that large language models (LLMs), such as GPT-3 (Brown et al., 2020) InstructGPT (Ouyang et al., 2022) and PaLM (Chowdhery et al., 2022), can effectively decompose goals into procedural steps in a zero/few-shot manner. To train specialized models, researchers have proposed datasets for the automatic understanding and generation of script knowledge (Schank and Abelson, 1975; Regneri et al., 2010; Wanzare et al., 2016; Lyu et al., 2021; Sak-
4303
aguchi et al., 2021). However, previous work mainly focuses on planning for the abstract goals of stereotypical activities (Abelson, 1976). Planning for goals with specific constraints (e.g., for diabetics) still remains under-studied.
In this paper, we define the problem of constrained language planning, which imposes different constraints on the goals of planning. An abstract goal, for example, make a cake, can be inherited by different real-life specific goals with multi-faceted constraints. A cake can be made for 1) different ingredients (e.g., chocolate or vanilla); 2) various tools (e.g., with a microwave or an oven); or 3) different purposes (e.g., for a wedding or a birthday party). A good planner should write scripts that are reasonable and faithful to constraints. However, LLMs sometimes do not plan faithfully toward the constraints. As showcased in Figure 1, InstructGPT suggests adding sugar to the cake for diabetic patients. Also, due to a shortage of datasets for constrained language planning, the ability of smaller but specialized models to plan with specific constraints has been underexplored.
In this paper, we aim to evaluate and improve the constrained language planning ability of LLMs, while distilling a dataset from LLMs to train specialized models. Our empirical study finds that LLMs tend to plan fluently but unfaithfully to the constraints. Thus, we employ an over-generatethen-filter approach (Wiegreffe et al., 2022) to satisfy the quality of the generated scripts to constraints. The main idea is to select high-quality ones from multiple generated scripts. Then, we use LLMs (e.g., InstructGPT) with this approach to generate a dataset for constrained language planning, which inherits the idea of symbolic knowledge distillation from models (West et al., 2022). We thus arrive at a Constrained Script dataset, i.e., CoScript, which consists of 55,000 high-quality scripts with specific goals and steps. Experiments show that, when trained on CoScript, smaller models such as T5 (Raffel et al., 2020) can achieve good performance, even surpassing that of LLMs.
Our contributions are summarized as follows: 1) To our knowledge, we are the first to establish the constrained language planning problem, which advances language planning toward more specific goals. 2) We evaluate the few-shot constrained language planning ability of LLMs and develop an over-generate-then-filter method for LLMs, resulting in a 26% increase in accuracy. 3) Based
on our method, we use LLMs to generate a highquality script dataset (CoScript) for constrained language planning. By leveraging the CoScript, we endow specialized and smaller models with constrained language planning ability, which achieves comparable performance to that of LLMs.

heading: 2 Related Work
text: Language Planning Language planning aims to decompose a goal into sequences of steps (Kaplan and Baldauf, 1997), which is widely used in robotics (Kaiser et al., 2014; Paxton et al., 2019; Berg et al., 2022) and procedural text generation (Goldfarb-Tarrant et al., 2020; Hu et al., 2022). Early studies approach language planning with syntactic parsing for the context (Koller and Stone, 2007; Garoufi and Koller, 2010). Recently, researchers have investigated the planning capability of language models in various domains (Olmo et al., 2021; Valmeekam et al., 2022). However, they mainly focus on generating scripts for stereotypical activities toward abstract goals. For example, Huang et al. (2022) proposes to plan for the general-typed tasks for embodied agents, while Yang et al. (2021) edits actions for abstract goals to video retrieval. In contrast, we explore planning for specific goals (e.g., “make a cake for diabetics”). Collins et al. (2022) has benchmarked LLMs for planning with included/excluded objects, but they merely study this problem in a limited scope (only dozens of cases) without further in-depth analysis.
Scripts A structure describing a sequence of events in a particular scenario is script (Schank and Abelson, 1975), consisting of two types: 1) Narrative script: a narrative chain of events describing a particular scenario derived from narrative texts such as recipes (Fang et al., 2022) or stories (Tandon et al., 2020); 2) Goal-oriented script (Regneri et al., 2010; Wanzare et al., 2016): an appropriate sequence of steps as instructions to achieve a goal. In this work, the steps for achieving a given goal in language planning can be categorized into the second class. Many datasets for goal-oriented scripts have been proposed to improve the language planning ability of LMs (Sakaguchi et al., 2021; Lyu et al., 2021). However, they mainly consist of abstract goals with prototypical instructions and thus are not built to train LMs for planning with more specific goals.
The constraint within each example is highlighted .
In-Context Learning With the great success of LLMs (Brown et al., 2020; Ouyang et al., 2022; Chowdhery et al., 2022), in-context learning (Brown et al., 2020; Min et al., 2022) has established its great task-solving potentials with a textual task instruction and a few examples. Moreover, when being used for dataset construction, the data samples that LLMs generate can sometimes outperform crowd-sourced human-authored data in factuality and fluency (Lu et al., 2022a; Min et al., 2022). This shows a promising alternative to costly large-scale crowd-sourcing to construct datasets using LLMs (Wiegreffe et al., 2022; Liu et al., 2022a; West et al., 2022). Inspired by these studies, in our work, we adopt the in-context learning for LLMs not only for better language planning, but also as a reliable crowd-worker to scale up the planning data into a reusable dataset to train smaller models.
3 Definitions
Before diving into technical details, we first clarify some important terms used in the paper.
Scripts A goal-oriented script is a list of steps (S = {s1, s2, · · · , s|S|}) that fulfill a certain goal (G) (e.g., “make a cake”) (Suddendorf and Corballis, 2007; Schank and Abelson, 2013). The language planning task is defined as M : G → S, where M is the planning model. Goals Different from previous studies that focus mostly on abstract goals with prototypical scripts,
we define a taxonomic structure of goals by extending the derivatives of abstract goals. We define a specific goal that inherits from an abstract one but with new information as a constraint to limit the scope. An abstract goal, denoted as Ga, refers to stereotypical activities, e.g., “make a cake”. A specific goal, denoted as Gc, is derived from the corresponding Ga with various constraints, e.g., “make a chocolate cake”.
Constraints and Constrained Language Planning To enrich the semantics of specific goals, we define three types of constraints, i.e., modifier, method and intent, as shown in Table 1. They express different angles of extending an abstract goal and can be further instantiated and concreted. Constrained language planning denotes generating a constraint-faithful script S : S = M(Gc) toward specific goals (Gc) with various constraints (C).

heading: 4 Constrained Language Planning with LLMs
text: In this section, we evaluate and enhance the constraint language planning ability of LLMs. The overall workflow is illustrated in Figure 2. We first extend the specific goals Gc from the abstract ones Ga using a human-in-the-loop acquisition approach with LLMs (§ 4.2, Step 1), and propose an over-generate-then-filter framework to
obtain scripts (§ 4.3, Step 2-3). Then, we reveal that LLMs (e.g., GPT-3 (Brown et al., 2020), InstructGPT (Ouyang et al., 2022)) are prone to be unfaithful to the constraints in Gc, and our approach can alleviate this problem (§ 4.4). We use text-davinci-002 as the default InstructGPT variant, which has ≥175B parameters.2

heading: 4.1 In-Context Learning for LLMs
text: We deploy LLMs for constrained language planning via in-context learning (Brown et al., 2020; Ouyang et al., 2022). Given a task input (X), we first write a task prompt (T ) describing the task, and then provide several examples (E = {Ei}|E|i=1, where Ei = (Xi, Yi) are used for few-shot learning). An LLM generates output (Y ) by completing the prompt (Y = M(T,E,X)). The whole process does not require any gradient update, allowing LLMs to generate new specific goals and scripts without massive training data.
2Code names and approximated parameters of GPT-3 models are based on https://blog.eleuther.ai/ gpt3-model-sizes/ and https://beta.openai. com/docs/models. Note that OpenAI does not release detailed information about later versions of GPT-3, and thus for brevity, we default its size to 175B.
Data Source for Examples We adopt wikiHow (Koupaee and Wang, 2018), a data source of instructional articles on various topics, as the initial dataset for providing examples. The articles are titled as “how to ...?”, describing abstract goals, and consist of steps to achieve them. We use the titles (Ga) and steps (S) as examples.

heading: 4.2 Acquisition of Specific Goals
text: Since no dataset of specific goals exists to support our study, we have to acquire these goals first. As elaborated in Table 1, we extend the abstract goals with multi-faceted constraints for human-in-theloop data acquisition using InstructGPT.
First, we manually prepare a pool of examples that derive specific goals from an abstract one with constraints.3 Each example is attached to a constraint type (i.e., modifier, method or intent), and contains more than one constraint and specific goal so that InstructGPT is prompted to generate multiple Gc for one Ga. Next, given an abstract goal from wikiHow, we enumerate each constraint type to ensure data diversity. Then, we sample several examples of the constraint type from the pool. Finally, we input the task prompt, examples and the Ga into InstructGPT for the completion of Gc.
An example in Table 2 (I) shows InstructGPT generates constraints “chocolate” and “vanilla” for Ga (“make a cake”) given the constraint type modifier and some examples, and completes the specific goals (“make a chocolate cake” and “make a vanilla cake”).

heading: 4.3 Acquisition of Scripts
text: After getting specific goals with constraints, we can test the ability of LLMs to fulfill them.
Planning with InstructGPT We first write a task prompt T . Given the Gc, we back-trace its Ga and extract the verbs (“make”) and nouns (“cake”) from Ga. Then we use the verbs and nouns as keywords to retrieve two similar goals as examples E from the wikiHow dataset. Finally, the task prompt T , examples E and Gc with constraint C are fed into InstructGPT. As shown in Table 2 (II), we adopt the scripts, i.e., “make a cake” and “make a cupcake” to prompt InstructGPT to generate a script for “make a chocolate cake”.
Over-Generation and Filtering Using the above-mentioned approach, generated scripts by
3Complete examples can be found in Appendix B.1.
InstructGPT are reasonable and fluent. However, they sometimes are not faithful to the constraints under closer examination (§ 4.4). Previous studies have shown that the output quality of LLMs falls in high variance (Wiegreffe et al., 2022), leading to bad performance. Thus, we adopt the idea of over-generate-then-filter to improve generation quality, which is shown to be effective in previous work (Wiegreffe et al., 2022; Liu et al., 2022a). We over-generate K sampled from InstructGPT.4
Next, a filter model is developed to select the faithful scripts. Due to the diverse expressions of language, we rely not on rules and patterns (i.e., constraint words must appear in the script), but on the semantic similarity between goals and scripts for filtering. For example, “decorating the cake with candles” could be a faithful step to make a cake “for a birthday party”. Motivated by this, we first collect a set of goals, consisting of the target goal (G+c ) as a positive sample and others ({G−c }) generated from the same abstract goal (Ga) as negative samples. In the previous case, the negatives include “make a cake in the microwave” and “make a cake for a wedding”. We convert scripts and goals into InstructGPT embeddings (text-embedding-ada002) and calculate cosine similarity as similarity scores to measure semantic similarity. Additionally, we reward the script that explicitly contains the keywords of the target constraint. We only keep the script if G+c scores the highest in the goal set.

heading: 4.4 Evaluation
text: We randomly collect 100 abstract goals (e.g., “make a cake”) from wikiHow and conduct manual evaluations on the generated specific goals and their scripts. We compare our methods with instruction tuning methods, T0 (Sanh et al., 2022) and FlanT5 (Chung et al., 2022), vanilla GPT-3 (Ouyang et al., 2022) with different sizes, Codex (Chen et al., 2021) and InstructGPT (Ouyang et al., 2022) with different sizes. We also add “Let’s think step by step” before each answer for script generation, which is a simple but effective trick to improve zero-shot reasoning for LLMs (Kojima et al., 2022). For a retrieval baseline, we directly use the goals to search and retrieve the most relevant scripts from
4In practice, K = 2 is sufficient, as shown in Appendix B.3. Intuitively, the reason this approach works is that the generation accuracy can be improved from 1− p to 1− pK (at least one is correct), where p is the probability that InstructGPT generates a wrong script.
the wikiHow website5 as results.
Are specific goals generated by LLMs of high quality? We ask InstructGPT to generate 300 (3×) specific goals for 3 constraint types based on the 100 abstract goals from wikiHow. For evaluation, we recruit annotators on Amazon Mechanical Turk to check whether these goals are correct. Each case is examined by three annotators, who reach an inter-rater agreement at Fleiss’s κ = 0.86 (Fleiss et al., 1981). InstructGPT achieves 98.00% accuracy, indicating that LLMs can derive specific goals of rather high quality.
Can LLMs write scripts for specific goals? To answer this question, we first let InstructGPT generate scripts for the 100 abstract goals from wikiHow and ask three annotators to check the correctness of the scripts (with Fleiss’s κ = 0.79). The correctness is decided by both the fulfillment of the goal and the completeness of the semantics. InstructGPT achieves 97.00% accuracy, proving that LLMs can plan for abstract goals very well. However, it is not the case for specific goals. We sample 100 specific goals from 300 generated ones (mentioned above) and evaluate the scripts generated from baselines and our method.
Table 3 reports the overall accuracy of the results.
5https://www.wikihow.com/Main-Page
FE1: No constraint
FE2: Unrelated step(s)
FE3: Incoherent
step(s)
SE3: Wrong order
SE2: Repeated step(s)
SE1: Missing step(s)
InstructGPT(6.7B) InstructGPT(13B) InstructGPT(175B) InstructGPT(175B)+step InstructGPT(175B)+ours
10
100
Figure 3: Errors of the generated scripts by human evaluation. The axis of the radar chart is in log-scale. Notably, ours reduces to virtually one dot in the graphic because it does not have many errors (0-1%). SE and FE denote semantic completeness and faithfulness error.
We find that: 1) Overall, all baselines achieve unsatisfactory results on planning for specific goals, with InstructGPT outperforming others. Especially, the scripts with intent-type constraints have the worst accuracy, and adding “let’s think step-by-step” does not help much; 2) The retrieval from wikiHow does not lead to the desired script; 3) With our method, InstructGPT can generate scripts of higher quality by a large margin; 4) Replacing the similarity function with embeddings from other pretrained models results in performance drops.
What types of errors do LLMs usually make in this task? To respond to the motivations of our methods, we conduct detailed analyses to investigate why LLMs fail. We evaluate the model planning performance in two aspects: 1) Semantic completeness (SE): whether the steps in the script are missing, repeated or in the wrong order; 2) Faithfulness to the constraints (FE): whether the script is faithful to the constraints and the steps are coherent (related) within the script. We define six types of errors upon the two, i.e., i) SE: missing, repeated step(s) and wrong order and ii) FE: no constraint, unrelated step(s) or incoherent step(s).6 Annotators are asked to review 100 scripts generated by InstructGPT and mark the error types.7 Results in Figure 3 show that: 1) The semantic completeness in generated scripts is acceptable, but the faithfulness to the constraints can not be guaran-
6The detailed definitions can be found in Appendix B.4. 7The case study of how InstructGPT fails at planning for
specific goals is shown in Appendix B.5.
6.7B 13B 175B Step Ours Arts
Vehicles Electronics Education
Family Life Business
Food Health
Hobbies Traditions
Home Personal Care Relationships
Sports Work 0.33 0.29 1.0 0.5 1.0 0.0 0.0 0.75 0.5 1.0 0.25 0.38 0.6 0.62 0.92 0.2 0.33 0.5 0.67 1.0 0.0 0.0 1.0 0.67 1.0 0.33 0.5 1.0 0.67 1.0 0.28 0.66 0.78 0.78 0.94 0.33 0.5 0.57 0.75 0.9 0.1 0.42 0.6 0.67 0.88 0.25 0.4 0.5 0.57 1.0 0.17 0.3 0.5 0.6 1.0 0.14 0.33 0.5 0.6 1.0 0.0 0.5 1.0 1.0 1.0 0.25 0.5 1.0 1.0 1.0 0.0 0.67 1.0 1.0 1.0
Loading [MathJax]/extensions/MathMenu.js
Figure 4: The heat-map depicts the human-evaluated script accuracy of different methods in different topic categories for specific goals.
teed; 2) Our method greatly improves the planning quality both in semantic completeness and faithfulness.
What kinds of goals do InstructGPT typically fail? By far, we already know that LLMs fail at specific goals, especially for intent-type constraints. We dig into more fine-grained topic categories of constraints defined in wikiHow. The heat map in Figure 4 shows that the planning performance of InstructGPTs varies considerably for goals of different categories, and the planning accuracy for each category improves greatly with our method.

heading: 5 Script Distillation from LLMs
text: Since LLMs are costly to deploy, it is essential to enable language planning ability for smaller, specialized models. Creating datasets is an inevitable step to this end. However, previous datasets do not enable planning for specific goals (Sakaguchi et al., 2021; Lyu et al., 2021), and manual dataset annotation is expensive and highly demanding. Thus, we follow the idea of symbolic knowledge distillation (West et al., 2022) to distill constrained language planning datasets from LLMs.
5.1 CoScript: A Dataset for Constrained Language Planning
We now apply our method for building a first-of-itskind Constrained Script dataset of language planning, named as CoScript. Experiments in § 4 show that LLMs can generate high-quality specific goals and scripts with our over-generating-thenfilter framework. We now scale up the experiments
for a large-scale dataset. We collect 14,945 article titles as seed abstract goals and retrieve 34,260 similar goals with scripts from wikiHow as examples to prompt InstructGPT (175B) for data generation. Following § 4, dataset construction process consists of three steps, as in Figure 2: 1) We first enumerate constraint types with examples for InstructGPT and obtain specific goals (after de-duplication) based on the seed abstract goals. 2) Then, InstructGPT over-generates K scripts for the specific goals and 3) our filter framework selects the faithful scripts as the final data.8
In total, we generate 55,000 specific goals with corresponding scripts. We randomly choose 2,000 data as the validation set and 3,000 data as the test set. To ensure the quality of the validation and test set, we ask crowd-sourced workers to find and revise the incorrect samples. By collecting the annotation data for error identification of these 5,000 samples, we estimate to achieve 97.80% accuracy for specific goals and 94.98% for constrained script generation, consistent with the results in Table 3.

heading: 5.2 Dataset Analysis
text: Script Diversity Analysis As shown in Table 4, despite the larger scale of wikiHow, CoScript has more specific goals than wikiHow and thus is valuable for the constrained language planning task. Besides, previous studies (Fu et al., 2021; Narayan et al., 2022) find that the texts generated by LMs may be too repetitive and less diverse. For this concern, we compare our CoScript with a recent goal-oriented script dataset proScript (Sakaguchi et al., 2021) created by crowd-sourcing. As reported in Table 4, 1) CoScript is much larger than proScript, with more scripts and a higher number of steps per script; 2) CoScript exhibits high lexical diversity, with more unique words than human-written proScript.
8Details about hyper-parameters and costs can be found in Appendix C.1.
Constraint Analysis Figure 5 shows the constraint distribution of CoScript. We compute the proportions of constraint types with their representative categories obtained from Probase (Wu et al., 2012), and the initial words of constraint instances. We find CoScript shows high heterogeneity and pluralism in the generated specific goals. Interestingly, InstructGPT tends to start with the word “if ” or “when” for hypothetical constraints (e.g., “if someone is lactose intolerant” for “make a cake”), suggesting the potential for future research on counterfactual reasoning in language planning. We also analyze the domain distribution of CoScript in the Appendix C.2

heading: 6 Constrained Language Planning with Specialized Models
text: With CoScript, we can train smaller but specialized models for constrained language planning.

heading: 6.1 Experimental setup
text: Baselines We use GPT-2 (causal LM) (Radford et al., 2019) and T5 (encoder-decoder LM) (Raffel et al., 2020) as baselines. Given goals, the models are trained to generate a list of steps S for planning. Moreover, we adopt the idea of retrievalaugmented text generation (Lewis et al., 2020) and add retrieved examples in the input to improve generation quality.
Metrics We use BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004) and BERTScore (Zhang et al., 2020) as automatic metrics to measure semantic completeness. We also train a binary classification model to decide whether the generated texts are faithful to the constraints. Specifically, we
collect 50,000 data from CoScript as positive samples, and shuffle the goals and scripts to construct 50,000 negative ones. Then, we fine-tune a DeBERTa (v3 large) model (Khashabi et al., 2020) for classification, achieving 91.53% accuracy on the test set.
Training Data To gain a fine-grained perspective on planning toward specific goals, we train LMs on both wikiHow (Dwitr ) and CoScript (Dcotr ), and test them on CoScript test set (Dcote ). Both datasets share similar scripts, but the goals in wikiHow are mostly abstract ones. For wikiHow, we also randomly collect 50,000 goals with scripts as Dwitr .

heading: 6.2 Results
text: The comparison for models trained on wikiHow and CoScript are shown in Table 5. In general, LMs trained on CoScript outperform that on wikiHow. T5 outperforms GPT-2 in faithfulness, possibly due to its encoder-decoder framework being better at handling input information. However, GPT-2 outperforms T5 on other text generation metrics for scripts. This could be because CoScript
is distilled from InstructGPT, leading to a biased data distribution that favors decoder-only causal language models, e.g., the GPT family.
Based on Table 5, we find that augmenting models with retrieved examples can improve semantic completeness. However, the constraint faithfulness could be undermined as models tend to mimic the retrieved examples. To further understand the role of retrieval augmentation, we conduct a manual evaluation that based on 100 random samples generated by T5 (3B) with and without retrieval augmentation. We discover that 57% of T5’s results are correct, and the number goes up to 70% with retrieval augmentation. Thus, although we observe a slight drop in faithfulness score (93.00 → 92.53 from Table 5), retrieval augmentation still brings much improvement over the base model.
Faithfulness of Constraints of Different Types Will LLMs’ planning preferences for constraint types pass to the specialized models? We find the results in Table 6 are consistent with that of LLMs (Table 3). Specialized models are also the worst at specific goals with intent-typed constraints.
CoScript vs. wikiHow We mix two datasets together with a hyper-parameter α to control the proportion of two datasets, where the new training
set Dtr = αDcotr +(1−α)Dwitr . By altering α (constant data size), the faithfulness curves in Figure 6 shows that adding more data from CoScript consistently improves model performance in constraint faithfulness. Thus, training on CoScript contributes to more faithful planners.
Specialized Models vs. LLMs We further finetune a T5 (3B) on CoScript and wikiHow to generate scripts for the specific goals in § 4.4, which are held out from the training set. Table 7 shows that T5 fine-tuned on CoScript with retrieval augmentation can generate scripts of higher quality than most LLMs in Table 3, indicating that smaller models can surpass larger models when properly trained on suitable datasets.

heading: 7 Conclusion
text: In this paper, we define planning toward specific goals with constraints. We propose a better prompting method for LLMs, and distill a novel dataset from LLMs (CoScript) to improve the constrained language planning ability of specialized models. Experiments show that our method improves the planning quality of LLMs for specific goals, and smaller models trained on CoScript even outperform LLMs. We hope the CoScript dataset will be a valuable resource to advance the research on language planning with more complex and diverse goals and constraints.
Limitations
The proposed method for improving LLMs is a post-hoc re-ranking approach, and we do not improve LLMs themselves due to the difficulty of fine-tuning LLMs. Besides, we improve the ability of constrained language planning for smaller models from the perspective of building task-related datasets, but do not consider investigating the model itself, other than adopting retrieval augmentation. In addition, because automatic metrics for generated text are limited, the automatic evaluation of this paper may result in an overestimation or underestimation of the mentioned methods, though we attempt to mitigate this by incorporating a moderate amount of human evaluation. Despite the advanced planning capabilities of newer language models, our work remains significantly valuable to the knowledge distillation of these LLMs into smaller and more cost-effective models.
We also discover several limitations of the proposed CoScript datasets. First, the specific goal
explored in this work only inherits from an abstract one with one extra constraint. However, in real-life situations, complex planning may involve multiple constraints, which we do not investigate in this work. Another limitation of CoScript is that our dataset is generated from InstructGPT, and thus the data distributions may be biased to favor causal language models. This is a common issue with machine-generated datasets, which we address by manually curating CoScript’s validation and test sets. Furthermore, there are still some incorrect samples (about 5%) in the training data without manual correction due to the limits of budget and time. Last but not least, we only consider whether the script can be executed at the human level. The script execution for robots (Huang et al., 2022; Lu et al., 2022b) is unstudied in our work, and there still exist huge gaps in transferring complex human language to one that is understandable and executable by robots.
Ethics Statement
Use of Human Annotations We protect the privacy rights of crowd-sourced workers and pay them above the local minimum wage. We use Amazon Mechanical Turk (AMT) and require 300 annotators to be located in the U.S. as a proxy for English competency. We pay at a rate of $6/hour for 20 samples. We acknowledge that constructing datasets from large language models may suffer from toxic language and cause severe risks for social society (Ousidhoum et al., 2021; Baldini et al., 2022). Therefore, we ask the annotators to discard the offensive and harmful data when reviewing the CoScript. However, there may still be prejudicial data in our final dataset that goes unnoticed.
wikiHow Source The content available on wikiHow is shared under a Creative Commons License (CC-BY-NC-SA) 9, which permits others to share, copy, distribute, and adapt the content for noncommercial purposes. In our research, we use wikiHow as an initial dataset for providing examples to construct our dataset. Our dataset is released on GitHub and is only used to advance academic research on language planning with more complex and diverse goals and constraints. Therefore, we emphasize that our usage aligns with the requirements under the license.
9https://creativecommons.org/licenses/by-nc-sa/3.0/
Covered Domains in CoScript CoScript is derived from wikiHow and encompasses 19 daily life goal categories (as illustrated in Figure 8). These categories cover a wide range of practical topics of everyday life. However, as shown in Figure 8, we emphasize that sensitive and high-risk domains, including medical, legal, and high-stakes financial advice, are excluded from the dataset to minimize potential risks related to inaccurate or misleading information. We encourage researchers and developers to leverage this dataset to build models that accurately understand and respond to user queries on various non-sensitive, non-critical topics.
Factuality, Toxicity and Biases We recognize that the factuality of generated content is crucial, especially in high-stakes scenarios. Therefore, annotators are asked to verify the consistency between generated scripts and goals with constraints for validation and test sets. They also assess and revise the content to minimize hallucinations, factual errors, and any inappropriate or misleading information.
Previous work found that LLMs may generate toxic contents (Cao et al., 2022; Liu et al., 2022b). We highlight that our dataset is not intended for safety-critical applications or as a substitute for expert advice in such domains. Annotators are specifically instructed to discard offensive and harmful data during the review of the validation and test sets in CoScript. However, despite these precautions, there may still be some prejudicial data that goes unnoticed in our final dataset.

heading: Acknowledgement
text: We thank the anonymous reviewers for their valuable comments, and Wei Shi and Shuang Li from Fudan University for their useful suggestions for the manuscript. This work is supported by the Chinese NSF Major Research Plan (No.92270121), Shanghai Science and Technology Innovation Action Plan (No.21511100401) and the Science and Technology Commission of Shanghai Municipality Grant (No. 22511105902).

heading: A Author Contributions
text: Siyu Yuan Lead the project, develop the original method and original code, lead the curation of the dataset, contribute to the original experiments, and contribute to the original manuscript.
Jiangjie Chen Conceptualization of the original idea, supervision of the research activity planning and execution, contribution to the original manuscript and figures, and acquisition of financial support for the project.
Ziquan Fu Contribute to the original idea, provide financial support for the project, revise the manuscript, and contribute to data curation.
Xuyang Ge Contribute to the experiments and data curation.
Soham Shah Provide financial support for the project and revise the manuscript.
Charles Robert Jankowski Provide financial support for the project and revise the manuscript.
Yanghua Xiao Provide financial support for the project and revise the manuscript.
Deqing Yang Provide financial support for the project, revise the manuscript, and oversee the research activity execution.
B Implementation Details
B.1 Handpicked Examples for Specific Goal Generation
We follow the instructions proposed by Mishra et al. (2022) to better construct the three examples for specific goal generation. As shown in Table 13, we turn long descriptions into bulleted lists in the task prompt for better generation. In addition, in each example, we list two specific goals with constraints, which can prompt InstructGPT to generate multiple specific goals for the given abstract goals. In our experiment, we conduct the specific goals generation with different numbers of examples and report the accuracy and the total number of generated specific goals for 100 abstract goals. The results in Table 8 show that three examples are good in our settings.
B.2 Prompts Format To explore the prompt formats on script generation, we test 100 samples mentioned in § 4.4 without using the task prompt or replacing the original words with other words. The results in Table 9 show that: 1) task prompt can help InstructGPT to better understand the task and thus can improve the model performance on script generation; 2) adopting popular words to write the prompts can better improve the effect of prompts.
B.3 Over-generation Hyper-parameters To evaluate the LLMs on planning for specific goals, we randomly sample 100 specific goals and
then generate scripts from the baseline and our method. Figure 7 reports the faithfulness to constraints. We find that: 1) the output quality of InstructGPT falls in high variance, and the script may be unfaithful to the constraints; 2) over-generation can amplify the likelihood of constraint satisfaction, and K = 2 is sufficient.
B.4 Error Types
As shown in Figure 10, we evaluate the model planning performance on two aspects, i.e., Semantic completeness and Faithfulness to the constraints (§ 4.4), and define six types of errors.
B.5 Case Study
Table 14 lists three examples by InstructGPT (175B) and our approach. The first and second examples show that the scripts generated by InstructGPT may fail in unfaithfulness to the constraints. The third examples demonstrate that although the scripts generated by InstructGPT can be faithful to the constraints, they may suffer from other error types. In contrast, the over-generating-and-filtering method can amplify the likelihood of high-quality generations and thus can make LLMs better planners.
C CoScript Details
C.1 Generation Hyper-parameters
We queried the text-davinci-002 model through the OpenAI API on June 25 to July 5, 2022. CoScript is generated under a specified license that is compatible with the conditions under OpenAI API. In total, the generation for CoScript costs about $5,000. The hyper-parameters for script generation are shown in Table 11. If both generations are faithful, we randomly select one into the dataset.
C.2 Domain Examination
As shown in Figure 8, CoScript is derived from wikiHow and encompasses 19 daily life goal categories. These categories cover a wide range of practical topics of everyday life, excluding sensitive and high-stakes topics like medical and legal matters. In addition, we adopt ChatGPT to assess each specific goal in CoScript to further mitigate risks associated with sensitive domains. The instruction is shown in Table 12. Upon examination, ChatGPT identifies that 0.24% of specific goals (132) within the CoScript involve a sensitive domain, such as "relieve head pain with medication" in the Health domain. We manually remove these data and sub-
stitute them with new specific goals (e.g., relieve head pain with meditation) to ensure the safety of our dataset. We encourage researchers and developers to leverage this dataset to build models that accurately understand and respond to user queries on non-sensitive, non-critical topics.
C.3 Qualitative Generations
We randomly select qualitative generations from the CoScript. Table 15, Table 16 and Table 17 show some specific goal generations under different types of constraints. Table 18 shows some scripts generated based on our proposed pipeline.

heading: D Crowd-sourcing Details
text: Interface Details We conduct human evaluations on Amazon Mechanical Turk. Screenshots of the instructions and annotation interface are shown in Figure 9 and 10.
Step 1: Specific Goal Evaluation We first assess the specific goals generated by InstructGPT with three types of constraints. We ask the turkers to check the specific goal whether inherits the abstract goal and contains a constraint.
Step 2: Script Evaluation In the second step, we show a script of the specific goal with actionable steps. We then ask two questions:
1. Does the script meet the constraint in the specific goal? (Yes, No, or Not sure). In our preliminary experiments, we found that the semantic completeness in the scripts generated based on InstructGPT (175B) is acceptable, but faithfulness to the constraints can not be guaranteed. This question assesses this tricky error;
2. Are the steps in the script correct in achieving the specific goal? (Yes, No, or Not sure). This question is to assess whether the script can indeed accomplish the given goal. Although we have checked the constraint in the first question, there are still other error types (as shown in Figure 10). Then, we ask the turkers to review the generated scripts. If the scripts cannot achieve the given goal, they must point out the wrong steps and select the error types.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Check grammar for the whole paper.
B 3 Did you use or create scientific artifacts? Section 5.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4.1 and Section 5.1 and Appendix C.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4.1 and Section 5.1 and Appendix C.1

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethics Statement

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5
C 3 Did you run computational experiments? Section 4.4 and Section 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.4 and Section 6.1 and Appendix C.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Appendix D

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix D

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Ethics Statement

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix D
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We do not have any human subjects research and use filtered model-generated data as datasets.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Ethics Statement

heading: Future Work
text: , “if someone is lactose intolerant” for “make a cake”), suggesting the potential for future research on counterfactual reasoning in language planning. We also analyze the domain distribution of CoScript in the Appendix C. 2