PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset (2024)

Abstract: Recently, there has been growing interest within the community regarding whether large language models are capable of planning or executing plans. However, most prior studies use LLMs to generate high-level plans for simplified scenarios lacking linguistic complexity and domain diversity, limiting analysis of their planning abilities. These setups constrain evaluation methods (e.g., predefined action space), architectural choices (e.g., only generative models), and overlook the linguistic nuances essential for realistic analysis. To tackle this, we present PARADISE, an abductive reasoning task using Q&A format on practical procedural text sourced from wikiHow. It involves warning and tip inference tasks directly associated with goals, excluding intermediary steps, with the aim of testing the ability of the models to infer implicit knowledge of the plan solely from the given goal. Our experiments, utilizing finetuned language models and zero-shot prompting, reveal the effectiveness of task-specific small models over large language models in most scenarios. Despite advancements, all models fall short of human performance. Notably, our analysis uncovers intriguing insights, such as variations in model behavior with dropped keywords, struggles of BERT-family and GPT-4 with physical and abstract goals, and the proposed tasks offering valuable prior knowledge for other unseen procedural tasks. The PARADISE dataset and associated resources are publicly available for further research exploration with https:// github.com/GGLAB-KU/paradise.

heading: 1 Introduction
text: Recent breakthroughs in emergent (or lack of) abilities of large language models (LLMs) have given rise to empirical studies that employ language models as planners (Huang et al., 2022; Zhao et al.,
2023; Song et al., 2023) (i.e., agentic models) and analysis studies that investigate their planning capabilities (Valmeekam et al., 2023; Pallagani et al., 2023; Valmeekam et al., 2022). Majority of these studies use toy simulation environments like ALFRED (Shridhar et al., 2020), BlocksWorld, and VirtualHome (Puig et al., 2018) which have little lexical and domain variance and limited number of actions (e.g., verbs). Additionally, the planning task is mostly formulated as a generation problem which can only be evaluated on the closed problem domain; and models with decoder-based architectures. Hence evaluating open-domain planning abilities for a wide range of models still remain as a challenge.
Planning requires a combination of a wide range of complex reasoning abilities. One line of research focuses on distinct set of reasoning abilities (e.g., commonsense (Huang et al., 2019), arithmetic (Cobbe et al., 2021), logical (Han et al., 2022), temporal (Wang and Zhao, 2023) etc...) of language models
10085
on more realistic, open-domain text. However, such open-domain text mostly does not contain a goal or a plan, hence lack the complex linguistic phenomenon (e.g., implicit relations, complex temporal, and coreference links, large event-cause chains etc...) that is common in procedural text. On the other hand, existing studies that utilize plans as a test bed for reasoning are mostly limited to extracting direct and explicit relations (Dalvi et al., 2019; Zhang et al., 2020b), i.e., more simplistic reasoning compared to implicit reasoning. Here, we formulate the implicit planning skills as abductive reasoning skills over the procedural plans with missing information. We hypothesize that a model with planning abilities would be able to infer the warnings and tips about the plan without seeing the explicit instructions (see Fig 1). Although abductive reasoning has been studied in the past, they either employ additional source of information (Huang et al., 2019; Bhagavatula et al., 2020) or focus on consecutive elements (Zellers et al., 2018, 2019; Tandon et al., 2019), both of which diminish the notion of implicity.
In this work, we present PARADISE, an extensive, expert-curated dataset for warning and tip inference tasks covering a wide range of domain derived from wikiHow1. Unlike previous works, our tasks focus on the implicit relationship between goals and warnings/tips, bypassing intermediate steps (instructions). This requires a model to possess implicit knowledge of intermediate steps (i.e., the plan) solely based on the provided goal in the absence of explicit instructions. Furthermore, we use a question answering formulation, which allows for easier evaluation with standard metrics, and testing of broader-range of model architectures including encoder-based ones. We establish robust baselines by fine-tuning pretrained language models like DeBERTa (He et al., 2020) and zero-shot prompting with a varied set of large language models such as Mistral-7B (Jiang et al., 2023) and GPT-4 (OpenAI, 2023b). Our extensive experiments address a broad range of research questions, delving into the relationship between memorization and performance (§4.1), the differences in failures between PLMs and LLMs (§4.2), and the knowledge transfer capacity of the proposed tasks to unseen tasks (§4.5). We observe
1https://www.wikihow.com
that fine-tuning small models tailored to specific tasks proves more effective than zero-shot prompting across all LLMs, including GPT-4. However, it’s noteworthy that all models, despite these efforts, still lag behind human performance. Our exhaustive analysis also provide interesting insights such as large models getting less affected from dropping matched keywords; BERT-family struggling more with physical goals, while GPT-4 struggling with abstract, digital and social objectives; and proposed tasks providing beneficial prior knowledge to unseen procedural tasks. We release all the resources at https://github.com/GGLAB-KU/paradise.

heading: 2 PARADISE
text: Initially, we augment the wikiHow corpus (Zhang et al., 2020b) by integrating it with a recent compilation2 of 21K tutorials. The extended corpus maintains the JSON format, except for the segregation of warnings and tips, an example of which can be seen in Appendix A. Each wikiHow tutorial comprises procedural steps to achieve its objective, with some tutorials featuring step-specific or general warnings and tips. We automatically generate downstream task data incorporating these warnings and tips, as elaborated in Sec. 2.2. The process also involves expert human annotation, detailed in Sec. 2.3.

heading: 2.1 Task Formulation
text: We define warning and tip inference tasks as multiplechoice question answering tasks, in which a system needs to choose the correct warning or tip for a given goal among candidates. In this context, goals are questions, while warnings and tips are the choices. An example for both tasks can be seen in Fig. 2.

heading: 2.2 Candidate Sampling
text: Acquiring the goals and positive candidates is straightforward, involving iterative selection from each tutorial in our corpus. For negative candidate sampling, we modify the approach outlined by Zhang et al. (2020b). In contrast to their reliance solely on verbs, we note that verbs prove inadequate in capturing meaning because warnings and tips, on average, are much longer than individual steps (~40 versus ~11 tokens). This leads to the generation
2Scrape date: November, 2022
of low-quality negative candidates. To address this limitation, we enhance our negative candidate sampling strategy by incorporating embeddings of noun tokens.
We begin by encoding each warning and tip using BERT (Devlin et al., 2019). We calculate the average of verb and noun tokens, identified with spaCy (Honnibal et al., 2020). Subsequently, we employ FAISS (Johnson et al., 2019) to conduct a semantic similarity search, identifying the top three warnings and tips with the highest cosine similarity score relative to the positive candidate.
Following Zhang et al. (2020b), we randomly reassign one of the negative candidates as positive and correct the labels and goals accordingly with a probability of 0.15 to avoid sampling bias and filter the examples as described in Appendix B.

heading: 2.3 Test Set Construction
text: As our datasets are automatically generated, they may include undesired elements like multiple plausible candidates for a given goal. For instance, consider the goal “Deal with Your Step Mother”, which has a positive candidate of “Stay connected with relatives such as grandparents and close friends for extra support” and a negative candidate of “Recruit help from friends and family”. Although the negative candidate is chosen due to its high semantic similarity with the positive candidate, it is also a reasonable choice for the given goal, introducing noise into the dataset. To mitigate such issues, we employ expert annotation to validate the test splits.
The expert annotation process consists of three stages. First, experts verify that each example contains no more than one plausible candidate. Second, they meticulously examine examples to ensure that the positive candidate is genuinely relevant to the content of the associated wikiHow tutorial. Finally, experts assess the appropriateness of examples for gauging reasoning skills, excluding instances that demand expert-level knowledge or domain-specific high-level information. This annotation process yields approximately 80% of annotations as valid examples. Consequently, the test splits for each task are expanded with such valid examples from the pool of automatically generated examples until reaching the predetermined size of 500 examples.
Apart from expert annotation, we leverage the dataset cartography tool (Swayamdipta et al., 2020) to uphold the high quality of our data, probe our datasets, and gain a deeper understanding of their features. Further details can be found in Appendix C.

heading: 2.4 Dataset Statistics
text: The statistics of the corpus and final datasets are given in Table 1. We specify the validation and test split sizes as 5K and 500, respectively, with the remaining data serving as the training set. Tip inference dataset is nearly twice the size of the warning inference dataset, but the average token counts per goal and candidate are comparable (~7 for goal, ~40 for candidate). We employ a nearly uniform sampling approach across various categories to ensure a high level of domain diversity.

heading: 3 Experimental Setup
text: To evaluate language models in our tasks, we establish two setups: 1) finetuning setup for pretrained encoder LMs such as BERT (Devlin et al., 2019) and 2) zero-shot setup for large language models such as GPT-4 (OpenAI, 2023b).

heading: 3.1 Finetuning Setup
text: We fine-tune a set of models from the BERT family: DistilBERT (Sanh et al., 2019), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), DeBERTa (He et al., 2020), which show strong performance in procedural and reasoning tasks (Zhou et al., 2022; Tandon et al., 2019; Zellers et al., 2019; Zhang et al., 2020b). For fine-tuning, we concatenate each candidate (warning or tip) with the question (goal) using a [CLS] token, i.e., the model receives [CLS] question [SEP] candidate as input. Subsequently, we apply an additional projection layer, followed by a softmax function that receives the representation of the [CLS] token for each candidate. The candidate with the highest probability is selected as the answer. The models are optimized through cross-entropy loss. Further implementation details can be found in Appendix D.

heading: 3.2 Zero-Shot Setup
text: We identify five popular and capable3 large language models that are diverse in architecture, scale, avalability, and performance, namely as GPT-44 (OpenAI, 2023b), PALM-2 5 (Anil et al., 2023), LLaMA-2
3These LLMs are chosen from the models that rank high in the Hugging Face Chatbot Leaderboard.
4Model variant: gpt-4-1106-preview 5Model variant: text-bison
Model Warning Tip Random 25.0 25.0 Majority 26.0 26.0
PLMs
DistilBERT 22.44 ±3.88 21.48 ±4.84 BERT 25.52 ±4.56 26.88 ±5.02 RoBERTa 20.88 ±4.80 20.36 ±4.20 DeBERTa 23.40 ±6.76 22.60 ±8.61
Fine-Tuned PLMs
DistilBERT 82.16 ±0.79 87.48 ±0.65 BERT 83.92 ±0.68 89.80 ±0.91 RoBERTa 87.92 ±0.60 91.00 ±0.34 DeBERTa 90.68 ±0.41 93.68 ±0.48
Open-Source LLMs
70B (Touvron et al., 2023), Mistral 7B6 (Jiang et al., 2023), and Vicuna 33B (Chiang et al., 2023).
We first perform preliminary experiments with default prompts on a small subset of the validation set. We, then, iteratively refine the prompts to fit the specific model’s template. For instance Vicuna expects a certain template with ###Human and ###Assistance roles specified in the text. We use the respective model APIs, where available. Preliminary tests on the subsets were conducted for each model to identify optimal temperature and top_p parameters. The best-performing configurations were then applied to the entire datasets for a thorough evaluation. Further details on the prompt templates and parameters are given in Appendix E.

heading: 4 Experiments and Results
text: We experiment with the PLMs and LLMs on PARADISE using the setup explained in Sec 3. We also
6Model variant: Mistral-7B-Instruct-v0.1
calculate the random and majority baselines, and evaluate the human performance by averaging the accuracy of two human annotators7 on a random set of 100 examples. Our main results are given in Table 2.
Fine-tuning Among the fine-tuned models, DeBERTa performs the best in both tasks; yet, it still falls behind human performance. Considering DeBERTa’s performance in previous abductive reasoning datasets, such as CosmosQA (Huang et al., 2019), SWAG (Zellers et al., 2018), and HellaSWAG (Zellers et al., 2019), such results indicate that its success is not task-specific and its performance is transferable to warning and tip inference. DistilBERT, BERT, and RoBERTa cannot perform on par with DeBERTa and fall well behind human performance, although they perform considerably well on the proposed tasks. All models perform better in tip inference compared to warning inference.
Zero-shot Although their performances are worse than fine-tuned PLMs, proprietary LLMs perform considerably better than open-source LLMs. GPT-4, which also tops many benchmarks (OpenAI, 2023b), is the best-performing LLM, while PALM-2 is a close runner-up. One surprising finding is that the performances of open-source LLMs are not correlated with their size, as Mistral 7B outperforms Vicuna 33B and LLaMA-2 70B in both tasks. Similar to PLMs, LLMs also perform better in tip inference compared to warning inference, with LLaMA-2 70B being an exception.
To gain further insights on behaviours of the models we ask the following research questions:
• RQ1: Do the models perform well due to sim7Two university students majoring in computer science, be-
tween the ages of 20 - 24.
ple keyword matching?
• RQ2: Do different model families fail on different instances? Is there a certain pattern?
• RQ3: How does the performance compare for the explicit (i.e., directly related to a step) and implicit warnings/tips (i.e., more general and not directly related to any of the steps)?
• RQ4: Are the models also good at the reverse task, i.e., can they find the goal most related to the warning/tip?
• RQ5: Can the proposed tasks help improve performance in other procedural tasks?

heading: 4.1 RQ1: Keyword Matching
text: If the goal and only one candidate share a common keyword, an example might become trivial and the task might develop into simple keyword matching. For example, when the goal and the positive candidate are about cats while negative candidates are about other animals, the positive candidate becomes easily distinguishable. Therefore, we drop such keywords in both positive and negative candidates in order to evaluate our tasks’ dependency on keywords.
This manipulation, averaging approximately 2 tokens per candidate, induces a 4.5% change on average. As illustrated in Fig. 3, the omission of such keywords results in a 15% to 20% decrease in prediction accuracy for Pre-trained Language Models (PLMs). The impact diminishes with increasing original model performance; DistilBERT is most affected, while DeBERTa is least affected. In contrast, Language Models (LLMs) experience a milder accuracy decline of 5% to 15%. In addition to dropping keywords, we also experiment with other keyword manipulation methods as detailed in Appendix F.

heading: 4.2 RQ2: Failures of Different Model Families
text: To better understand the behaviours of the models we experiment with, we generate correlation matrices for their incorrect predictions in both tasks. As depicted in Fig. 4, models within the same group (PLMs, Open-Source LLMs, and Proprietary LLMs) exhibit the highest correlation. Notably, incorrect predictions diverge more between PLMs and LLMs, while open-source LLMs and proprietary LLMs display higher inter-group correlation. Task-wise, correlation levels remain consistent, underscoring data quality and overall consistency.
As evident in Fig. 4, DeBERTa and GPT-4 exhibit divergent failure patterns. To understand their distinctions in distinguishing PLMs and LLMs, we manually inspect instances of failure. Our analysis reveals that DeBERTa struggles more with tangible, physical, and craft-related goals, while GPT-4 encounters challenges with abstract, digital, and social objectives. We validate these findings by generating the category distribution of unique failures for DeBERTa and GPT-4, as shown in Table 3. Notably, GPT-4 falters in social and digital categories like Youth, Relationships, and Computer & Electronics, while DeBERTa encounters difficulties in tangible categories such as Sports & Fitness, Pets & Animals, and Home & Garden. Specific instances of failures
for DeBERTa and GPT-4 are detailed in Appendix G.

heading: 4.3 RQ3: Implicit versus Explicit
text: As outlined in Sec. 2, warnings and tips within the dataset exhibit a distinction: some are specific to individual steps, while others are general. Although they are related to steps from the wikiHow tutorials, step-specific warnings and tips are not matched with their associated steps manually by editors. To assess the implicit reasoning skills of language models, we curate distinct subsets of test splits for both tasks, comprising warnings and tips demonstrating high semantic similarity with steps from relevant wikiHow tutorials. Employing SBERT (Reimers and Gurevych, 2019) for encoding steps and warnings/tips, we conduct a semantic similarity search. We retain examples with a cosine similarity score
surpassing a threshold with at least one step, resulting in 35 warnings and 52 tips with high similarity scores (cosine similarity score with the step > 0.75) and 368 warnings and 382 tips with decent similarity scores (cosine similarity score with the step > 0.5).
We assess the performance of PLMs and LLMs, as detailed in Sec. 3, on subsets outlined in Table 4. Results show improved model performance as the similarity between warnings/tips and steps increases. Higher accuracy in these subsets suggests a strong capacity for implicit reasoning, given that warnings and tips become more representative of intermediate steps with increased similarity. Notably, BERT and DeBERTa excel among PLMs, while Mistral 7B and LLaMA-2 70B lead among LLMs, exhibiting the highest accuracy increase in warning and tip inference tasks, respectively.

heading: 4.4 RQ4: Reverse Inference Tasks
text: If models can effectively reason about the relationship between warnings/tips and goals, it suggests they can correctly identify the goal corresponding to a given warning or tip. To examine this hypothesis, we construct reversed versions of our tasks, requiring the system to select the correct goal for a provided warning or tip. Using the candidate sampling method detailed in Sec 2.2, we randomly select 500 examples for evaluation with fine-tuned PLMs
and LLMs. Results in Table 5 indicate zero-shot performances for PLMs align closely with the random baseline, except for DeBERTa, which achieves a 10% higher accuracy. This suggests limited inherent reasoning capabilities over procedural warnings and tips. However, fine-tuned models exhibit a significant performance increase, supporting our hypothesis. In contrast, LLMs maintain similar accuracy scores in reverse tasks without experiencing a performance loss observed in fine-tuned PLMs.

heading: 4.5 RQ5: Transfer Learning
text: To examine the impact of our tasks on reasoning over procedural documents, we conduct cross-tests between warning and tip inference tasks and out-ofdomain transfer learning on goal and step inference tasks from Zhang et al. (2020b).

heading: 4.5.1 Cross Domain
text: While categorized separately, both warnings and tips share the common objective of enhancing reader understanding in a wikiHow tutorial. As a result, they often exhibit similarities in structure and semantics, with occasional interchangeability. To assess the generalizability of Pre-trained Language Models (PLMs) across warning and tip inference tasks, we conduct cross-tests. Specifically, we evaluate PLMs fine-tuned on tip inference data using the test split of the warning inference dataset and vice versa.
As depicted in Table 6, models fine-tuned on tip inference data demonstrate comparable performance to those fine-tuned on warning inference data for the warning inference task. Conversely, models fine-tuned on warning inference data exhibit slightly lower performance than those fine-tuned on tip inference data for the tip inference task. The nearly identical results in cross tests affirm the high similarity between warnings and tips, highlighting their interchangeability.

heading: 4.5.2 Out-of-Domain
text: The goal and step inference tasks focus on identifying goal-step relationships in procedural how-to
tutorials. Goal inference involves selecting the plausible goal from candidates for a given step, while step inference entails the reverse process. Although similar to the proposed tasks, they aim to measuring explicit procedural reasoning abilities.
For both step and goal inference tasks, we finetune three BERT models: i) BERT trained from scratch, ii) BERT previously fine-tuned on warning inference data, and iii) BERT previously fine-tuned on tip inference data. We report their performances on the test split throughout the training. Notably, prior fine-tuning on warning and tip inference tasks consistently improves performance during training for both goal and step inference tasks, as illustrated in Fig. 5. The second and third BERT models exhibit significantly enhanced zero-shot performances, with average accuracy increases of 21.16% and 22.98% for goal inference, and 34.75% and 39.77% for step inference, respectively. While the performance gap diminishes, the second and third BERT models continue to outperform at the end of training, with average accuracy increases of 2.09% and 2.27% for goal inference, and 0.15% and 0.53% for step inference, respectively.

heading: 5 Related Work
text: Commonsense reasoning is a broad domain that branches into a wide range of subdomains such as linguistic reasoning (Şahin et al., 2020; Liu et al., 2022; Lin et al., 2021), abductive reasoning (Tandon et al., 2019; Zellers et al., 2019), reasoning about the physical world (Bisk et al., 2020; Khot et al., 2019; Aroca-Ouellette et al., 2021), temporal reasoning (Zhang et al., 2020b; Qin et al., 2021), etc. (Bhargava and Ng, 2022). Although there exists
some abductive reasoning tasks that involve finding the most likely explanation for a set of incomplete observations (Bhargava and Ng, 2022), they are outside of the domain of procedural language understanding. For example, CosmosQA (Huang et al., 2019) present commonsense abductive reading comprehension and ART (Bhagavatula et al., 2020) propose abductive natural language inference with narrative contexts. Thus, they depend on additional paragraphs provided by the question, which enrich the level of information provided to the model. Furthermore, abductive reasoning resources obtained from procedural texts are either artificially made difficult with targeted models (such as SWAG (Zellers et al., 2018) and HellaSWAG (Zellers et al., 2019)) or covers another form of procedural texts (e.g., natural phenomenons of WIQA (Tandon et al., 2019)). Moreover, they focus on the continuous elements (i.e. consecutive steps or events), which we believe diminishes the degree of implicity.
Furthermore our main resource wikiHow has been extensively used for a wide range of tasks thanks to its rich body of well-structured procedural documents, including but not limited to summarization (Koupaee and Wang, 2018; Ladhak et al., 2020), intent detection (Zhang et al., 2020a), reasoning (Zhang et al., 2020b), linking actions (Lin et al., 2020; Zhou et al., 2022), and next event prediction (Nguyen et al., 2017; Zellers et al., 2018, 2019).

heading: 6 Conclusion
text: There has been a growing interest in procedural data, tasks, and reasoning. However, the spotlight has been on explicit and direct relations when studying reasoning within procedural documents. To address the the lack of resources to study implicit relations and reasoning, we introduce PARADISE and strong baseline models evaluated and analyzed with extensive experiments. PARADISE contains +104K warnings and tips in total and serves as a reliable testbed for the evaluation of abductive and implicit commonsense reasoning skills of language models. Moreover, it brings improvement to zero-and-few-shot performances in out-of-domain procedural tasks. Our experiments reveal that PLMs do not possess inherent reasoning skills; yet, most of them outperform LLMs when fine-tuned. However, best-performing models from
both groups fall behind human performance, indicating room for improvement. We release all the resources publicly to further research.
Limitations
We evaluate LLMs with their respective APIs due to their proprietary nature or heavy computation costs. Therefore, although we explain our evaluation setup in detail, the performances of LLMs we evaluate might not always be reproducible due to potential future changes or deprecations of their APIs.
Ethics Statement
We utilize the content from wikiHow, adhering to the specific circumstances outlined in the Creative Commons license. We fully comply with all conditions stipulated by the Creative Commons license, and these requirements facilitate the utilization of the wikiHow corpus upon which we build.

heading: Acknowledgements
text: This work has been supported by the Scientific and Technological Research Council of Türkiye (TÜBİTAK) as part of the project “Automatic Learning of Procedural Language from Natural Language Instructions for Intelligent Assistance” with the number 121C132. We also gratefully acknowledge KUIS AI Lab for providing computational support. We thank our anonymous reviewers and the members of GGLab who helped us improve this paper. We especially thank Aysha Gurbanova, Şebnem Demirtaş, and Mahmut İbrahim Deniz for their contributions to evaluating human performance on warning and tip inference tasks.

heading: A Example JSON File
text: An example JSON file from our corpus can be seen in Listing 1.
Listing 1: Example JSON file for the procedural tutorial with the goal "Rip Your Own Jeans" from wikiHow.
1 { 2 "title": "How to Rip Your Own Jeans", 3 "url": "https://www.wikihow.com/Rip -Your -Own -Jeans", 4 "title_description": "Distressed denim is a popular style,
but buying jeans that are already ripped can be expensive. Luckily, you can create this trend yourself by roughing up the fabric with a piece of sandpaper, then snipping a hole with a pair of scissors .",
5 "category_hierarchy": [ 6 "Hobbies and Crafts", 7 "Crafts", 8 "Decoration Projects" 9 ],
10 "author": { 11 "name": null, 12 "n_coauthors": 128, 13 "blurb": null 14 }, 15 "time_updated": "April 12, 2020", 16 "n_views": 3443113, 17 "rating": { 18 "n_votes": 76, 19 "helpful_percent": 77 20 }, 21 "methods": [], 22 "parts": [], 23 "video": "https://www.wikihow.com/Video/Rip -Your -Own -Jeans ", 24 "related_articles": ["ABRIDGED"], 25 "tips": [ 26 "Washing the jeans right after ripping them causes the
fibers to loosen more and create a more distressed look.",
27 "Avoid adding rips too near the seams, as they may cause them to begin to unravel .", 28 "For exact rips, use a sewing needle to pull out individual stitches from the fabric ." 29 ], 30 "warnings": [ 31 "Don ’t make the rip too big at first. Washing the
fabric will increase the size and fray of the hole.",
32 "Never attempt to rip or fray your jeans while you ’re wearing them.", 33 "Use caution with sharp tools." 34 ], 35 "QAs": ["ABRIDGED"], 36 "refs": [ 37 "https://www.marieclaire.co.uk/news/fashion -news/how - to -rip -jeans -821", 38 "https://www.marieclaire.co.uk/news/fashion -news/how - to -rip -jeans -821", 39 "https://www.cosmopolitan.com/style -beauty/fashion/a58 592/5-easy -tricks -for -distressing -your -jeans/" 40 ], 41 "quizzes": [], 42 "other_languages": {"ABRIDGED"}, 43 "steps": ["ABRIDGED"] 44 }

heading: B Filtering Examples
text: After sampling the negative candidates, we apply a set of filters introduced by Zhang et al. (2020b) to ensure the high-quality of the pairs and the challenge they bring. However we change some of these filters as follows:
Similarity filter: We obtain the cosine similarity scores using Supervised SimCSE-RoBERTa-Large
(Gao et al., 2021), since sentence embeddings capture sentence-level information, resulting in better filtering.
Length filter: We also set an upper bound to ensure they are long (> 8 tokens) enough to contain relevant information yet short (< 128 tokens) enough to be on-point and coherent.
Category filter: We exclude examples from some categories (e.g., Philosophy and Religion, Celebrities, Holidays and Traditions, etc.) that might not be suitable for evaluating language models’ reasoning skills due to their subjectivity.

heading: C Dataset Analysis with Cartography
text: We utilize the dataset cartography tool (Swayamdipta et al., 2020), which processes a model’s behaviour on training instances (also known as the training dynamics) for mapping the dataset, to better understand the characteristics of our datasets. To this end, cartography derives three metrics from the training dynamics: confidence (the mean model probability of the true label across epochs), correctness (the fraction of times the model correctly labels an observation across epochs), and variability (the spread of the model’s probability of correctly labeling observations across epochs). With these metrics, cartography reveals three regions in a data map, each with distinct features.
Using the cartography tool, we generate data maps for warning and tip inference datasets. As seen in Fig. 6, both of our datasets have a high density in the positive ends of confidence and correctness and in the negative end of variability, indicating that BERT is capable of confidently choosing the correct warnings and tips throughout the training. However, tip inference dataset shows greater density in those ends, demonstrating higher easiness that reinforces our reasoning in Appendix H.
Additionally, we implement the noise detector (Swayamdipta et al., 2020) using a Gaussian Naive Bayes classifier model (Chan et al., 1979) to identify mislabeled instances in our datasets. We train our classifier model on a small set of equally distributed mislabeled (randomly re-assinged) and correctly labeled data instances. Although simple, it achieves 95.2% accuracy on the test set. We, then, use the
classifier model on the entire warning and tip inference datasets. Our classifier model finds 1022 noisy instances in the tip inference and 746 noisy instances in the warning inference datasets, indicating a 1.3% and 1.9% of noise respectively. Such levels of noise in automatically generated datasets with no human supervision (other than curating the test splits) illustrates the success of the filtering described in Appendix B.
D Implementation Details
We use the base versions of DistilBERT8, BERT9, RoBERTa10, and DeBERTa11 and implement them using Hugging Face libraries, namely transformers (Wolf et al., 2020), evaluate, and accelerate (Sylvain et al., 2022).
We fine-tune each model for 1 epoch with its unique set of hyperparameters that can be seen in Table 7. We use a batch size of 8 and keep the default values for all other hyperparameters for testing. In fine-tuning, we use five different seeds (42, 2717, 6802, 9893, and 7818) to conduct statistical significance analysis. In testing, we set the seed to 42.
For both fine-tuning and testing, we utilize halfprecision floating point format (FP16) with the accelerate library. DistilBERT, BERT, and RoBERTa models are fine-tuned on a single NVIDIA T4, while DeBERTa is fine-tuned on two NVIDIA T4s. Computational costs of fine-tuning each model across
8https://huggingface.co/distilbert-base-uncased 9https://huggingface.co/bert-base-uncased
10https://huggingface.co/roberta-base 11https://huggingface.co/microsoft/
deberta-v3-base
each task can be seen in Table 8.

heading: E Prompting LLMs
text: In order to effectively engage with a language model, it is essential to meticulously construct a prompt template and finetune specific parameters, notably temperature (sampling temperature between 0 and 1 or 0 and 2) and top p (nucleus sampling, where the model considers the results of the tokens with top p probability mass) (OpenAI, 2023a).
For the prompt template construction, we referred to the official API documentation of each model to ascertain the recommended templates. In cases
where no specific templates were provided, we directly used the questions as prompts, omitting any additional tokens.
Regarding the calibration of the temperature and top p settings, we initiated our tests with the default values as specified in the model’s API and playground interface. This approach was followed by a systematic tuning process to optimize performance. Notably, we observed that a lower temperature setting, as compared to the default, yielded more accurate results. This aligns with the general understanding that lower temperatures are preferable for fact-based prompts, while higher temperatures are better suited for tasks requiring creativity and an element of randomness.
The accompanying tables 9 and 10 show the specific prompts and parameter settings employed for each model (all the experiments were done in December 2023).

heading: F Additional Keyword Manipulation Methods
text: In addition to dropping, we manipulate the common keywords in the candidates using the following methods, as exemplified in Table 11:
1. Synonym Replacement: We replace such keywords with their synonyms using WordNet (Miller, 1994) with NLTK (Bird et al., 2009).
2. Replacing with a Placeholder: We replace such keywords with a placeholder word, which is simply PLACEHOLDER.
3. Replacing with the BERT Prediction: We mask such keywords out with the [MASK] token and use BERT to predict the most likely token other than the original one.
We evaluate fine-tuned PLMs on these manipulated examples. As seen in Fig. 7, synonym replacement causes the least decrease in the performance with an approximate average of 10% drop in accuracy across models and tasks. Dropping, placeholder replacement, and BERT prediction replacement closely follow each other respectively, with average decreases in accuracy varying from 15% to 20%.

heading: G DeBERTa and GPT-4 Failures
text: Specific examples that DeBERTa and GPT-4 fail can be seen in Table 12.

heading: H A Comparison Between Warnings and Tips
text: As discussed in Sec. 4.5.1, warnings and tips share the common purpose of presenting additional information to user for better instruction execution within procedural wikiHow tutorials. Thus, they generally have high similarity in semantics and structure.
Overall, warning inference poses a greater challenge compared to tip inference. We believe this is due to two main reasons. First, tips are more goalspecific, while warnings are more general, yet still distinguishable. For example, any goal that contains sharp objects might have a warning towards using those sharp objects carefully; yet, the same does not hold true for tips. Second, tips are ample in quantity; therefore, there are more examples to learn from.
Our reasoning behind tips being more goalspecific and informative regarding the procedural tutorial compared to warnings is reinforced by the following findings:
1. Both PLMs and LLMs perform better in tip inference compared to warning inference, indicating a greater capability in reasoning with tips due to tips being more directly connected to their goals.
2. Previous fine-tuning on tip inference better improves the performance in transfer learning compared to previous fine-tuning on warning inference, demonstrating that tip inference contributes to model’s learning over procedural tasks more.
3. Tip inference is affected more by keyword manipulation, showing that tip inference data is more vulnerable to keyword alteration because its positive candidates contain more goalspecific vocabulary.

heading: Future Work
text: Therefore, although we explain our evaluation setup in detail, the performances of LLMs we evaluate might not always be reproducible due to potential future changes or deprecations of their APIs