Are All Steps Equally Important? Benchmarking Essentiality Detection in Event Processes
Authors: Haoyu Wang
Hongming Zhang
Yueguan Wang
Yuqian Deng
Muhao Chen
Dan Roth
Venue: EMNLP 2023 Main (2023)
Keywords: Event Granularities
Essentiality
Event Processes
Goals
Steps
TL;DR: This paper studies the essentiality of step events towards goal events, using a high-quality corpus from WikiHow, and finds that current models significantly lag behind human understanding, highlighting the need for further research.

Abstract: Natural language often describes events in different granularities, such that more coarse-grained (goal) events can often be decomposed into fine-grained sequences of (step) events. A critical but overlooked challenge in understanding an event process lies in the fact that the step events are not equally important to the central goal. In this paper, we seek to fill this gap by studying how well current models can understand the essentiality of different step events towards a goal event. As discussed by cognitive studies, such an ability enables the machine to mimic human’s commonsense reasoning about preconditions and necessary efforts of daily-life tasks. Our work contributes with a high-quality corpus of (goal, step) pairs from a community guideline website WikiHow, where the steps are manually annotated with their essentiality w.r.t. the goal. The high IAA indicates that humans have a consistent understanding of the events. Despite evaluating various statistical and massive pre-trained NLU models, we observe that existing SOTA models all perform drastically behind humans, indicating the need for future investigation of this crucial yet challenging task.

Are All Steps Equally Important? Benchmarking Essentiality Detection in Event Processes
Haoyu Wang', Hongming Zhang', Yueguan Wang2., Yugian Deng', Muhao Chen², Dan Roth
Department of Computer and Information Science, UPenn DDpartment of Electronic Engineering, THU 'Department of Computer Science, USC [why1 6gz1, hzhangal, yuqiand, danroth upenn. wangyuel1 8@mails. tsinghua. edu. cn muhaoche@usc. edu
Abstract
Obtain the offer
Natural language expresses events with vary- ing granularities, where coarse-graines events (goals) can be broken down into finer-grained event sequences (steps). looked aspect of understanding event processes is recognizing that not all step events hold equal importance toward the completion of goal. In this paper, we address this gap by examining the extent to which current models comprehend the essentiality of step events in relation goal event. Cognitive studies suggest that such capability enables machines to emulate human commonsense reasoning about preconditions and necessary efforts of everyday tasks. We contribute a high-quality corpus of (goal, step) pairs gathered from the community guideline website WikiHow, with steps manually anno- lated for their essentiality concerning the goal by experts. The high inter-annotator agreement demonstrates that humans possess a consistent understanding of event essentiality. However, after evaluating multiple statistical and large- scale pre-trained language models, we find that existing approaches considerably underperform compared to humans. This observation high- lights the need for further exploration into this critical and challenging task!
Find internship
Get Ph.D. degree
Pass the quelif cation exam
critical yet over-
Publish papers
Datanse the dissertation
Figure I: Illustration of steps to obtain Ph.D. degree, with essential steps marked by red stars. Successfully achieving the overall goal typically necessitates the com- pletion of these crucial steps.
et al., 2023), and event-centric knowledge graph construction (Tandon et al., 2015; Zhang et al., 2021a). However, most of these studies primarily concentrate on modeling horizontal relationships between events, neglecting the internal components of an event (i.c., how an individual perceives an event mention).
Computational and cognitive studies (Schank and Abelson, 1977; Zacks and Tversky, 2001) in- dicate that humans can deconstruct a goal event into a discrete representation of finer-grained step events, ultimately facilitating the hierarchical or- ganization of event-related knowledge. As illus- trated in Figure 1, when discussing the goal event of "obtaining a Ph.D. degree", we understand that several steps may occur along the way. For in- stance, one might receive the offer, pass the qual- ification exam, complete internships, publish pa- pers, and defend the dissertation. Among these steps, some are deemed essential to the goal, while others are not. For instance, passing the qualifi- cation exam is crucial for earning Ph.D. degree, whereas securing an internship is often not re- quirement. This ability to discern the essentiality of steps pertaining to various goals equips humans with the commonsense needed address problems and carry out daily tasks. Similarly, understanding which steps are essential can profoundly benefit numerous NLP applications. For instance, event
1 Introduction
As fundamental semantic primitive unit in hu- man language (Jackendoff, 1992), events play a pivotal role in facilitating efficient communica- tion among humans and safe interactions with the world. Recently, the natural language processing (NLP) community has made significant strides in helping machines comprehend events through vari- directions, such event extraction (Grishman al.. 2005; Lin et al., 2020), event relation ex- traction (Ning al., 2018;; Wang et al., 2020a), event schema induction (Chambers, 2013; Dror
'The dataset and code are available at http://cogeomp.org page/publication view/1023.

---

schema induction (Dror al., 2023) relies on event- centric information extraction derive graphical representations of events from text. this context, understanding essentiality can enhance the qual- ity of induced schemas by eliminating hallucina- tions and suggesting the addition of missing crucial events. Moreover, grasping essentiality can poten- tially benefit intelligent systems for QA tasks (Bisk et al., 2020) and task-oriented dialogue process- ing (Madotto al., 2020).
Essential
Non-essential
Total
Number of instances Average step length
1,118
397
1,515
17.1
17.4
17.2
Table Dataset statistics of ESD, The average step length represents the mean number of tokens per step.
To achieve balance in the dataset, we enlist crowd- sourcing workers to contribute optional steps (i.e., those that could occur as part of the process but are not essential)'. We employ three annotators from Amazon Mechanical Turk', who are native English speakers, provide optional steps for each goal. To ensure high-quality annotations, we require an- notators to hold the "Master annotator" title. The average cost and time for supplying annotations are 0.1 USD and 32 seconds per instance (approxi- mately 12 USD per hour).
In this paper, we aim to assess the depth of un- derstanding that current NLU models possess re- garding events in comparison to human cognition. To accomplish this, we introduce new cogmitively inspired problem of detecting essential step events in goal event processes and establish novel bench- mark, Essential Step Detection (ESD), to promote research in this area. Specifically, we gather goals and their corresponding steps from WikiHow and manually annotate the essentiality of various steps in relation to the goal. Our experimental findings reveal that although humans consistently perceive event essentiality, current models still have a long way go to match this level of understanding.
2.2 Essentiality Annotation
Given that our task necessitates a profound under- standing of the events and careful consideration, we ensure annotation quality by employing three well-trained research assistants from our depart- ment rather than ordinary annotators to conduct the essentiality annotations. For each goal-step pair, annotators are asked to rate it as 0 (non-essential), (essential), or -1 (the step is not valid step for the target goal, or the goal/step contains confidential or hostile information)". Since all annotators are well- trained and fully comprehend our task, we discard any pair that is deemed invalid (i. by at least one annotator. This results in 1,515 pairs being retained. We determine the final label based on majority voting. The dataset statistics can be found in Table 1. Altogether, we compile 1,118 essential and 397 non-essential "goal-step" pairs. The inter- annotator agreement, measured by Fleiss's Kappa is 0.611, signifying the high quality of ESD.
2 Task and Data
The essential step detection task is defined as fol- lows: for each goal and one of its sub-steps S. the objective is to predict whether the failure of will result in the failure of G. In our formulation, G and S' are presented as natural language sentences. The construction of ESD includes two steps: (1) Data Preparation and (2) Essentiality Annotation. Details of these steps are provided below.
2.1 Data Preparation
WikiHow is widely-used and well-structured resource for exploring the relationship between goal-oriented processes and their corresponding steps (Koupaee and Wang, 2018: Zhang et al., 2020b). To the best of our knowledge, is the most appropriate resource for the purpose of our re- search. Consequently, we begin by collecting 1,000 random goal-oriented processes from WikiHow. To avoid oversimplified and overly complex processes, we only retain those with three to ten steps. Further- more, given that all WikiHow processes and their associated steps are carefully erafted by humans, the majority of the steps mentioned are essential.
3 Experiments
Recently, large-scale pre-trained language models have exhibited impressive language understanding capabilities. To assess the extent to which these models truly understand events, evaluate them using ESD. Specifically, we benchmark their per- formance by examining a range of inference meth-
"The survey template is shown in Appendix Figure 2. 'The survey template is shown Appendix Figure "We utilize tools from http::/githuh.coo//haagg FleissKappa.
'WikiHow is a community wehsite featuring extensive collections of step-hy-step guidelines.

---

ods as detailed below.
Model
|| Full
Core
Random
0.5000
0.5000
Next Sentence Prediction: To ensure that the essentiality detection task aligns with the train- ing objectives of pre-trained masked Language Models (LMs) (Devlin et al., 2019), we verbal- ize each pair of goal G and step S into two sentences: "To G", and "we must 5" Then leverage the LM predict the probability of "we must S to be the next sentence of "To G".
Corpus Statistics
0.5043
0.4987
Next Sentence Prediction
0.5659
0.5503
Perplexity
0.5934
0.5793
Intent Detection
0,5461
0.5449
Textual Entailment
0,5813
0.5630
Unitied QA Prompt with GPT-3
0.6012
0.6067
0.6358
0.6043
Prompt with GPT-4
0.6574
0.6283
Human
|| 0.8750
0.8100
2. Perplexity: We also attempt to verbalize goal- step pair into sentences and employ the perplex- ity predicted by the Language Model (i.e., GPT-
Table 2: Results of essentiality detection ESD. The best results are highlighted in bold font.
2 (Radford al., 2019)) as indicator for the predicted perplexity.
York Times Corpus (NYT) (Sandhaus, 2008) to indicate the relationship between them.
3. Intent Detection: We assess the performance of Intent Detection model (Zhang et al., 2020b), which is designed to predict the correct intent given an utterance. By setting S as the provided utterance, we employ the model to predict its corresponding goal G.
Examples of all utilized templates and prompts can be found in Appendix Table 4. Given that we have formulated the task as a binary choice problem, which differs from the output of the Per- plexity and TE models, we evaluate all models based on the AUROC score (Hanley and McNeil, 1982; Narkhede, 2018) enable fair compari- son. For Perplexity, we use the perplexity score as the predicted essentiality (lower scores are bet- ter). For TE and Intent Detection models, we use the predicted probability of "Entailment" and the likelihood of being entailed as the predicted essen- tiality. The frequency as the essentiality indication signal. For all other baselines, use the predicted probabil- ity of "Y "Yes" the predicted essentiality (higher scores are better). All experiments are conducted using the largest available models and the default hyperparameters.
4. Textual Entailment:
Another alternative is to leverage the logical inference capabilities of textual entailment model (Williams et al., 2018). In our experiment, we treat G as the premise and S as the hypothesis. If a model understands the essentiality of completing S in order to achieve G, it should be able to infer S from G.
model uses normalized
5. Unified QA: We also experiment with a SOTA QA model. Specifically, We follow the setting in Unified QA (Khashabi et al., 2020) to convert each goal-step pair into a "Yes/No" question and then use the predicted probability of "Yes" as the indicator for the essentiality.
6. Prompt with GPT-3 & GPT-4: We test the prompt-based methods as well, which have proven to be powerful for many NLU tasks. Specifically, we manually design templates to convert each goal-step pair into prompts and then ask GPT-3 (Brown et al., 2020) & GPT- 4 (OpenAI, 2023) to generate True or False la- bels based on our input.
3.1 Result Analysis
In WikiHow, each goal (e.g.. "Toast Sunflower Seeds") is typically associated with modifier (e.g., "Microwave Toasting") to provide a more precise definition. In our experiment, we evaluate whether such modifier impacts the models' comprehen- sion of the goal by employing two settings: (1) Full: we concatenate the goal and the modifier as the goal input: (2) Core: we only use the original goal as the goal input.
7. Corpus Statistics:
Last, we present the perfor- mance of a corpus statistics model to determine whether such knowledge is explicitly expressed in free text. For each goal-step pair, we first extract the central verbs from the goal and step using dependency parsing tools as their repre- sentatives. Subsequently, we employ their nor- malized co-occurrence frequencies in the New
We also report human performance an upper bound for this task. Specifically, we randomly lect 200 instances, ask three ordinary annotators label them, and report the average performance. As annotators provide binary annotations for the es- sentiality of a goal-step pair instead of real value

---

Model
Encoder
Parameter
AUROC
essentiality, we present the performance of various LM variants in Table The results demonstrate that the model size plays critical role in the suc- cess of these language models. Particularly for GPT-3, reducing the number parameters to 6.7 billion results in the prompt-basee method becom- ing ineffective. Meanwhile, it raises concerns about the diminishing gain when we further increase the model size. Given that current LMs are already extremely large and expensive to train, it may not be feasible to fully understand events by solely in- creasing model sizes and corpora. We hope that ESD can promote further research on knowledge acquisition and reasoning, fostering a deeper un- derstanding of events.
BERT-base
110 M
0.5631
NSP
BERT-large
340 M
0.5659
GPT2
117
0.5676
GPT2-Medium
345
0.5889
PPL.
GPT2-Large
774
0.5927
GPT2-XL
1.6
0.5934
GPT3-small
350
0,5512
GPT3-Medium
1.3
0,5318
Prompt
GPrge
6.7 B
0.5205
GPT3-Full
175
0.6358
GPT4
>175 B
0.6574
Table 3:
Performance with different Model Sizes. NSP, PPL, and Prompt mean the next sentence prediction, perplexity, and prompt model, respectively.
like other models, the AUROC score is equivalent to accuracy. The performances of all models under both settings are presented in Table 2, from which we draw the following observations.
Related Works
The NLP community has increasingly focused on event understanding (Chen al., 2021), with search divided into event-centric information ex- traction (IE) (Grishman et al., 2005; Lin al., 2020; Wang et al., 2020b; Lyu et al., 2021; Zhang et al., 202 lb; Feng et al., 2023) and structural event prediction (Zhang et al., 2021 Event-centric IE includes recognizing, typing events and inferring their relations (Ning et al., 201 8b; Glavas et al., 2014; Wang et al., 2023), while structural event prediction involves context-independen inferences about event structures, causality (Gordon al., 2012: Sap et al., 2019; Li et al., 2021; Zhang al., 2022), discourse (Chaturvedi et al., 2017), sum- maries (Li et al., 2021), and memberships (Zhang et al., 2020a; Chen et al., 2020; Zhang et al., 2020b; Wang et al., 2021).
The corpus statistics method performs poorly. This could be attributed to two possible reasons: (1) The triggers might not adequately represent the semantics of events, leading to the co-occurrence information between the two triggers being insuffi- cient for predicting the relationship between them; (2) Considering that essentiality knowledge is form of implicit commonsense knowledge that peo- ple seldom diseuss, it difficult directly identify references to such knowledge within raw corpora. Indirect supervision from other NLU tasks proves beneficial. The experiments involving the TE and QA models demonstrate that fine-tuning with these tasks enhances the language models' ability to better comprehend event essentiality.
Current NL. models, including the massive GPT models, still fall drastically behind human on ESD. This implies that the pre-training and fine-tuning over a limited-size dataset might not be enough to uncover the implicit knowledge we need to un- derstand events, which further proves the value of proposing this new and challenging task.
This work pertains to the second research direc- tion, focusing on the internal structure of events rather than relations between them. Unlike previ- ous event membership studies (Zhang al., 2020a; Chen et al., 2020), this work predicts the essential- ity of decomposed subevents, assessing the under- standing of internal steps by SOTA LLMs.
Almost all models? exhibit better performance when the modifier is provided. which aligns with human performance. This observation suggests that when the description of the goal event is clearer and less ambiguous, the model, similar humans, can indeed comprehend events more effectively.
Conclusion
We introduce ESD, an event understanding task assessing state-of-the-art NLP models' comprehen- sion of events by identifying essential ones for goal achievement. Experiments show that com- plex event knowledge rarely expressed in text, and current large-scale language models struggle with complex event understanding. We will release all data and code to encourage research on complex
3.2 Impact of the Model Size and Discussion
To investigate the impact of pre-trained LMs' model sizes on their abilities to understand event
Thhe only exception is UnificdQA.

---

event knowledge collection and improved reason- ing for deeper event understanding.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Tiric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish. Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of NeurlPS 2020.
Acknowledgement
We appreciate the reviewers for their insightful comments and suggestions.
Yueguan Wang was supported by the USC Viterbi-THU Summer Research Program. Muhao Chen is supported by the NSF Grant IIS 2105329, the NSF Grant ITE 2333736, the DARPA MCS pro- gram under Contract No. N66001 1924033 with the United States Office Of Naval Research, a Cisco Research Award, two Amazon Research Awards, and Keston Research Award.
Nathanael Chambers. 2013, Event schema induction with probabilistic entity-driven model. Proceed- ings of EMNLP 2013, pages 1797-1807.
Snigdha Chaturvddi, Haoruo Peng, and Dan Roth. 2017. Story comprehension for predicting what happens next. In Proceedings of EMNLP 2017, pages 1603- 1614, Association for Computational Linguisties.
This research is based upon work supported in part by the Office of the Director of National Intel- ligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2022-22072200003 under the HIATUS Pro- gram and IARPA Contract No. 2019-1905 600000 under the BETTER Program. This work was also supported by Contract FA8750-19-2-11 1004 with the US Defense Advanced Research Projects Agency (DARPA). Approved for Public Release, Distribu- tion Unlimited. The views and conclusions con- tained herein are those of the authors and should not be interpreted necessarily representing the official policies, either expressed or implied, of ODNT, IARPA, DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown, and Dan Roth. 2021. Event-centric natural language processing. In Pro- ceedings of ACI. 2021 Tutorial, pages 6-14.
Muhao Chen. Hongming Zhang. Haoyu Wang. and Dan Roth. 2020. What are you trying to do? semantic typing of event processes, In Proceedings of CoNLL 2020. pages 531-542.
Jacob Devlin. Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. Proceedings of NAACL-HLT 2019. pages 4171-4186.
Rotem Dror, Haoyu Wang, and Dan Roth. 2023, Zero- shot on-the-fly event schema induction. In Findings of the Association for Computational Linguistics: EACL 2023, pages 705-725, Dubrovnik, Croatia. As- sociation for Computational Linguistics.
Ethical Statement
Yu Feng, Ben Zhou, Haoyu Wang. Helen Jin, and Dan Roth. 2023. Generie temporal reasoning with dif- ferential analysis and explanation.
To the best of our knowledge, this work has no ethical concerns. All the collected data are anonymized, and all annotators are paid higher than the minimum payment requirement.
Proceedings of the 61st Annual Meeting of the Association for Computational I Linguistics (Volume I: Long Papers), pages 2013-12029, Toronto, Canada, Association for Computational Linguistics.
Limitations
Goran Glavas, Jan Snajder, Marie-Francine Moens, and Parisa Kordjamshidi. 2014. HiEve: A corpus for extracting event hierarchies from news stories. Proceedings of LREC 2014, pages 3678-3683.
A potential limitation of this work the data scale, which not enough for training a decent model. However, as we mainly use the dataset as test set, the current scale is enough for this purpose.
Andrew S. Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. Semeval-2012 task 7: Choice plausible alternatives: An evaluation com- monsense causal reasoning. In Proceedings of SemEval @ NAACL- 2012, pages 394-398
References
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. Pro- ceedings of AAAI 2020, pages 7432-7439.
Ralph Grishman, David Westbrook, and Adam Meyers. 2005. Nyu's english ace 2005 system description. ACE, 5.

---

James A Hanley and Barbara McNeil. 1982. The meaning and use of the area under a receiver operat- ing characteristic (roc) curve. Radiology, 143(1):29- 36.
ATOMIC: an atlas of machine commonsense for if- then reasoning. In Proceedings of AAAI 2019, pages 3027-3035.
Roger C Schank and Robert P Abelson. 1977. Scripts, plans, goals and understanding: An inquiry into hu- man knowledge structures.
Ray Jackendoff. 1992. Semantic structures, volume 18. MIT press.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020. UnifiedQA: Crossing format boundaries with single QA system. In Findings of EMNLP 2020, pages 1896-1907.
Niket Tandon, Gerard de Melo, Abir De, and Ger- hard Weikum. 2015. Knowlywood: Mining activity knowledge from hollywood narratives. In Proceed- ings of CIKM 2015, pages 223-232.
Haoyu Wang. Muhao Chen, Hongming Zhang, and Dan Roth. 2020a. Joint constrained learning for event- event relation extraction. In Proceedings of EMNLP 2020, pages 696-706.
Mahnaz Koupaee and William Yang Wang. 2018. Wik- ihow: A large scale text summarization dataset. CoRR, abs/1810.09305.
Haoyu Wang, Hongming Zhang. Muhao Chen, and Dan Roth. 2021. Learning constraints and descriptive seg- mentation for subevent detection. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 5216-5226, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Manling Li, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Gao, Heng Ji, and Kathleen McKeown. 2021. Timeline summarization based on event graph compression via time-aware optimal transport. In Proceedings of EMNLP 2021, pages 6443-6456.
Ying Lin, Heng Ji, Fei Huang. and Lingfei Wu. 2020. A joint neural model for information extraction with global features. Proceedings ACL 2020, pages 7999-8009.
Haoyu Wang. Hongming Zhang, Yuqian Deng. Jacob Gardner, Dan Roth, and Muhao Chen. 2023. Extract- ing or guessing? improving faithfulness of event tem- poral relation extraction. In Proceedings of the 17th Conference of the European Chapter of the Associa- rion for Computational Linguistics, pages 541-553, Dubrovnik, Croatia. Association for Computational Linguisties.
Qing Lyu, Hongming Zhang, Elior Sulem, and Dan Roth. 2021. Zero-shot event extraction via transfer learning: Challenges and insights. Proceedings of ACL/JJCNPP 202/, pages 322-332. Association for Computational Linguistics.
Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020b. MAVEN: A massive general domain detection dataset. In Proceedings of EMNLP 2020, pages 1652-1671.
Andrea Madotto, Samuel Cahyawijaya, Genta Indra Winata, Yan Xu, Zihan Liu, Zhaojiang Lin, and Pas- cale Fung. 2020. Learning knowledge bases with parameters for task-oriented dialogue systems. In Findings of EMNLP 2020.
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Pro- ceedings of NAACL-IHI.. 2018, pages 1112-1122.
Sarang Narkhede. 2018. Understanding auc-roc curve. Towards Data Science, 26(1 2:220-227.
Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth. 2018a. Joint reasoning for temporal and causal relations. In Proceedings of ACI. 2018, pages 2278-2288.
Jeffrey M Zacks and Barbara Tversky. 2001. Event structure perception and conception. Psychologi- cal bulletin, 127(1):3.
Qiang Ning, Hao Wu, and Dan Roth. 2018b. A multi- axis annotation scheme for event temporal relations. In Proceedings of ACL 2018, pages 1318-1328.
Hongming Zhang, Muhao Chen, Haoyu Wang, Yangqiu Song, and Dan Roth. 2020a. Analogous process structure induction for sub-event sequence prediction. In Proceedings of EMNLP 2020, pages 1541-1550.
OpenAl. 2023. Gpt-4 technical report.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188)9
Hongming Zhang, Xin Liu, Haojie Pan, Haowen Kc, Jiefu Ou, Tianqing Fang, and Yangqiu Song. 2021a. ASER: towards large-scale commonsense knowledge acquisition via higher-order selectional preference over eventualities. CoRR, abs/2104.02137.
Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
Hongming Zhang, Haoyu Wang, and Dan Roth. 2021b. Zero-shot label-aware event trigger and argument classification. In Proceedings of ACL 2021, Find- ings. volume ACL/UJCNLP 2021 of Findings of ACL, pages 1331-1340. Association for Computational Linguistics.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi, 2019.

---

Jiayao Zhang, Hongming Zhang, Dan Roth, and Wei- jie J. Su. 2022. Causal inference principles for reasoning about commonsense causality. CoRR, abs/2202.00436.
Li Zhang. Qing Lyu, and Chris Sattison-Burch. 2020b. Intent detection with WikiHow. In Proceedings of AACI. 2020, pages 328-333.

---

Model
Encoder Model
# Parameters
Formatted Input
Output Format
Next Sentence Pre- diction
BERT-large
340 M
Sentence 1: grow magno- tree; Sentence plant the seeds.
Yes/No
Perplexity
GPT-2-xl
1.6
Sentence: order to grow magnolia tree, it essential to plant the seeds.
Perplexity
Intent Detection
RoBrRei-large
355 M
prompt: Plant the seeds
logits score
choice: Grow magnolia tree
Textual Entailment
RoBgRTa-large
355 M
Premise: Grow magnolia
EntailnendCCntraaiic//uaaaa
tree Hypothesis: Plant the seeds
Unitied QA
T5-large
770 M
Question: Is it essential to plant the seeds for growing magnolia tree?
Yes/No
Prompt with GPT-
GPT33
B
Input: [Statement]: To grow Magnolia Tree, you need plant the seeds. [Answer]
Yes/No
Prompt with GPT-
GPT-4
>175
Input: [Statement]: To grow a Magnolia Tree, you need to plant the seeds. [Answer]
Yes/No
Table 4: Demonstration of used templates and prompts and summarization of implementation details. The original goal-step pair is ("Grow magnolia tree", "Plant the seeds "). Templates and prompts used different models are presented in the "Formatted Input" column.
Instructions
Summary
Detalled Instructions
Examples
this task, you are given goal name and a list steps complete this goal. You are required provide a missing optional stop.
Summary
Detailed Instructions
Examples
By optional, we mean that the goal can still happen without that step.
Summary
Detailed Instructions
Examples
Good examples
Bad examples
goal 'get Ph.D. degree"
For goal 'get a Ph.D. degree"
"Dafense the dissertation' essential step.
"Find internstiip" la not essential stop.
To finish the goal '$igoal_naman _ammea,a need the following
Step
$[stop_1]
Step
S[step_2)
Step_3 3
$(step_3)
Please provide missing optional step could be of the goal essential? (PS: You should say no)
Please write here
Figure Survey template adding non-essential steps.

---

Instructions
Whether 'S(tte))' is essontial towards finishing goal "S(process)
this task given goal name and complesing this goal. You required desermen step accensal completing impossible
Select an option
Essential (1)
The meaningo of diffrrott labols aro os follows:
Not Essential (0)
Eszential
Essential
Invalid (-1)
3
1: maild step the step should not par of the goal there exiot confidential or hestle information the descriptinn)
Examples:
For
dagree"
"Defense
stop. se 1,
For task "Get Ph. D. degree". "Find an internsiip' not an escentia
stop. You should annotate m For task 'Get a Phi D. degree". 'Sell cat" is not a step of tin shing that goal. You should annolate ao 1.
Figure Survey template for annotating the essentiality.


Reviews:
[{"review_id": "kCGzoj5fAG", "reviewer": "Reviewer_PfMd", "summary": "", "questions": "* Is there a threshold for determining a low PPL (Perplexity) in the evaluation? The other models and tasks are classification tasks, where predictions can be discretely labeled.\n* When forming the dataset, is the order of the steps considered? (i.e., priority of steps that must be done first)\n* Regarding the 1,000 random goal-oriented processes from WikiHow, does this number represent the total number of goals or the approximate total number of steps? If it is the number of goals and each goal consists of three to ten steps, shouldn't there be more than 1,500 instances?\n* Could you provide examples of instances that are selected as not essential or invalid?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments. Below are the responses to the questions:\n\nRR. How discriminating essential steps can improve the performance of language models: the experiments involving the TE and QA models demonstrate that fine-tuning with these tasks enhances the language models’ ability to better comprehend event essentiality (line 256).\n\nQ1. Threshold for determining a low PPL: we evaluate all models based on the AUROC score, so we did not set a specific threshold for perplexity.\n\nQ2. Order of the steps are not considered during the annotation. The annotators only see a pair of goal and sub-step and are asked whether this sub-step is essential to it.\n\nQ3. For those initial 1,000 goals selected, we only kept those with three to ten steps to avoid too simple/complicated processes, resulting in about 400 different goals in the dataset. \n\nQ4. Here are some examples that are selected as non-essential: \n(1) Goal: Open CSV Files - OpenOffice Calc; Sub-step: Select the “Separate By” radio button\n(2) Goal: Sew on a Wig - Preparing Your Hair and the Wig; Sub-step: Braid your hair\n(3) Goal: Draw Cute Animals - A Cute Tiger; Sub-step: Color your drawing", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Recent works, such as ChatGPT, have incorporated human preference into language models, resulting in an enhancement of the generated output's quality. Besides explicit training for natural language understanding and generation, capturing implicit facts or meanings that are evident to humans remains a challenge. To design robust and human-like language models, I agree with the authors on the necessity of creating new evaluation datasets to measure their abilities.", "reasons_to_reject": "This paper introduces an intriguing dataset and task that language models should be capable of performing as well as humans. However, it does not provide a simple solution to this problem or demonstrate how discriminating essential steps can improve the performance of language models. For example, it does not show empirical evidence that using only essential steps can enhance WikiHow QA performance. ", "questions_to_authors": "* Is there a threshold for determining a low PPL (Perplexity) in the evaluation? The other models and tasks are classification tasks, where predictions can be discretely labeled.\n* When forming the dataset, is the order of the steps considered? (i.e., priority of steps that must be done first)\n* Regarding the 1,000 random goal-oriented processes from WikiHow, does this number represent the total number of goals or the approximate total number of steps? If it is the number of goals and each goal consists of three to ten steps, shouldn't there be more than 1,500 instances?\n* Could you provide examples of instances that are selected as not essential or invalid?", "paper_topic_and_main_contributions": "This paper introduces a new benchmark dataset, Essential Step Detection (ESD). It utilizes the WikiHow dataset and involves three annotators labeling whether the sub-step of goal G is essential or not. To evaluate the dataset's difficulty, the authors conducted experiments using various models and tasks. The results indicate that current models are incapable of differentiating essential steps corresponding to goals, suggesting a new research problem for language models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "03zuhnzFoN", "reviewer": "Reviewer_eKZY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the comments. We will add results and discussions with the LLaMa model in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces an enhanced dataset based on WikiHow and proposes a new task of recognizing essential steps.\n2. Various methods were attempted to accomplish this task, and the results were compared, confirming significant differences between the model performance and human performance on this task.", "reasons_to_reject": "Additional experiments could be considered, using open-source large-scale models like LLAMA to further explore the task.\nA more detailed result analysis could be conducted specifically for the results obtained with the LLAMA model.\n\n\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces an enhanced dataset based on WikiHow, where the steps are manually annotated with their essentiality. It creates a novel task of identifying which steps are necessary in an event process. The study explores various methods to accomplish this task and compares their results.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5JH3rg5Of7", "reviewer": "Reviewer_piJ4", "summary": "", "questions": "- Do you need to differentiate between events which are strongly associated with each other (frequency) and events which have an essentiality relation to each other?\n- Was agreement higher on essential or non-essential items? (if you look at high-agreement items, are most of them essential or non-essential?)\n- It sounds a bit circular that you use perplexity predicted by a LM as predicted perplexity, can you rephrase this or elaborate on what you mean in terms of human cognition?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the comments. We will add the missing references in the final version. Below are the responses to the questions:\n\nQ1. Yes, essentiality and association are different concepts. For example, obtaining a PhD degree in computer science is frequently mentioned with doing an internship during the summer. However, doing an internship is not essential for obtaining a PhD degree.\n\nQ2. The agreement was indeed higher on essential items.\n\nQ3. We used the perplexity score as a proxy for the essentiality of a goal-step pair, assuming that it is more necessary for essential steps to be described in the same context, and therefore have lower perplexity scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very clear and the methodology is solid and clearly presented\n- The work connects to work in cognitive science about how people deconstruct goal events into smaller steps with a hierarchical structure\n-- The topic touches a raw nerve when it comes to limits of LMs and the difference with human performance (which can tap into common sense knowledge)\n- Clear human upper bound\n- Dataset provided for benchmarking\n- Sensible conclusion that essentiality knowledge is implicit / obvious and thus difficult to elicit from corpora", "reasons_to_reject": "There is some work that already tried to tap into the problem that common sense knowledge does not get explicitly mentioned. I think the authors could refer to that and also (as that work is already rather old) elaborate further and add their own conclusions on what LMs can or cannot achieve using text data when it comes to common sense knowledge.", "questions_to_authors": "- Do you need to differentiate between events which are strongly associated with each other (frequency) and events which have an essentiality relation to each other?\n- Was agreement higher on essential or non-essential items? (if you look at high-agreement items, are most of them essential or non-essential?)\n- It sounds a bit circular that you use perplexity predicted by a LM as predicted perplexity, can you rephrase this or elaborate on what you mean in terms of human cognition?", "paper_topic_and_main_contributions": "This paper presents work on commonsense events which have a clear goal and in particular at fine-grained steps of these events which are essential to the completion of the goal. They present a new task of ESD, essential step detection, contribute a corpus of goal, step pairs annotated wrt their essentiality relation and evaluate several models to find that this task is indeed very challenging for existing approaches.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Previous work on crowdsourcing event steps, which also touches on the optionality of certain steps and the difficulty in obtaining essential steps from corpora\n- A Crowdsourced Database of Event Sequence Descriptions for the Acquisition of High-quality Script Knowledge (Wanzare et al., LREC 2016)\n- MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge (Ostermann et al., LREC 2016)"}}]

Author rebuttal:

Meta-review: The paper presents an event essentiality task with respect to a goal. It introduces a dataset and benchmarks the performance of various existing models on this task. 

The paper is has many positives. First, detecting essential events is well-motivated. Not all events in steps that lead to a goal are essential. The task stands on its own for event and goal related reasoning but can be useful for downstream applications as well. The dataset is a useful addition to the broad collection of event related datasets. The paper is written clearly (given the page limits of a short paper). The primary significance of this work is the potential for improving and evaluating event essentiality reasoning in language models using this resource.

The main weaknesses are that the downstream uses of the dataset or the task is unclear. This is indeed a challenge for many event related resources and not just this specific resource. In addition to the presentation and clarity questions raised by the reviewers, I have one additional concern. The definition of what makes something essential, while it sounds reasonable, can be hard to pin down in specific cases. There is not much discussion in the paper about this aspect.
Decision: Accept-Main