SAGA: A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Events (2024)

Abstract: Interpreting and assessing goal driven actions is vital to understanding and reasoning over complex events. It is important to be able to acquire the knowledge needed for this understanding, though doing so is challenging. We argue that such knowledge can be elicited through a participant achievement lens. We analyze a complex event in a narrative according to the intended achievements of the participants in that narrative, the likely future actions of the participants, and the likelihood of goal success. We collect 6.3K high quality goal and action annotations reflecting our proposed participant achievement lens, with an average weighted Fleiss-Kappa IAA of 80%. Our collection contains annotated alternate versions of each narrative. These alternate versions vary minimally from the “original” story, but can license drastically different inferences. Our findings suggest that while modern large language models can reflect some of the goal-based knowledge we study, they find it challenging to fully capture the design and intent behind concerted actions, even when the model pretraining included the data from which we extracted the goal knowledge. We show that smaller models fine-tuned on our dataset can achieve performance surpassing larger models.

heading: 1 Introduction
text: Understanding goals is central to human understanding of text (Foss and Bower, 1986). However, this is challenging, as slight variations in actions reported in text can lead to vastly different goals, achievement outcomes and future actions. Consider the stories in Fig. 1, where the actions of Manny (the participant) in the “original” story (top left) indicate his goal of saving a life, which he is able to achieve at the end of the story. However, small changes in the narrative can lead to vastly different inferences. Consider the three alternative stories shown: in alternative 1 (top right), a different action points to Manny having a different
goal. On the other hand, while the different actions in the other two alternatives leave Manny’s goal unchanged, future actions and goal achievement are different. In this work, we simplify the complex task of understanding and reasoning about a participant’s goal in a narrative by decomposing it into the actions, intentions, and plans that the participant takes or may make in the future.
While knowledge of participants’ intentions and how they are impacted by situations is strategic for communication and language understanding (Zhang et al., 2023; Ammanabrolu et al., 2021; Callison-Burch et al., 2022; Cao et al., 2022), acquiring the necessary goal and plan knowledge is not straightforward. First, situations in narrative text can be very complex! This is partly due to the implied nature of described situations (Vallurupalli et al., 2022), the subjective nature of someone’s understanding of a participant’s goal (Graesser et al., 1994; Foss and Bower, 1986), and our ability to understand a goal at varying levels of abstraction and specificity. Second, subjectivity and incomplete
15396
information often require prospective reasoning to identify the most likely outcome in the immediate future, inferring information that is unstated yet assumed (Hovy and Yang, 2021; Davis and Marcus, 2015). We also must be able to distinguish reasonable vs. unreasonable alternatives (Niven and Kao, 2019; Zellers et al., 2018). Both of these are challenging. Third, while pre-trained large language models (LLMs) are powerful (Brown et al., 2020; Wei et al., 2022, inter alia), they may perform poorly on tasks requiring robust reasoning (Ghosh et al., 2023; Zellers et al., 2019; Qin et al., 2019).
Our formulation ultimately supports inferences of the form “who achieved what and under what conditions,” i.e., inferences useful in application areas that rely on broader reasoning, like claim verification or question answering. We consider the rationale for “why” a participant carries out certain actions (goal knowledge) and their plans to achieve their goal by considering “what-next” type of future actions and plans (prospective knowledge). To aid our study and future work, we use our framework to analyze alternative stories and how small variations in the narrative affect what inferences can be made. Via a multi-stage annotation pipeline, we collected 6225 goal annotation sets (consisting of 886 “actual” & 951 “alternative” stories). From this rich data, we formulate multiple inference tasks of increasing difficulty, seen in Fig. 2, including what can be inferred about the goals of participants (Tasks 1 & 2), future actions that affect the goal outcome (Tasks 3 & 4) and achievement (Task 5).
We examine LLMs on these tasks; since models trained with human feedback have shown impressive performance on understanding human intent (Stiennon et al., 2020; Ouyang et al., 2022), we compare GPT versions 4, 3.5-Turbo, and various sizes of the T5 family of models Flan-T5 and T5
models. We use prompting and fine-tune models on our collected dataset to examine the differences with both both options and the benefits of finetuning smaller models. Our results and analysis show the strengths and weaknesses of these various models. We find that pre-trained larger models generally perform better than smaller models and contain less factual errors. Few-shot prompting is especially useful for tasks requiring within-story details. Some of the stories that are part of our dataset are part of the pretraining data for Flan-T5 (Wang et al., 2022; Longpre et al., 2023); we examine how models handle subtle changes in these stories and find that few-shot prompting and fine-tuning help correct these errors. Overall, we find that even the larger models struggle to hone-in on the human intent behind a set of actions. Fine-tuning and few-shot prompting improve smaller models to be competitive with, or surpass, the larger models.
We summarize our contributions as follows: (1) We construct a dataset, SAGA (Story Alternatives and Goal Applicability), of overarching goals from alternative stories to help gain a broader and deeper understanding of complex events through goal-based reasoning. (2) We developed a multi-tier pipeline that allows crowd workers to provide subjective judgements and free-form text annotations for different story versions in several progressive stages. (3) We leveraged cross-disciplinary narrative understanding research in psychology, philosophy and linguistics to inform and develop streamlined annotation and evaluation processes to collect a high quality dataset. (4) We formulated important inferences that can be made from our dataset, designed challenge tasks with these inferences and benchmarked several intent based LLMs, demonstrating areas where they under-perform humans. Our data and code are avail-
able at https://github.com/saiumbc/SAGA.

heading: 2 Related Work
text: Goals: Cross-disciplinary research in psychology argues that super-ordinate goals disentangle the temporal order from the discourse order of their actions (Ajzen and Kruglanski, 2019; Graesser et al., 1994) and that cognitive modeling of the goal oriented actions can seamlessly combine situational and commonsense knowledge leading to a deeper understanding of situations (Graesser et al., 2020; Zacks, 2020; Carpendale and Lewis, 2015).
Natural Language Understanding explored goal oriented reasoning through activities in first person narratives (Rahimtoroghi et al., 2017), locationbased actions in news-text (Jiang and Riloff, 2018), chaining and ordering of action plans from procedural text (Zhang et al., 2020a) and schema construction (Lyu et al., 2021). Leveraging participants’ goals, recently, Bellos et al. (2024) examined LLMs’ sequential reasoning capabilities. These works explore the procedural and sequential relationship between activities, plans and explicit goals. Another line of research, commonsense based implicit question answering (Lal et al., 2022, 2021; Geva et al., 2021) collected goals as answers; the goals are implied by the context but their level of abstraction is question-dependent. We collect implied goals at the highest level of abstraction from the actions and intentions of a volitional participant and from varied alternative situations.
Inferences in Alternative Situations: Proposing the common sense natural language inference task, SWAG (Zellers et al., 2018) and HellaSWAG (Zellers et al., 2019), explored sentence completion in alternative situations. While these tasks examined counterfactual reasoning in discriminative models, TIMETRAVEL (Qin et al., 2019) studied counterfactual knowledge possessed by generative models for alternative story rewriting. PASTA (Ghosh et al., 2023) introduced implied states that stories depend upon and examined the tasks of state inference from alternative stories and story rewriting for alternative states. We use the alternative stories from PASTA to examine changes in goal inference in alternative situations. Storks et al. (2021); Jiang et al. (2023) used alternative stories to examine situation plausibility through procedurally tracking natural physical laws or participants’ physical states; we examine goal plausibility through participants’ intentional actions.

heading: 3 Decomposing Goal Understanding
text: Reasoning over goals has natural subjectivity and complexity. We handle this in a few ways. First, we borrow from cross-disciplinary research (Austin and Vancouver, 1996; Gollwitzer and Bargh, 1996) to reduce the subjectivity in goals by collecting overarching super-ordinate goals: a super-ordinate goal is a complex higher order goal that is achieved through several actions. This super-ordinate goal licenses the rationale behind all of a participant’s actions in a situation. Second, we focus on intentions, which can be viewed as a representation of planned actions that lead to achieving a goal (Locke and Latham, 2002; Gollwitzer, 1993; Locke et al., 1981). Third, we capitalize on prior research that people are primed to think of future details for a goal that is yet to be accomplished (Zacks, 2020; Harmon, 2005; Keefe and McDaniel, 1993) and predict future events that help in assessing goal achievement and obtain plans (revised as necessary and when possible) that lead to achievement.

heading: 3.1 Stories and Participants
text: We select ROCStories (Mostafazadeh et al., 2016) and their corresponding alternative stories from the PASTA dataset. In PASTA (Ghosh et al., 2023), original ROCStories (“actual stories”) have up to three “alternative” stories.1 In Fig. 3 we contrast an actual vs. alternative story.
Each actual story Sa has five sentences, {Sa1 , . . . , Sa5}. As a story can include multiple participants, we first limit our analysis to participants that can act volitionally (Binswanger, 1991). We then highlight mentions of a participant Pi in each sentence, and instruct annotators to infer the intentions of the highlighted participant.
Starting with an actual story then followed by an alternate story, for each participant, we obtain J = 3 goal annotation sets based on within-story (§3.2) and after-story knowledge (§3.3). Unless otherwise specified, our 5-point Likert scales range from “most” to “least” (with a middle uncertain
1Each instance from PASTA has 4 components: a story, a participant-state supported by the story, an opposing state value (a “counterfactual” state), and an alternative story supporting this opposite state. The original ROCStories were annotated with pairs of contrasting states, such that one of the states is supported by the story and the other is supported by a minimally revised counterfactual (“alternative”) story. Also available is a list of story sentences in the original story that support the participant state. While this paper only uses the stories, rather than the states, the inherent link between those states and the novel goal and intentionality annotations we collect provide exciting avenues for future research.
option). In §4.2 we discuss evaluating these annotations. We provide a detailed list of these annotations with examples in the appendix (Table 9).

heading: 3.2 Story Goal Knowledge
text: We obtain a “super-ordinate goal” (a goal achieved through several actions) of a participant by asking for the aim of the participant’s actions, i.e., the rationale for “why” the actions are performed. To prevent comprehender-bias of selecting an immediate unfinished goal at the end of a story (Trabasso and Suh, 1993), we ask annotators to provide an overarching goal description Gij that is supported by as many of Pi’s actions as possible. For this goal, we also obtain goal success at the end of the story via a 5 point Likert scale.

heading: 3.3 Prospective Goal Knowledge
text: People can use situational context to infer future actions, especially for goals that have not yet been completed, helping assess whether the goal is achievable (Zacks, 2020; Harmon, 2005; Keefe and McDaniel, 1993). Thus we ask annotators to predict and describe a likely next action involving the participant that follows the story. To ensure the next action is logical, we ask the annotators to provide an explanation for the action appropriateness.
The next action helps determine the goal (achievement) direction, i.e., whether the goal is achievable or not with the action, via a 5 point Likert scale. When a goal is not achieved after the next action, more actions in the form of a Future Plan might help achieve it. We ask annotators to determine if a revision to the original plan observed from the story can help with goal achievement. We obtain an appropriate original or revised plan unless the annotator thinks the goal is unachievable.
While the next-action is not part of the story, it is highly likely to be logically continuous with the story, and so provides an appropriate point to consider whether the participant would be satisfied with their progress. We collect the goal satisfaction annotation because, despite the expectation that a participant would be satisfied with their goal achievement, it is possible the goal and the situational context could cause the participant to be unsatisfied (likewise satisfied with an unachieved goal) and identifying such goals is useful.

heading: 3.4 Goals in Alternative Situations
text: We expect the alternative stories Sck (k ≤ 3) from PASTA to alter goal inferences seen in an actual story Sa. A participant’s actions in Sck could indicate that their goal is different from their goal in Sa, or that their goal is unchanged from Sa albeit with an altered course of goal achievement. To capture these variations, we identify whether the participant Pi in Sck intends to achieve the goal Gij (annotated for Sa) and achieves them in Sck .
We notice that revisions made to obtain an alternative story can lead to missing context for the specific goals we want to reason about. This increases the difficulty in reasoning and, given the subjective nature of story understanding, can lead to diverging goal inferences and annotations. To control for this, we obtain annotations from 3 workers and take a majority vote (see Table 10 in the appendix for IAA). Our goal inference annotations as shown in Fig. 3 (in the right column) include grading Sck on its overall coherence (obtaining a list of contributing sentences when a story is confusing). When story coherence allows goal reasoning, we obtain annotations for whether Pi intends to and achieves the goal Gij in Sck .
For obtaining a complete goal annotation set, for each intended goal we also collect prospective goal knowledge using the Sck context but use a single annotator as was done for the actual story. Since
Sck was obtained from Sa with minimal revisions, when possible, we encourage minimal revisions to the Next Action and Explanation by providing these values from Sa and allowing the annotator to update them as needed.

heading: 4 Dataset Collection & Evaluation
text: We annotated ROC stories (Mostafazadeh et al., 2016) that have corresponding alternative stories in the PASTA dataset (Ghosh et al., 2023). We selected up to 5 “story participants,” defined as volitional entities that were mentioned relatively frequently in the story. This yielded participantspecific story instances for a random 886 actual, unrevised stories from the PASTA dataset. In Appendix A.1 we provide additional details on the processing. We annotated all of the participantspecific story instances for these selected stories and all the alternative PASTA stories corresponding to over 50% of these stories. See Tables 1 and 2 for high-level dataset and annotation statistics.

heading: 4.1 HIT Design & Worker Compensation
text: We developed 3 different HITs for our tiered approach to annotation. We first collect 3 sets of goal annotations for each volitional participant in an actual story (HIT 1), then apply the collected annotations to each corresponding alternative story identifying if the goal is valid using multiple annotator judgements (HIT 2) and for the valid goals obtain new annotation sets suited to the alternative context (HIT3). See Appendix A.2 and Table 9 for more details on HIT design. Aiming to collect strictly human annotations, we followed the approach used by Zaidan and Callison-Burch (2011) and converted each participant specific story into an image to disallow easy copying of text into ChatGPT or other such AI tools. Our HIT price was calculated with the aim to pay $12-$15 per hour (see Appendix A.4.3 for details). Our protocol is
IRB approved.

heading: 4.2 Dataset Quality and Analysis
text: We assess the quality of our annotations using rational and narrative understanding-based criteria described below. Additionally we use the same criteria for evaluating model generations.
Aiming to have completely different stories in our training, dev and test sets, we evaluated all goal annotation sets for all participants for 100 actual and all 209 corresponding alternative stories covering 390 actual and 796 alternative annotation sets (≈ 20% of the annotations). We obtained 3 crowd evaluations paying an average of $.30 per evaluation (see Appendix A.5.1 for more details). We obtained an average inter-annotator agreement of 80% for all annotated features, using a weighted Fleiss’s Kappa (Marasini et al., 2016), indicating very high agreement. Our experts’ evaluations show high agreement with the crowd, indicating high quality annotations; Table 10 shows per-feature scores.

heading: Story Goal Knowledge Evaluation Criteria:
text: Based on the argument that goals are rational statements that should be logical (Graesser et al., 2020; Setiya, 2011; Grice, 1971; Davidson, 1963), we extend the two qualities required of a narrative to be logical (Graesser et al., 2020; Pennington and Hastie, 1992) to the goal description: coherence and explainability. We evaluate the goal description on coherence which measures how consistent the described goal is, i.e., the degree to which it makes sense, without any conflicting or made up information. explainability measures whether the participant’s purposive actions, even when unsuccessful, are addressed (and can be explained) by the goal description. Since these qualities inherently require goals to be truthful, faithful and purposive, we explicitly evaluate if the goal description is truthful and faithful for the story context and if it reflects
the participant’s intentionality. We also evaluate whether the goal achievement is accurate based on the story information.
Prospective Knowledge Evaluation Criteria: The Next Action and Future Plan are extensions to the story and as such should be consistent with it, using the same entities, time and space constraints as the story. We use Cohesion to measure the consistency with story events and any dissonance with story events lowers this score. Additionally, the Next Action should be logically consistent with the story without contradicting common sense or story information when using new information and we measure this with Coherence. The Future Plan should also be logically consistent with the participant’s goal without contradicting story information and provide an appropriate goal achievement plan which we measure with Correctness. We verify the suitability of Explanation to justify the next action and the correctness of goal achievement direction, participant’s satisfaction and plan type annotations.

heading: 4.3 Data Splits
text: Our evaluation resulted in quality scores for 13 features for each annotation set. While we use these to ensure high overall quality, given the depth of knowledge we are probing, there can be legitimate nuance and variability in reasonable responses. We therefore further use these scores to create our train/dev/test splits. Our process resulted in evaluation (dev/test) sets where a majority of workers had high agreement on all 13 features. We provide extensive details of our scoring process in Appendix A.5.2 and data quality details in Table 11.

heading: 5 Models & Evaluation

heading: 5.1 Models
text: We examine a number of well-known pretrained and large language models.
Instruction Fine-tuned with RLHF Language Models Both GPT-3.5 Turbo and GPT-4 models are powered by IntentGPT (Ouyang et al., 2022), a model architecture that was fine-tuned to generate outputs aligned with human intentions of being helpful, truthful and harmless, when responding to a query. We want test to what extent this alignment with human intentions extends to goals reasoning.
Multi-task trained Language Models FlanT5 (Chung et al., 2022) and T5 (Raffel et al., 2020)
are text-to-text models that can be used for any task that can be converted into textual input and output format. T5 uses transfer learning where the model is pre-trained on a mixture of unsupervised and supervised tasks in a multi-task setting. Flan-T5 extends T5 pre-training with fine-tuning on 1.8K tasks and incorporates chain-of-thought prompting, both of which improve model performance on human instruction following tasks (Wu et al., 2023). Since instruction following hinges on understanding intent we want to test whether this extends to goal reasoning. We want to explore to what extent Story Cloze (a task based on ROC Stories that is part of the 1.8K tasks) influences goal inferences from our dataset.

heading: 5.2 Evaluation Metrics
text: We use both manual and automated metrics for our text generation tasks (Tasks 1, 3, and 4). Our automated metrics consist of the classic ROUGE (Lin, 2004), METEOR (Lavie and Agarwal, 2007), corpus and Google’s version of sentence BLEU (Papineni et al., 2002)) and BertScore (Zhang et al., 2020b) for text and F1, weighted F1 and macro F1 for NLI tasks 2 and 5. We compute these metrics using the evaluate module in HuggingFace; we compute the score for the generated text using 3 reference labels (obtained from the 3 annotations per story participant) and report results for the test split (and some dev split results in the appendix).
For Human Evaluation we randomly sample 100 test and 50 dev generations taking one from each story (equal numbers from actual and alternative story contexts). We evaluate these using the same criteria we used for annotations described in §4.2 with 3 workers. For each evaluated feature we report the overall average of Likert scores for all samples and the number of generations where the average score from the workers is ≥ 3.

heading: 6 Inferences from Goal Knowledge
text: Annotations in our dataset mostly consist of implied information often unstated in the story. Inferring a participant’s goal, future goal achieving actions and plans across a variety of alternative story contexts requires a deeper understanding of text using commonsense and world knowledge. We present a variety of goal based inference tasks that benchmark GPT-3.5 Turbo, GPT-4 and various sizes of Flan-T5 and T5 using prompting and fine-tuning. Refer to Table 25 for our task prompts.
We explored in-context learning using 0 to 6 examples (balancing labels in classification tasks). Performance improves slightly with more examples but a 3-shot setting allows for an even comparison across models; GPT models produce long multisentence generations in a zero-shot setting, unlike the short Flan-T5 generations; token length of a base Flan-T5 and T5 limits us to a max of 3 complete examples without any text cut-off. T5-11b model generates additional gibberish text along with task related generation making human evaluation cumbersome; we only report automated scores for this model. We fine-tuned several sizes of FlanT5 and the T5 base model on our training dataset consisting of both actual and alternative stories.2
We compare all models and settings using automated evaluation metrics described in §5.2 and some models using our human evaluation metrics. We present human evaluation metrics in the main paper and present automated metrics in the appendix. The human evaluated generations across the various models are based on the same stories, but their reference goal annotations may differ (for these we report variance for the averaged scores). Scores significantly3 lower than reference scores with a p-value < .05 are underlined.

heading: 6.1 Task 1: Goal Inference
text: In this task, we compare model performance on goal generation for an actual story context. This is a standard sequence generation task where models generate a goal for a given (Sa, Pi).
Are models capturing the intent behind participants’ actions? Results in Table 3 show larger models are better at generating goals that explain a participant’s actions staying mostly truthful and faithful to the story. GPT models tend to summa-
2Due to memory limitations, we were unable to fine-tune larger models like the 11B Flan-T5-XXL model on a single 80GB A100. Nevertheless, our results demonstrate that finetuning smaller models can lead to strong performance.
3Statistical significance was computed using mlxtend
rize the story capturing most details, although, GPT4 generations are more succinct. We find that succinct goals (from both GPT-4 and Flan-T5-XXL) are more overarching, capturing intentionality better, but, are likely to be less truthful and faithful to the story due to their lack of expressivity when compared to the more descriptive GPT-3.5 Turbo generations. Pretrained Flan-T5 generations are generally succinct but the base model performs poorly as the identified goals tend to focus on later story events leading to reactionary goals and do not have a participant-focus leading to wrong goals in multi-participant stories. Fine-tuning the base T5 and Flan-T5 lead to more participant-focused generations but still contained reactionary goals. 3-shot prompting helped the fine-tuned model to generate overarching, faithful and truthful goals surpassing the performance of larger models (as seen in model generation examples in Table 20). Both fine-tuning and in-context examples help a model capture the intent behind a participant’s actions. Automated results do not reveal any differences; see Table 19 and Table 3 for additional models and metrics. Supplemental examples are in Table 20.

heading: 6.2 Task 2. Inference Transferability
text: We compare models at identifying the transferability of goal from an actual to an alternative story (a binary inference where a model identifies if goal Gij obtained from Sa is applicable to Sck ). Since identifying ‘a goal that is not applicable to a story’ belongs to negative knowledge that LLMs struggle with (Chen et al., 2023; Hossain et al., 2022), we evaluate models on identifying the negative label (‘a goal not applicable to the alternative story’).
Can models transfer goal inference to alternative stories? Larger models are better than smaller models especially at identifying positive labels in
full agreement situations as seen in Table 4 and Table 13. Lower performance on this task is attributable to two aspects. (1) We assigned labels through a majority label assignment; while this works in many cases, some stories are nuanced and open to legitimate interpretation. This nuance is reflected in both label types leading to an overall lower model performance. (2) Goals that do not apply to the alternative story generally require additional reasoning (such as consideration of story conditions that rule out goal applicability) making the negative label identification more difficult. Incontext examples lead to an improvement in GPT models but do not change Flan-T5 model predictions. Comparing performance in full and partial agreement settings we show that fine-tuning on our data boosts model performance helping smaller models outperform larger pretrained models.

heading: 6.3 Task 3: Explainable Next Actions
text: We consider the task of generating an action most likely to happen after the story and justifying it taking a participant-centric view. This is a standard sequence generation task, where for a story and selected participant ((Sa, Pi) or (Sck , Pi)) a model generates explainable actions. These are actions followed by the explanation with the connecting phrase “and the reason for this action is.”
Can models generate explainable participantcentric actions? Larger models generate more coherent and cohesive next actions; GPT models generate good explanations as seen in Table 5 although 3% are code generations instead of text. Pretrained Flan-T5 models struggle to generate explanations with the actions: the base model generates explanations for only 60% of actions and XXL for 90%.
The performance for generating next actions for alternative stories was comparatively lower than for actual stories. We noticed that larger models tended
to infer next actions for the alternative stories that were not consistent with the alternative story, but rather consistent with the original story. We show examples in Table 22. We note that ROCStories are part of the fine-tuning instruction sets used in the Flan-T5 model pretraining (Longpre et al., 2023; Wang et al., 2022); it is possible this leads to some degree of memorization and impacts the generalization ability of models. Smaller models fine-tuned on our data do not contain these errors and generate explanations for all actions. However, 3-shot prompting of fine-tuned models does not help on this task as it leads to generations that repeat story actions; lower scores reflect this logical inconsistency. See Tables 14 and 21 for more results.
Inference Transferability: Since next actions and their explanations from Sa are not always logical when applied to Sck , our annotations include updated next actions and explanations. We cast identifying whether the next action is most likely, somewhat likely or unlikely as a 3-way entailment given Sck and the next action from S
a. Similarly we cast identifying whether an explanation logically justifies the next action as a binary entailment.
Are models able to identify participant-centric next actions and justifications? Although larger models are better at generating next actions, they struggle to identify the likelihood of next action classifying most as “unsure” (see Table 6 and, in the appendix, Table 15). They are better at identifying justifications as with the generations, although, the performance on positive labels is better than the negative labels. Fine-tuning on our dataset helps smaller models reach or surpass larger models.

heading: 6.4 Task 4. Goal Achievement Plan Inference
text: This task generates a goal achievement plan (for the unachieved goals, roughly a fourth in actual
stories and a half in alternate stories) and identifies whether it is based on the participant’s original intent or revised based on the story outcome. We found joint generation caused all models to incorrectly identify nearly all plans as revised. This suggests a potential limitation in the models. We therefore examine generating a plan and identifying a plan type separately.
Do models generate reasonable plans? We show in Table 7 that fine-tuning on our data helps smaller models generate plans score on par with GPT plan generations; they are better at identifying plan type but are unable to generate a plan reflecting the correct type. GPT models generate generic plans which did not always address the specific situation and they are not of the expected plan type. Fine-tuning generates story specific plans and more of the expected plan type as indicated by the human evaluation Type score. See examples in Table 24 and additional metrics and models in Table 23.

heading: 6.5 Task 5. Goal Achievement Inference
text: We examine whether models can identify when a goal is achieved in the story (Task 5a); (possibly) achieved after the story with the next action (Task 5b); and whether the participant is satisfied with the goal achieving (Task 5c). We cast each task as a 3-way entailment, given Sa (or Sck ), a next action with explanation and a participant’s goal.
Are models able to distinguish between implied details across story variations? Identifying goal achievement within a story is limited by a model’s ability to identify implied achievement and degree of achievement. As seen in the first 2 columns of Table 8, larger models are better, but smaller models improve with fine-tuning. Achievement is implied in fewer alternative stories than actual stories leading to more negative labels which larger
models are better at identifying. GPT models improve with 3-shot prompting and they are able to identify some of the unsure instances (< 5% are unsure labels) improving performance. Fine-tuned smaller model performance is on par with larger models and are able to identify unsure labels in alternate stories.
The next action after a story helps identify achievable vs. unachievable goals and thus influences a participant’s satisfaction. Larger models are better at identifying after-story achievement and participant’s satisfaction for actual stories. 3- shot prompting helps improve smaller models, but the constraints placed by the examples dampen the performance of larger model slightly on identifying achievement but improve the satisfaction identification. All models perform poorly on alternate stories, though some errors that may be attributable to memorization are corrected with 3-shot prompting. Smaller models fine-tuned on our dataset also identify unsure labels outperforming larger models.

heading: 7 Conclusions
text: We showed examining goal achievement in alternative stories can lead to a deeper and nuanced understanding of complex events. Focusing on a specific participant’s actions we developed a multitiered crowd sourcing process to obtain 6.3k goal annotation sets for 1.3K alternative stories. We captured highly subjective story aspects with our annotations and validated 20% with high inter-annotator agreement. We formulated 5 inference tasks and several sub-tasks to evaluate current SOTA intentfollowing LLMs. Our evaluations show that each model differs in specific aspects of goal reasoning providing multiple future avenues to study. We think modeling advances can achieve a broader and deeper narrative understanding and hope that our work can help further this research.

heading: 8 Limitations
text: We acknowledge our work has the following limitations:
1. While our annotations are based on well known NLP data sources, our efforts focus on more formal written english. We tried to control for human subjectivity when trying to identify with participants in the stories we annotated through specific instructions.
2. We use pre-trained large language models in our experiments. These models can echo biases and mis-information either implicitly or explicitly. We do not attempt to control for these in this work.
3. Model generation and classification abilities can vary as the formality, style, or language change across the crowd written stories we annotated.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their comments, questions, and suggestions. We would also like to thank Sayontan Ghosh and Niranjan Balasubramian for early discussions and feedback. This material is based in part upon work supported by the National Science Foundation under Grant No. IIS-2024878. Some experiments were conducted on the UMBC HPCF, supported by the National Science Foundation under Grant No. CNS-1920079. This material is also based on research that is in part supported by the Army Research Laboratory, Grant No. W911NF2120076, and by the Air Force Research Laboratory (AFRL), DARPA, for the KAIROS program under agreement number FA8750-19-2-1003. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either express or implied, of the Air Force Research Laboratory (AFRL), DARPA, or the U.S. Government.

heading: A Appendix
text: No AI assistants were used for writing either this paper or code for producing the work in this paper. All writing is original and produced by the authors.

heading: A.1 Story Preparation
text: This section provides additional details for the annotation protocol described in §4.
For stories in the ROC story collection, we identified entities using NER4 and selected the types that are capable of employing volition to influence or achieve a desired state.5 In each story, we identified volitional entities using dependency parsing 6 and identified any co-referring entity mentions using an automatic entity coreference system7. We selected up to 5 volitional entities (aka. story participants) with the most mentions in a story and created participant-specific story instances.
As a first step in the annotation process, 3 crowd workers verified that the selected participants are volitional entities. When a worker identified a participant as non-volitional, we verified and discarded the participant-specific story (Total discarded :29). For the remaining 995 participant specific stories, we obtained 3 goal annotations and for a random 50% of these stories we annotated all the corresponding counterfactual (in our terminology, “alternative”) stories in PASTA. We obtained a total of 2985 actual and 3240 alternative goal annotations using a chain of Human Intelligence Tasks (HITs) to identify a participant’s goal in story and follow its achievement-arc in both the actual and alternative stories (for the annotated alternative stories). See §4.1 for HIT design and compensation.

heading: A.2 HIT Designs
text: This section provides supplementary details for the HIT design described in §4.1. We developed 3 different HITs for collecting annotations for the actual story and the corresponding alternate stories. As a reminder, each story can have up to 3 alternate stories. In all three HITs, crowd workers are provided general annotation instructions along with a consent notice allowing them to leave the HIT if they choose not to annotate. If they choose to annotate they are instructed to read a story displayed in the left column in a graphical format and
4using Spacy https://spacy.io/ 5volition capable types include EVENT, LAW, PERSON,
NORP & ORG 6volition-capable entities that are subjects of predicates 7using SpanBERT (Joshi et al., 2019)
follow instructions in the right column to produce a goal annotation set. We also provide several annotated examples, highlighting both good and bad annotations for easy reference.
The actual story annotation set collected with the first HIT captures three types of inferred knowledge: 1) the selected participant’s overarching goal information as inferred from the story events, 2) anticipatory information of what happens next involving the participant after the story and whether it helps the participant with their goal achievement and 3) the participant’s satisfaction level with their goal achievement and their subsequent plan, possibly involving a revision, to achieve their goal. The revision and plan are annotated only for goals that have not already been achieved in the story or later with the next action. This includes goals that are not fully successful in the story and where the next action does not fully help achieve the goal. The revision annotation decides if the collected plan is a revised plan or an original plan. If the next action does not help or contradicts with the goal achievement and a revision is not possible, the goal may not be achievable and we do not require a plan annotation (we do not force an annotator to provide a plan when one is not possible for the goal and story context). Additionally, to understand annotator preference for the goal, we ask annotators to relate each story sentence to the goal using one of 6 story-goal relations: enabling the goal justifying the goal, blocking the goal, being the effect of an event in another sentence, being related to another event in another sentence but not related to the goal, being unrelated to the goal. Relating this annotation to the example in Fig. 2, the participant Marie’s goal “to enjoy some me time.” is enabled by sentences 2 & 3 and prevented by 4. We use these relation annotations during evaluation.
The second HIT verifies two distinct aspects of an alternative story, Sck : whether Sck makes sense and whether a selected participant still aims to achieve the goal (annotated for them in the Sa) in Sck . We estimate that 10-15% of the alternative stories in PASTA are under-specified and do not make sense from a goal annotation perspective (due to the constraints placed on implied states and story rewriting in PASTA). In this HIT, we present an alternative story and ask 3 crowd workers to identify if the story makes sense (selecting from 3 possible options: ‘Does Not Make Sense’, ‘Somewhat Makes Sense’ and ‘Fully Makes Sense’) and select
the story sentences that lead to the incoherence in the story when it does not ‘Fully Make Sense’. We keep track of this annotation to assess if story incoherence affects model performance in alternative situations (performance decrease is <.5%).
The selected participant’s actions in the alternative story may no longer aim to achieve the goal (which they aimed to achieve in Sa). To confirm if the goals annotated for Sa are still valid in Sck , in this second HIT, we present the 3 goals collected for Sa and ask which goals can be inferred from the participant’s actions in Sck and whether the inferred goals are achieved in the story. If all 3 annotators find that a goal cannot be inferred from Sck , we mark the story as not goal transferable and do not collect any further annotations. When any of the workers annotates a goal as inferrable, we use another HIT to follow the achievement arc for the goal and obtain further annotations. The decision to gather goal annotations even if a single worker identifies the goal as inferrable was based on the high IAA (96% weighted fleiss-kappa) for the 3 workers on whether a goal can be inferred from the participant’s actions. However, we use the majority agreement for the gold label of goal transferability.
We use a third HIT when one or more annotators identify that a participant’s goal is inferrable from Sck . With this HIT we obtain a new set of goal
annotations reusing and modifying the free-form text annotations from the actual story to obtain annotations that are also minimally updated reflecting the process used for obtaining the counterfactual in PASTA (Ghosh et al., 2023). See Table 9 for the annotations and the HITs used for obtaining them.

heading: A.3 Screenshots of Annotation HITS
text: We present a few screen shots of the annotation HITs to show our general design. Fig. 4 shows the annotation instructions for this HIT, while Fig. 5 shows the main goal annotation HIT. Our additional annotation and evaluation HITS are similar; in the interest of space utilization we are not including them here, though the templates are available with our released code.

heading: A.4 Crowd Sourcing Setup

heading: A.4.1 HIT Streamlining
text: We ran several alpha runs of HITS (both for annotations and evaluations) and streamlined our instructions and examples until we were able get focused responses even with the subjectivity elicited by some of the annotations. For example, what happens next after the story or a future plan to achieve the goal can lead to widely varying annotations. This was observed in the evaluation IAA for both annotations and model generations.
Figure 5: Screenshot of HIT used in the goal annotation in actual stories.

heading: A.4.2 Worker Selection and Qualifications
text: For our initial alpha runs of the HITs we used all workers who meet our community standard quality criteria, such as requiring a 98% or greater HIT acceptance rate and the completion of 1000 approved HITs. In addition, we required the worker’s stated location to be in the USA, UK, Canada, Australia, or New Zealand. We used the location requirement to avoid language-based artifacts given the language-dependent semantic phenomena and the high subjectivity our work can elicit. We did not use requester-generated qualification tests, though in early iterations we found that annotators who had completed at least 50 HITs in our prior work (Vallurupalli et al., 2022) provided the most reliable annotations; the vast majority of our responses are
from this group.

heading: A.4.3 Annotations and Pricing
text: Our dataset contains 1024 participant-specific stories, each annotated by 3 workers. Of these 29 stories were discarded because one or more annotators identified the selected participants to be nonvolitional. For the remaining 995 stories and their corresponding 1060 counterfactual stories from PASTA, we obtained 6225 goal annotation sets. Workers were paid an average of $0.60 for each set of goal annotations for the actual story and an average of $0.45 for each set of alternative annotations. We targeted a pay of $12-$15 per hour and calculated HIT prices based on the average time spent by annotators on several test batches (we later verified the average time with actual annotation batches).
We believe the nature of this work allows us to obtain informative and generalizable knowledge in our subject area circumventing the typical positivity bias seen in AMT work (Matherly, 2018).

heading: A.5 Evaluation and Quality Analysis

heading: A.5.1 Annotation Evaluation
text: The type of annotations and the nature of the stories in our dataset elicit a variation in the text style and format even from the same worker. Our experiments (§5) did not uncover any easy biases attributable to the small number of workers selected for annotation. Since the second annotation HIT employed 3 crowd workers for the goal inference annotations, we report the IAA for the workers for these annotations in Table 10.
We evaluate the remaining annotations using 4 different evaluation HITs. The first evaluation HIT verifies the correctness of the assigned relationship between each story sentence and the 3 goals obtained for each story. With this HIT we evaluated 75 actual stories and the 225 annotated goals for these stories and verified that the described goals were overarching goals based on as many of the story sentences as possible. The other 3 evaluation HITs evaluate the goal and prospective knowledge annotations in actual and alternative stories using criteria described in §4.2. We also have an expert
(the first author) evaluate 20% of the evaluations and obtained a combined crowd and expert IAA. The quality of our goal annotations in actual stories is quite high with a crowd IAA above 80% (except for the more subjective aspects of faithfulness to the story and participant intentionality). We believe the difference between crowd and combined (crowd and expert) IAA is between 1-9% shows that our crowd evaluations are of the expected quality. Goal transferability to the alternative story with our crowd IAA of 96% shows that our goal annotations for the actual story are of high quality and their applicability to the alternative story is well justified. The quality of our prospective knowledge annotations is quite high with an IAA above 80% in actual stories and above 70% in alternative stories (for all features except for goal direction because when a goal succeeded in the story the next actions and plan are subjective).

heading: A.5.2 Quality Analysis and Data Splits
text: For each annotation set, our evaluation using the above criteria resulted in scores for 13 features (not counting the evaluation of sentence features). While the IAAs for all the evaluated features are good, we need a quality rating for an entire annotation set. For this, we look to reasons for disagreement between the workers. Since we evaluate 3 goals for each participant in a story, 12 workers comprehend each participant specific story either to annotate or to evaluate (3 annotators and 9 evaluators). We find that both nuances in semantic understanding and incorrect annotations lead to disagreements between the 12 workers. So, we consider worker agreement in deciding the quality of an annotation set. Annotation sets where each of the 13 features score ≥ 3.0 from a majority of annotators are considered to be of high quality.
We keep the stories in the train, dev and test splits mutually exclusive and use the actual story annotation set quality to decide the splits. This is because goal descriptions were obtained from the actual stories and may not necessarily be a good fit for an alternative story. Not have a single high quality annotation set out of 12 possible ones (3 sets per single-participant story from 4 actual and alternate stories) implies that the actual story is either complex or under-specified, leading to a difference in semantic understanding among the workers and low agreement on 1 or more features. Multiple participants in a story could also lead to low agreement, however, this was not the case; of the 14
stories and the 42 goal annotation sets that belong in this group all of them were either under-specified or had nuanced semantic meaning. The stories with at least one high quality story were randomly split in a 2:1 ration to create the test and validation splits respectively with all other annotated stories (including the ones identified as lower quality based on the agreement) making up the training split.
Our quality assessment for assigning stories to data splits is very stringent requiring all 13 features to have a high agreement from a majority of evaluators. However, we relax this requirement and heuristically apply one more filter to collect test and dev sets that are of good quality for both actual and alternative stories. We use features from story types for this process; since we do not collect goal descriptions for an alternative story, we use the goal features from the actual story. We allow 1 or 2 features out of the 13 to have a lower agreement when the overall average Likert score of all 13 features is ≥ 3.5. With this we discard any substandard evaluations that assign a Likert score of 3 for all annotations (approx. 7%) but allow for some disagreement from nuanced stories (approx. 8%). We examined the nuanced stories and found that they belong to one of two types making it difficult to identify goal annotations: 1) the story consists of several participants where the selected participant has minimal actions. 2) the story consists of a single participant but latter story events substantially distract the participant from their original intended goal. Overall, despite allowing some lower quality features, we employed a very strict requirement to ensure a high quality evaluation set.

heading: A.6 Task Setup
text: Zero-shot vs. Few-shot We use prompts similar to the RTE and WSC templates from the Flan-T5 templates collection (Wei et al., 2022) which we list in Table 25 with examples. We prompted models in a zero-shot and a few-shot setting with varying number of examples. While Flan-T5 model generations do not change between the two settings, the GPT-3.5 Turbo and GPT-4 models generate multi-sentence goals and next actions repeating story information along with an explanation in the zero-shot setting. We were unsuccessful at shortening the generations from these GPT models in zero-shot setting even after trying a number of variations to the prompt including asking for a ‘concise’, ‘short’ or ‘brief’ generation. In the fewshot setting, GPT models performance improves slightly with more number of examples however the input token limit of 1K tokens for Flan-T5 and 512 tokens T5 performance places a hard limit on the number of examples. For an even comparison across all models we use a few-shot prompt setting with 3 examples for generative tasks and compared both settings for the NLI-type inferences. We used 3 different random seeds and averaged the results (these differ by < .01%).
Fine-tuning We found that fine-tuning on both actual and alternative annotations leads to better performance than fine-tuning on just the actual stories. Additionally, 3-shot prompting a finetuned model improves performance significantly for some tasks (see task1 in §6). In our early development experiments, we used 5-fold cross validation training for 5 to 10 epochs. Noticing that evaluation loss plateaus in 3 to 5 epochs depending on the model type, we stop training after 5 epochs, saving a checkpoint at each epoch. We train all the training data and evaluate with both the validation and test splits using the checkpoint at the 3rd epoch for generative tasks and goal applicability and epoch 4 for the other NLI type tasks. Results reported are an average of 3 model runs with different initial random seeds of 4, 7 and 11 (the variance in results across the various scores is < .01%).
Infrastructure We trained our models we used both RTX 8000 with 48GB of GPU memory and Nvidia A100 with 80GB GPU memory. Approximate run time for a model is a less than 30 minutes.
Hyperparameters For all experiments we used AdamW (Loshchilov and Hutter, 2017) optimizer, a
learning rate of 10−4, a weight decay of 10−4 and 3 different random seeds of 4, 7 and 11. We applied manual tuning and tried various learning rates from .001 to .00001 as suggested for T5 models. For the generation we used Top-K sampling with a beam size of 2 or a 3-shot prompt setting. These parameters worked well for all the models and were selected based on the initial tests across all tasks.

heading: A.6.1 Additional Metrics and Models
text: We present additional automated metrics (Rouge1, Rouge2, RougeL, BERTScore, Corpus and Google’s version of Sentence BLEU in this section with additional models). We note that given the wide variety of possible wordings, BLEU is not well suited for this type of generation. While this is a known issue with generation involving deeper natural language understanding, these results highlight the shortcomings, and provide strong evidence for future work to continue examining how to effectively automatically evaluate generated natural language.
We present results for GPT-3.5 Turbo, GPT-4, T5-11b and various Flan-T5 model sizes (base, large, XL and XXL), using both 0-shot and 3-shot prompt settings. Additionally, we present results for T5-base, Flan-T5 base, large and XL models
fine-tuned on the SAGA dataset. These results are as follows:
1. In Table 12 we present additional results for task 1 (human evaluation of model generated goals) for the dev split. As noted in the main paper, larger models perform well on this data split. See §6.1 in the main paper for a detailed discussion of models performance.
2. In Table 13 we list the macro F1 scores and F1 scores for both the positive and negative labels along with a weighted F1 for identifying goal applicability. Most models do well on identifying positive labels in the full agreement setting, but struggle with the negative labels. We discuss the F1 results for the negative labels in the main paper in §6.2. In the partial agreement setting, larger models perform similarly for both label settings.
3. In Table 14 we present results from the human evaluation of model generated next actions for the dev split. As noted in the main paper, larger models perform well on this data split. Flan-T5-XXL generations contain similar issues as previously discussed with the test data. See §6.3 for a detailed discussion on these issues.
4. In Table 15 we present macro F1 and the F1 scores for the individual labels along with the weighted F1 scores for the transferability of Next Actions and Explanations from the actual stories to alternative stories. We note most models except GPT-3.5 Turbo and fine-tuned Flan-T5-XL are unable to identify the ‘Unsure’ labels for the Next Action transferability inference. All models are better at identifying the positive labels than negative labels for Explanation transferabilty inference. See §6.3 in the main paper for a detailed discussion.
5. In Table 16, 17 and 18 we present macro F1 and the F1 scores for the individual labels along with the weighted F1 scores for identifying achievement in the story, achievement after the story with the next action and participant’s satisfaction towards goal achievement. We present results for both 0-shot and 3-shot prompting and show that it leads to improvement in most models. A few exceptions are where we see a decrease of 1-2% performance with 3-shot prompting in the larger models for identifying achievement after the story and identifying participant’s satisfaction. See §6.5 in the main paper for a detailed discussion.
6. In Table 19, 21 and 23 we report the automated metrics for model generations of goal descriptions (Task 1), next actions (Task 3a) and future plans (Task 4).
7. In Table 20, 22 and 24 we list a few example generations and identify some of the issues. T5-11b generations are only listed in Table 20 to show the additional generated text along with the goal description. We do not list T511b generations for the next actions and future plans for better utilization of space.
8. In Table 25 we list the prompts used in our tasks.

heading: Future Work
text: While this is a known issue with generation involving deeper natural language understanding, these results highlight the shortcomings, and provide strong evidence for future work to continue examining how to effectively automatically evaluate generated natural language. We present results for GPT-3. 5 Turbo, GPT-4, T5-11b and various Flan-T5 model sizes (base, large, XL and XXL), using both 0-shot and 3-shot prompt settings. Additionally, we present results for T5-base, Flan-T5 base, large and XL models
fine-tuned on the SAGA dataset. These results are as follows:
1. In Table 12 we present additional results for task 1 (human evaluation of model generated goals) for the dev split. As noted in the main paper, larger models perform well on this data split. See §6