Venue: NAACL 2019

# Asking the Right Question:

Inferring Advice-Seeking Intentions from Personal Narratives

 Liye Fu

Cornell University

liye@cs.cornell.edu &Jonathan P. Chang

Cristian Danescu-Niculescu-Mizill

Cornell University

jpc362@cornell.edu cristian@cs.cornell.edu

###### Abstract

People often share personal narratives in order to seek advice from others. To properly infer the narrator's intention, one needs to apply a certain degree of common sense and social intuition. To test the capabilities of NLP systems to recover such intuition, we introduce the new task of inferring what is the advice-seeking goal behind a personal narrative. We formulate this as a cloze test, where the goal is to identify which of two advice-seeking questions was removed from a given narrative.

The main challenge in constructing this task is finding pairs of semantically plausible advice-seeking questions for given narratives. To address this challenge, we devise a method that exploits commonalities in experiences people share online to automatically extract pairs of questions that are appropriate candidates for the cloze task. This results in a dataset of over 20,000 personal narratives, each matched with a pair of related advice-seeking questions: one actually intended by the narrator, and the other one not. The dataset covers a very broad array of human experiences, from dating, to career options, to stolen iPads. We use human annotation to determine the degree to which the task relies on common sense and social intuition in addition to a semantic understanding of the narrative. By introducing several baselines for this new task we demonstrate its feasibility and identify avenues for better modeling the intention of the narrator.

+
Footnote †: _Proceedings of NAACL-HLT 2019_, pages 528–541 Minneapolis, Minnesota, June 2 - June 7, 2019. ©2019 Association for Computational Linguistics

## 1 Introduction

"Computers are useless. They can only give you answers." - Pablo Picasso

People often share their personal experiences to elicit advice from others. These personal narratives provide the necessary context for properly understanding the informational goals of the narrators. Endowing automated systems with the capability to infer these advice-seeking intentions could support personalized assistance and more empathetic human-computer interaction.

As humans, to properly distill the narrator's intention from the events and situations they describe, we need to apply a certain degree of social intuition Conzelmann (2012); Conzelmann et al. (2013); Baumgarten et al. (2015); Kehler and Rohde (2017). As an example, consider the goals of a narrator sharing the personal story in Figure 1. We are presented with a wealth of information about the narrator's general sleep patterns, about a particular sleep deprivation situation and its physiological effects, about an upcoming lesson, about coffee intake, its effects, and potential health impacts, and about the current location of the narrator and its impact on food supply. Taking these facts separately, we can imagine providing advice on how to get more sleep, on whether to postpone the lesson,

Figure 1: An abbreviated instance from the ASQ dataset. A personal narrative is matched with two plausible advice-seeking questions, only one of which was actually asked by the narrator when sharing the story.



on how to get food delivered, or on the risks of caffeine intake. However, given how the narrative is constructed, we can intuit that the more likely goal of the narrator is to get advice on how to overcome the effects of sleep deprivation so that they can be alert for the upcoming programming lesson.

Importantly, the primary goal of our proposed task is not to understand details about the narrator's actions in the story ("Why is the narrator tried?", "When do they need to go to the lesson?"), but to infer the reason why the narrator is sharing this story (i.e., "To get advice on how to stay alert in the next few hours."). That is, we are not concerned with the _intradigeetic_ aspects of the narrative, but with the _extradiegetic_ intention of the narrator in sharing the story. While an understanding of the former is likely necessary for the latter, it is often not sufficient.

In this work, we introduce a task and a large dataset to evaluate the capabilities of automated systems to infer the narrator's (extradiegetic) intention in constructing and sharing an advice-seeking personal story. This complements existing narrative understanding tasks which focus on testing semantic understanding of events, actors and their (intradiegetic) intentions within the narrative itself. Table 1 contrasts the goals of these existing narrative understanding tasks with that of inferring a narrator's advice-seeking intention, in the context of our introductory example.

Formally, we implement the task as a binary choice cloze test, where the goal is to identify which of two candidate advice-seeking questions was actually asked by the narrator of a given personal narrative. Beyond collecting a large and diverse set of realistic personal stories that contain an advice-seeking question, the main challenge in constructing this task is finding a plausible alternative advice-seeking question for each given narrative. To address this challenge, we develop a methodology for identifying such questions by exploiting both the commonalities in experiences people share online and the diversity of possible advice-seeking intentions that can be tied to similar experiences.

By applying our methodology to a large collection of online personal narratives, we construct a dataset of over 20,000 cloze test instances, covering a very broad spectrum of realistic advice-seeking situations.1 Each instance contains a narrative that is matched with two advice-seeking questions, one of which is actually asked by the narrator (Q2 in our introductory example), and the other semantically related to the narrative (Q1).

Footnote 1: The dataset is available at [https://github.com/CornellNLP/ASQ](https://github.com/CornellNLP/ASQ).

We use human annotations to judge the relative difficulty of different subsets of the test instances and the type of reasoning necessary to solve them. We find that more than half of the instances contain pairs of questions that are not only semantically related to the narratives but also do not contain any explicit factual mismatches with the stories. These are thus unsolvable by pure logical reasoning and require some degree of common sense or social intuition. And indeed, simple

\begin{table}
\begin{tabular}{l l c} \hline \hline  & **Task** & **Desired output** \\ \hline A & Question generation & What do I need to do in 2 hours? \\  & Reading comprehension & \\  & Summarization & I must go for a lesson after getting little sleep. \\ \hline B & Ending generation & Lastly, I tried an energizing drink. \\  & Narrative chains, story cloze & \\ \hline C & Event2Mind & to learn to code, to be educated \\  & Desire fulfillment & \\ \hline D & **Our task** & How can I energize myself? \\ \hline \hline \end{tabular}
\end{table}
Table 1: Contrast with desired outputs in related narrative understanding tasks that focus on the within-story (intradiegetic) aspects of narrative understanding. Tasks are grouped according to the categories discussed in Section 2.1 (which also includes corresponding references). We assumed the second sentence (“My eyes hurt and two hours later I have programming lesson so I have to be alert.”) to be the answer span for question generation, and the input for Event2Mind (which operates at sentence level).

 

baseline approaches perform worse on these types of instances, highlighting the need for more direct modeling of the intention of the narrator.

To summarize, in this work we:

* formulate the task of inferring advice-seeking intents from personal narratives (Section 2);
* develop a methodology to construct a large dataset of personal narratives matched with plausible options for advice-seeking questions (the ASQ dataset) to be used for this task (Section 3);
* show the task is viable and evaluate the relative difficulty of its items (Sections 4 & 5).

We end by discussing the practical implications of endowing systems with the capability to infer advice-seeking intentions and use our results to identify avenues for developing better models.

## 2 Task formulation

To evaluate the capability of automated systems to infer advice-seeking intentions, we formulate a cloze-style binary choice test where the system is presented with a personal narrative and is required to choose between two plausible candidate questions: one actually asked by the narrator and the other one not (as exemplified in Figure 1).

We motivate the task by contrasting it with other (narrative) understanding tasks (Section 2.1), and provide the rationale for this particular formulation by discussing its advantages (Section 2.2).

### Related narrative understanding tasks

There are many tasks involving reading comprehension in general, and story understanding in particular. Given a narrative, there are a few broad categories of questions that may be asked to test different types and degrees of understanding. Table 1 follows directly from the discussion below, by contrasting the goal of our task with those of (intradigeetic) narrative understanding tasks in the context of our introductory example.

**A: What happened in the story?** The most direct approach to test story understanding is to check whether the reader could comprehend the events and actions that occur within the story. This requires semantic understanding, but nothing more. This type of task can be set up in various forms, as the system can be asked to summarize the story (summarization, see Nenkova (2011); Allahyari et al. (2017) for surveys), generate a question that is answerable from the text (question generation Du et al. (2017)), or answer a question for which the information can be retrieved or reasoned directly from the story (reading comprehension, see Chen (2018) for a survey; notable datasets include MCTest Richardson (2013) and NarrativeQA Kocisky et al. (2018)).

**B: What might happen next?** While reading the story, people not only grasp and process the events that already occurred but also have some intuition of its likely trajectory. Related tasks include the narrative cloze task Chambers and Jurafsky (2008), the story cloze test Mostafazadeh et al. (2016); Chaturvedi et al. (2016), and its generative versions Guan et al. (2019). These tasks might require some common sense reasoning on top of semantic understanding; the fact that they aim to predict the future might require a deeper level of understanding than the previous tasks.

**C: What can we infer about the characters?** When people read a narrative, they not only grasp the facts explicitly stated in the story, but also make inferences about the actors' mental states, such as their attitudes and desires, as the story unfolds.2 Oftentimes, such an understanding requires inference, either logical or based on common sense reasoning. Such tasks can aim to generate the likely intents and reactions from the actors involved in the events Rashkin et al. (2018, 2018), or to determine whether a given desire of the protagonist was fulfilled Rahimtoroghi et al. (2017).

Footnote 2: See Bratman (1987) for an account of the Belief–Desire-Intention model of human practical reasoning.

**D: What is the intention of the narrator in sharing their story?** While these prior tasks cover a wide range of angles to narrative understanding, they take an intradiegetic view by focusing on understanding the story itself. We propose another dimension to this line of work by taking an outside-the-story (extradiegetic) perspective3 and aiming to understand _why_ the story is shared by the narrator, potentially inferred from _how_ the narrator decides to construct it. In particular, the task introduced here is to infer the _advice-seeking_ intention of the narrator.4 

We argue that solving this task requires not only the semantic understanding and common sense reasoning involved in prior tasks but also a certain degree of social intuition. To uncover the goals of the narrator, one needs to find cues in the narrative construction--what has been selectively included or emphasized, and what might have been purposefully omitted Labov (1972). In fact, such intention-understanding tasks are often included in "social intelligence" tests Conzelmann et al. (2013); Baumgarten et al. (2015).

### Advantages of cloze test formulation

To evaluate the capacity of NLP systems to solve this task, we consider a binary choice cloze test formulation for two main reasons. First, it allows natural ground-truth labels: often, when people share their personal experiences to seek advice, they add explicit requests for the information they are seeking. After removing these requests from the narratives, we can use them as proxies for the narrators' intentions. Second, the binary choice operationalization also has the advantage of non-ambiguity in evaluations and ease of comparisons between systems (as opposed to a generation task).

It is worth noting that our dataset is constructed in a way that allows easy modifications into other task formats if so desired. For instance, the methodology of identifying a plausible false choice for a given narrative could be applied multiple times to extend the task to a more difficult multiple-choice version. Similarly, by ignoring the incorrect question in each instance, our dataset can be used as a source for a new generation task, i.e., generating the advice-seeking question from the given narrative.

## 3 Task implementation

For a meaningful implementation of the proposed task, the collection of test instances must conform to several expectations, in terms of both the narratives and their (actual) advice-seeking questions. In what follows we outline these desiderata and our method for collecting instances that meet them (Section 3.1).

Furthermore, as with any multiple-choice cloze test formulation, the difficulty of each test instance largely depends on how plausible the alternative answers are. Yet, finding plausible (but not actually correct) alternatives automatically is challenging. Not surprisingly, many of the cloze-style multiple-choice datasets use humans to write these alternatives Mostafazadeh et al. (2016); Xie et al. (2018), limiting their scalability.

We tackle this challenge by developing a methodology that exploits both the commonalities in human experiences shared online and the diversity in the types of advice needed for similar situations under different circumstances (Section 3.2).

### Collection of candidate instances

Narratives desiderata.As a pre-requisite, we need to start from personal narratives containing advice-seeking needs that are explicitly expressed (as questions), and that can be removed to form the cloze test instances.5 Ideally, these narratives would cover a broad range of topics, in order to be able to test how well a system can generalize to a diverse range of real-life scenarios, rather than apply only to restricted and artificial settings.

Footnote 5: An interesting future work avenue could be considering narratives that only have implicit advice-seeking intentions.

**Question desiderata.Not all questions contained within an advice-seeking narrative are suitable for our task. Some of the questions might be too general, while others might be rhetorical. For instance, _Any advice?_ holds no particular connection with the context of the narrative in which it appears. To contribute to meaningful test instances, questions need to meet a level of relevance and specificity such that (at least) humans could match them with the narratives from which they are extracted.

Data source.We start from a dataset of over 415,000 advice-seeking posts collected from the subreddit r/Advice, which self-defines as "a place where anyone can seek advice on any subject".6 We only use publicly available data and will honor the authors' rights to remove their posts.

Footnote 6: We start from an existing collection of Reddit posts Tan and Lee (2015) which we supplement with The Baumgartner Reddit Corpus retrieved via Pushshift API on Nov. 21, 2018.

**Applying cloze.For each post, we strip off all questions that appear in any position of the post, including the post title.7**

Footnote 7: To identify questions, we use the simple heuristic of looking for sentences that end with “?” or start with _why, hou; am, is, are, do, does, did, can, could, should, would_.

We keep the remaining narratives as the cloze texts.8 Figure 2 shows how the cloze transform 

tion is applied to the post containing our introductory example.

**Selecting ground-truth test answers.9** We select candidate ground-truth answers for the cloze test as the ?-ending sentences removed from narratives. In order to keep only well-formed information-seeking questions, we filter the candidate questions by keeping only those that start with interrogatives10 or _any, anyone, help, advice, thoughts_. To further discard questions that are too general, we compute a simple specificity score \(S(q)\) of a question \(q\) containing the set of words \(\{w_{1},w_{2},\ldots,w_{N}\}\) as its maximum inverse document frequency (idf):

Footnote 9: As it happens, test answers are actually questions.

\[S(q)=S(\{w_{1},w_{2},\ldots,w_{N}\})=\max_{i\in N}\text{idf}(w_{i}),\]

and filter out questions for which \(S(q)<5\) or questions that have less than 5 words. At the end of this selection process, from the example post in Figure 2, _Help?_ and _What has worked for you?_ are discarded and the title question is kept as the ground-truth answer to this cloze instance. If multiple questions survive the filtering process, we select one at random.

**Diversity evaluation.** To verify that the resulting data has broad topical diversity in both narratives and questions, we perform a two-step clustering analysis. First, we use singular value decomposition on tf-idf transformed narratives to obtain their vector representations, we then cluster similar narratives using k-means to surface underlying topics. Next, for each topic, we extract nouns and verbs from the questions attached to each narrative in the topic, and surface common question keywords as those with high document frequency within the topic, correcting for their global document frequency (via subtraction).

To provide a qualitative feel of the diversity of the data, Table 2 shows a selection of the resulting narrative topics and question keywords, together with example questions (corresponding narratives can be found in the data release). We find a

Figure 2: Cloze application to the post from which we obtain the introductory test instance. After filtering out questions that are too general, only the title question remains as a candidate for representing the actual advice-seeking intention of the narrator.

 

wide range of experiences represented in the narratives, from relationships to student life to apartment rentals. Furthermore, within each narrative topic, there is a variety of question types; for instance, questions related to housing could be about dealing with roommates, paying rent, or choosing a city to live in.

### Finding alternative test answers

To find plausible alternative answer options for each candidate cloze test instance, one direct approach could be to find questions that are semantically related to the ground-truth question. However, there are two underlying problems with this approach. First, the task of finding semantically similar questions is itself very challenging (Haponchyk et al., 2018), given their terseness and lack of context. Second, semantic similarity is arguably a different concept from _plausibility_ with respect to a narrative. For example, the two questions in the introductory example are semantically distant, but they are both plausible in the context of the narrative.

Our main intuition in solving this problem is that individuals who are in similar situations tend to have advice-seeking intentions that are related. For each candidate cloze test narrative instance, we can thus search for a similar narrative first (by exploiting commonalities in experiences people share online) and then select an advice-seeking question from that narrative as the _alternative_ answer for the test.

**Narrative pairing.** To operationalize this intuition, we first find pairs of similar narratives based on the cosine similarity of their tf-idf representations.11 A greedy search based on this similarity metric results in a set of pairs of related narratives (\({\text{N}}_{1}\), \({\text{N}}_{2}\)) with their respective advice-seeking questions (qn\({}_{1}\), qn\({}_{2}\)) identified in the previous step.

Footnote 11: We consider both unigrams and bigrams, and set a minimum document frequency of 50. We also remove likely duplicates (cosine \(>0.8\)) and cases for which the similarity between narratives is too low (cosine \(<0.1\)). We have also experimented with embedding-based representations to compute cosine similarities from, but they do not seem to produce qualitatively better pairings upon inspection.

**Narrative masking.** At this point, the pair of advice-seeking questions could be used with either narrative to form a test instance. For example, Figure 3 shows the other possible cloze instance corresponding to the introductory example if we were to use the other narrative in the narrative pair. This, however, would arguably be a poor test instance since Q2 is hardly applicable to this other narrative. More generally, we want to ensure that our choice of which narrative (\({\text{N}}_{i}\)) to include in the cloze test optimizes the plausibility of the question pair (qn\({}_{1}\), qn\({}_{2}\)).

To achieve this, we compute the similarity between each narrative in the pair and each of the two respective questions,12 and select the narrative that maximizes the minimum question-narrative similarity. Formally,

Footnote 12: To account for the terseness of the questions, we represent both narratives and questions with tf-idf weighted GloVe embeddings (Pennington et al., 2014) and compute the cosine similarity between them.

\[N_{i}=\operatorname*{arg\,max}_{i}\textsc{min}\{sim(N_{i},qn_{1}),sim(N_{i},q n_{2})\}.\]

Importantly, this selection criterion is purposely symmetric with respect to the two questions in order to avoid introducing any unnatural preference between the two that a classifier (with no access to the masked narrative) could exploit.

As a final check, we ensure that in each cloze instance the two questions are neither too similar to each other (and thus indistinguishable) nor too dissimilar (which may indicate unsatisfactory narrative pairings). To this end, we discard instances in which the questions have extremely high or low surface similarity according to their InferSent (Conneau et al., 2017) sentence embeddings.13

Footnote 13: We set a lower bound of 0.8 and an upper bound of 0.95. We choose this representation because questions are short and thus we anticipate tf-idf representation to be less informative.

This process leaves us with a total of 21,865 instances. A detailed account of the number of instances filtered at different stages of the construction process can be found in the Appendix.

Figure 3: Alternative cloze test instance corresponding to the introductory example.

 

Human performance

To understand the feasibility of the task, as well as the relative difficulty of the items in the dataset, eight non-author annotators labeled a random sample of 200 instances.14 Each annotator is asked to choose first, out of the two candidate questions, which they consider to be _more likely_ to have been asked by the narrator. Overall, human annotators achieve an accuracy of 90% (Cohen's \(\kappa\) = 0.79),15 showing that humans can indeed recover the advice-seeking intentions of the narrators, and thus validating the feasibility of the task.16

Footnote 14: See the Appendix for detailed annotation instructions.

Footnote 15: We obtained a second round of annotations on a subset of 75 task instances to compute agreement statistics.

Footnote 16: By construction, random accuracy is 50%.

We are also interested in understanding the types of skills needed to solve the task. In particular, we want to estimate the proportion of the task instances that can not be solved by mere factual reasoning. To this end, we ask humans to identify candidate questions that contain a factual mismatch with the narrative, making them Explicitly incompatible; 57% of the annotated instances do not contain any such mismatches in any of the questions. Similarly, we want to estimate how many instances require common sense expectations about the behavior of the protagonist (within the story). So we ask annotators to mark questions as being Implicitly incompatible if they do not contain any factual mismatches, but they are incompatible with what can be inferred implicitly about events and characters in the story.

The questions that are neither explicitly nor implicitly incompatible would be labeled as being Compatible, and as either Likely or Unlikely to represent the narrators' intentions. Test items in our data forcing a choice between Compatible questions are expected to be the hardest to solve, as they might require a certain degree of social intuition in addition to factual and common sense reasoning. Table 3 provides an example narrative and one representative question from each of the above-mentioned categories.17

Footnote 17: The example is adapted from our instructions to annotators, which includes further explanations for these categories. See the Appendix for details.

Table 4 shows a human performance breakdown according to some of the most common types of instances in our data.18 As expected, instances involving only compatible questions (C + C) are harder to solve,19 as they might require some social intuition, whereas when explicit contradictions exist (C + E), they are perfectly solvable. We also note that humans can perfectly solve the subset of task instances (L + {U, I}) that exhibit perceived qualitative differences between the actual and the alternative questions, but nevertheless, require more than semantic understanding (and sometimes require social intuition).

[FOOTNOTE:18][ENDFOO

\begin{table}
\begin{tabular}{l l c c c} \hline \hline
**Pair type** & **sim** & **ft-LM** & **human** & **\% in data** \\ \hline C + E & 86\% & 88\% & 100\% & 38\% \\ C + \{C,I\} & 68\% & 74\% & 89\% & 46\% \\ C + C & 66\% & 73\% & 84\% & 32\% \\ L + \{U,I\} & 75\% & 75\% & 100\% & 30\% \\ overall & 76\% & 80\% & 90\% & \\ \hline \hline \end{tabular}
\end{table}
Table 4: Breakdown of performances on selected actual + alternative question pair types. For instance, the pair type C + E corresponds to instances where the actual question asked by the narrator is compatible and the alternative question is explicitly incompatible.

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline
**Narrative**: I asked a girl that I really like if she would like to get coffee sometime. She said she’s really busy but that we’ll see. I can’t get her off my mind and I spend all day waiting for her to tell me she’s free.

**Explicitly incompatible (E)**: How to deal with my roommate?

**Implicitly incompatible (I)**: What to do if I asked a girl out and now regret it?

**Compatible (C) but unlikely (U)**: Which coffee place would you recommend?

**Compatible (C) and likely (L)**: Would it seem desperate if I asked her again in a more direct way a week later?

\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Pair type** & **sim** & **ft-LM** & **human** & **\% in data** \\ \hline C + E & 86\% & 88\% & 100\% & 38\% \\ C + \{C,I\} & 68\% & 74\% & 89\% & 46\% \\ C + C & 66\% & 73\% & 84\% & 32\% \\ L + \{U,I\} & 75\% & 75\% & 100\% & 30\% \\ overall & 76\% & 80\% & 90\% & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Example questions in each plausibility category for an example narrative.

 



## 5 Baseline systems performance

We divide our data into a 8,865-2,500 train-test split and have reserved 10,000 instances as a held-out set.20 In Table 5 we report accuracy for the best-performing model on the (never-before-seen) held-out for a simple similarity-based method and for a deep learning method.

Footnote 20: The set annotated by humans is disjoint.

**Narrative-question similarity.** We expect that questions would show greater similarity to narratives they are removed from. We thus establish a narrative-question similarity baseline by considering features based on cosine similarities between narrative and questions, with text represented as tf-idf vectors, tf-idf weighted GloVe embeddings, averaged GloVe embeddings, as well as word overlap between content words, all combined in a logistic regression model.

**Finetuned transformer LM.** We also use a Fine-tuned Transformer LM model Radford et al. (2018), which was shown to perform competitively on a diverse set of NLP tasks, achieving state-of-the-art results on the story cloze test.21

Footnote 21: We fine-tune with our training set on top of the pretrained transformer language model, using the implementation from [https://github.com/huggingface/pytorch-openai-transformer-lm](https://github.com/huggingface/pytorch-openai-transformer-lm).

### Error analysis

**Required skills.** As shown in Table 4, systems perform worst on items that do not exhibit any (implicit or explicit) mismatches (C + C), and thus might require some social intuition. Importantly, the largest gap between baseline and human performance (25%) is on the subset of items that can not be solved based solely on a semantic understanding (L + {U, I}). These results underline the need for models that can combine common sense reasoning about the events within the story with an intuition about the intention of the narrator.

**Question concreteness.** Questions may also differ in how concrete they are. In a preliminary analysis aimed at understanding how this property affects performance, we compare words used in ground-truth questions that the best-performing model predicts correctly with those used in questions that are classified incorrectly. We observe that questions that are predicted correctly have significantly higher average inverse document frequencies (t-test \(p<0.01\)). Intuitively, these more specific questions may be more concrete in nature, making them easier to connect to the narratives to which they belong. We also find that some common interrogatives have skewed distributions. For instance, questions starting with _Is_ are less likely to be classified correctly than those starting with _How_. A cursory manual investigation suggests that this can also be tied by concreteness, with the latter type of questions appearing to be more concrete than the former.

## 6 Further related work

One broad motivation behind our work is to eventually help better support personalized informational needs Teevan et al. (2007). This connects to several related lines of work that were not previously discussed.

**Query/question intents.** Datasets and models are proposed for understanding user intents behind search queries Radlinski et al. (2010); Fariha et al. (2018), or even more generally, user questions Haponchyk et al. (2018). To complement this line of work that looks at user intents behind the explicit request, our task aims to uncover user intents when they are implied in personal narratives (without access to the explicit question).

**Conversational search/QA.** One way to better satisfy user intents is by making such processes collaborative Morris and Horvitz (2007); Morris (2013), or conversational Radlinski and Craswell (2017). Conversational QA datasets Choi et al. (2018); Reddy et al. (2019) have been introduced to help develop systems with such capability.

**Social QA.** Some questions posed by users are inherently more social in nature, and require more nuanced contextual understanding Harabagiu (2008). The social nature may affect how people ask questions Dahiya and Talukdar (2016); Rao and Daume III (2018), and pose challenges for identifying appropriate answers Shtok et al. (2012); Zhang et al. (2017).

\begin{table}
\begin{tabular}{l c} \hline \hline
**Model** & **Accuracy (held-out)** \\ \hline narrative-qn-sim & 73.4\% \\ finetuned lm & **78.7\%** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance of different baselines.





## 7 Discussion

In this work, we introduce the new task of inferring advice-seeking intentions from personal narratives, a methodology for creating appropriate test instances for this task and the ASQ dataset. This task complements existing (intradiegetic) narrative understanding tasks by focusing on extradiegetic aspects of the narrative: in order to understand "Why is the narrator sharing this?", we often need to apply a certain degree of common sense and social intuition.

From a practical perspective, this extradiegetic capability is a prerequisite to properly address personalized information needs that are constrained by personal circumstances described as free-form personal stories. Currently, to address these types of information needs, people seek (or even hire) other individuals with relevant experience or expertise. As with conversational search Radlinski and Craswell (2017), we can envision systems that can more directly address complex information needs by better understanding the circumstances and intentions of the user.

Our analysis of the human and baseline performance on different types of test instances points to interesting avenues for future work, both in terms of designing better-performing systems and in terms of constructing better test data. We envision that (intradiegetic) narrative understanding could help identify the components of the narrative that are most relevant to the advice-seeking goal. For example, identifying the narrator's intentions and desires within the story Rashkin et al. (2018b), and whether these desires are fulfilled Rahimtoroghi et al. (2017) could help focus the attention of the model, especially when dealing with less concrete questions. Furthermore, a better representation of the structure of the narrative Ouyang and McKeown (2014), in terms of discourse acts Elson (2012) and sentiment flow Ouyang and McKeown (2015), could also help distinguish between spurious and essential circumstances of the narratives.

In terms of improving the task itself and the methodology for creating testing instances that better approximate the inferential task, we note a few possible directions. Firstly, better narrative modeling could lead to higher quality matching. Similarly, better representation of the questions can help select more appropriate candidate options (e.g., currently 6% of the questions are deemed by the annotators to be too general). In addition, the generative version of the task, when appropriately evaluated, could be a closer operationalization for intention inference, and also offer more potential for practical uses.

Finally, future work could expand on our methodology to formulate other more general tasks aiming to understand the reasons why a person is sharing a personal story. While we have focused on narratives shared with the intention of seeking advice, people may also share stories to express emotions, to entertain or educate others. A better understanding of these different (explicit or implicit) intentions could lead to more personalized and empathetic human-computer interaction.

**Acknowledgments.** The authors thank Tom Davidson, Tsung-Yu Hou, Qian Huang, Hajin Lim, Laure Thompson, Andrew Wang, Xiaozhi Wang and Justine Zhang for helping with the annotations. We are grateful to Thorsten Joachims, Avery Quinn Smith and Todd Cullen for helping us when our server crashed on the day of the deadline while testing the model on the held-out set, to Lillian Lee, Andrew Wang, Justine Zhang and the anonymous reviewers for their helpful comments, and to Fernando Pereira for the early discussions that inspired this research direction. This work is supported in part by NSF CAREER award IIS-1750615 and NSF Grant SES-1741441.

## References

* Allahyari et al. (2017) Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saeid Safaei, Elizabeth D. Trippe, Juan B. Gutierrez, and Krys Kochut. 2017. Text Summarization Techniques: A Brief Survey. _arXiv:1707.02268v3_.
* Baumgarten et al. (2015) Melanie Baumgarten, Heinz-Martin Suss, and Susanne Weis. 2015. The Cue Is the Key: The Relevance of Cues and Contextual Information in the Social Understanding Tasks of the Magdeburg Test of Social Intelligence. _European Journal of Psychological Assessment_, 31(1).
* Bratman (1987) Michael Bratman. 1987. _Intention, Plans, and Practical Reason_.
* Chambers and Jurafsky (2008) Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised Learning of Narrative Event Chains. In _Proceedings of ACL_.
* Chaturvedi et al. (2016) Snigdha Chaturvedi, Dan Goldwasser, and Hal Daume III. 2016. Ask, and Shall You Receive? Understanding Desire Fulfillment in Natural Language Text. In _Proceedings of AAAI_

* Chen (2018) Danqi Chen. 2018. _Neural Reading Comprehension and Beyond_. Ph.D. thesis, Stanford University.
* Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percry Liang, and Luke Zettlemoyer. 2018. QuAC: Question Answering in Context. In _Proceedings of EMNLP_.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In _Proceedings of EMNLP_.
* Useful Constructs?_ Ph.D. thesis, The Otto von Guericke University Magdeburg.
* Conzelmann et al. (2013) Kristin Conzelmann, Susanne Weis, and Heinz-Martin Suss. 2013. New Findings About Social Intelligence: Development and Application of the Magdeburg Test of Social Intelligence (MTSI). _Journal of Individual Differences_, 34(3).
* Dahiya and Talukdar (2016) Yogesh Dahiya and Partha Talukdar. 2016. Discovering Response-Eliciting Factors in Social Question Answering: A Reddit Inspired Study. _Proceedings of ICWSM_.
* Du et al. (2017) Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to Ask: Neural Question Generation for Reading Comprehension. In _Proceedings of ACL_.
* Elson (2012) David K. Elson. 2012. _Modeling Narrative Discourse_. Ph.D. thesis, Columbia University.
* Fariha et al. (2018) Anna Fariha, Sheikh M. Sarwar, and Alexandra Mediuou. 2018. SQuID: Semantic Similarity-Aware Query Intent Discovery. In _Proceedings of SIGMOD_.
* Guan et al. (2019) Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story Ending Generation with Incremental Encoding and Commonsense Knowledge. _Proceedings of AAAI_.
* Haponchyk et al. (2018) Iryna Haponchyk, Antonio Uva, Seunghak Yu, Olga Urupina, and Alessandro Moschitti. 2018. Supervised Clustering of Questions into Itents for Dialog System Applications. In _Proceedings of EMNLP_.
* Harabagiu (2008) Sanda M. Harabagiu. 2008. Questions and Intentions. In _Advances in Open Domain Question Answering_.
* Kehler and Rohde (2017) Andrew Kehler and Hannah Rohde. 2017. Evaluating an Expectation-Driven Question-Under-Discussion Model of Discourse Interpretation. _Discourse Processes_, 54(3).
* Kocisky et al. (2018) Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl M. Hermann, Gabor Melis, and Edward Grefenstette. 2018. The NarrativeQA Reading Comprehension Challenge. _TACL_.
* Labov (1972) William Labov. 1972. The Transformation of Experience in Narrative Syntax. _Language in the Inner City_.
* Morris (2013) Meredith R. Morris. 2013. Collaborative Search Revisited. In _Proceedings of CSCW_.
* Morris and Horvitz (2007) Meredith R. Morris and Eric Horvitz. 2007. SearchTogether: An Interface for Collaborative Web Search. In _Proceedings of UIST_.
* Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. In _Proceedings of NAACL_.
* Nankova (2011) Ani Nankova. 2011. Automatic Summarization. _Foundations and Trends in Information Retrieval_, 5(2).
* Ouyang and McKeown (2014) Jessica Ouyang and Kathleen McKeown. 2014. Towards Automatic Detection of Narrative Structure. In _Proceedings the LREC_.
* Ouyang and McKeown (2015) Jessica Ouyang and Kathleen McKeown. 2015. Modeling Reportable Events as Turning Points in Narrative. In _Proceedings of EMNLP_.
* Pennebaker (1997) James W. Pennebaker. 1997. Writing About Emotional Experiences as a Therapeutic Process. _Psychological Science_, 8(3).
* Pennebaker and Seagal (1999) James W. Pennebaker and Janel D. Seagal. 1999. Forming a Story: The Health Benefits of Narrative. _Journal of Clinical Psychology_, 55(10).
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. In _Proceedings of EMNLP_.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-training. _Preprint_.
* Radlinski and Craswell (2017) Filip Radlinski and Nick Craswell. 2017. A Theoretical Framework for Conversational Search. In _Proceedings of CHIIR_.
* Radlinski et al. (2010) Filip Radlinski, Martin Szummer, and Nick Craswell. 2010. Inferring Query Intent from Reformulations and Clicks. In _Proceedings of WWW_.
* Rahimtorogh et al. (2017) Elahe Rahimtorogh, Jiaqi Wu, Ruimin Wang, Pranav Anand, and Marilyn Walker. 2017. Modelling Protagonist Goals and Desires in First-Person Narrative. In _Proceedings of SIGDIAL_.
* Rao and Dahume III (2018) Sudha Rao and Hal Daume III. 2018. Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information. In _Proceedings of ACL_.
* Rashkin et al. (2018a) Hannah Rashkin, Antoine Bosseltut, Maarten Sap, Kevin Knight, and Yejin Choi. 2018a. Modeling Naive Psychology of Characters in Simple Commonsense Stories. _Proceedings of ACL_.
* Rashkin et al. (2018b) Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018b. Event2 Mind: Commonsense Inference on Events, Intents, and Reactions. In _Proceedings of ACL_ 

Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A Conversational Question Answering Challenge. _TACL_.
* Richardson (2013) Matthew Richardson. 2013. MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text. In _Proceedings of EMNLP_.
* Shtok et al. (2012) Anna Shtok, Gideon Dror, Yoelle Maarek, and Idan Szpektor. 2012. Learning from the Past: Answering New Questions with Past Answers. In _Proceedings of WWW_.
* Swanson et al. (2017) Reid W. Swanson, Andrew S. Gordon, Peter Khooshabeh, Kenji Sagae, Richard Huskey, Michael Mangus, Ori Amir, and Rene Weber. 2017. An Empirical Analysis of Subjectivity and Narrative Levels in Weblog Storytelling Across Cultures. _Dialogue & Discourse_, 8(2).
* Tan and Lee (2015) Chenhao Tan and Lillian Lee. 2015. All Who Wander: On the Prevalence and Characteristics of Multi-community Engagement. In _Proceedings of WWW_.
* Tevevan et al. (2007) Jaime Tevevan, Susan T. Dumais, and Eric Horvitz. 2007. Characterizing the Value of Personalizing Search. In _Proceedings of SIGIR_.
* Xie et al. (2018) Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy. 2018. Large-Scale Cloze Test Dataset Created by Teachers. In _Proceedings of EMNLP_.
* Zhang et al. (2017) Wei E. Zhang, Quan Z. Sheng, Jey Han Lau, and Emyras Abebe. 2017. Detecting Duplicate Posts in Programming QA Communities via Latent Semantics and Association Rules. In _Proceedings of WWW_.

## Appendix A1 Instructions to human annotators

As described in Section 4 of the main paper, we obtained human annotations on a small subset of our data for validation purposes. Annotators were shown instructions which included the definitions for different question-narrative plausibility types, together with two examples to help further clarify the task and the definitions. The exact instructions are reproduced in Table A8, while the examples provided are shown in Table A9.

### Distribution of plausibility categories

Table A6 shows the distribution of plausibility categories given out by our human annotators, for both the actual questions which belong to the given narrative (Column 2) and the paired alternative questions (Column 3).

### Further processing details

Table A7 provides the number of instances remaining at each of the processing steps. After masking one narrative from each narrative pair, we have a total of 21,865 narratives, each successfully paired with two plausible candidate questions.

### Examples of pair types

In our error analysis, we find that the performance of both our baselines, as well as that of our human annotators, vary depending on question pair type, where pair type is defined as the human-judged plausibility of the ground-truth question and that of the alternative question. To give a better sense of what these pair types look like in practice, Table A10 shows example instances for a few selected pair types 

You will be presented with one narrative and two advice-seeking questions (qn1 and qn2).

Firstly, you will need to indicate which of these questions is **more likely** to be asked by the narrator in the context of the narrative (Column D, in yellow). Use the dropdown menu to select the more likely question among the two. (You must pick one.)

In addition, for **each** question separately, you will need to provide a rating on how plausible the question is in the context of the narrative by choosing one of the following options (dropdown menu in Columns E and F, in green):

1. **Very general**: i.e., this question could follow most narratives, and it's not in any way specific to this narrative.
2. **Compatible and likely**: i.e., the question follows naturally from this narrative.
3. **Compatible but unlikely**: i.e., while there is no direct contradiction (either explicit or implicit) with the narrative, it seems unlikely that the narrator's intention was to ask this question.
4. **Incompatible (explicit)**: i.e., the question is incompatible due to clear factual mismatches with the information explicitly contained in the narrative, or it is completely irrelevant.
5. **Incompatible (implicit)**: i.e., the question is incompatible due to mismatches with something that you can indirectly infer from the narrative.

You should judge each question **separately** when selecting a category. It is possible for both questions to fall into the same category.

Now you need to read the example narratives, questions and explanations in the adjacent cells (B2 and C2) to get a feel of each of the categories, after which you could proceed to the sub-sheet **Items to annotate (see tab at the bottom of this page)** to complete the annotation task.

Optionally, you can also provide comments for each item (scroll to the right to see the comment column): Did you find an item particularly challenging or interesting? Was one of the questions not really asking for an advice? Do share your thoughts with us.

 



**Example 1**

\begin{tabular}{l l} \hline Narrative: "I am a freshman in college and I have a group of friends that I have been hanging out with for the past couple months. I feel like we have a good time when we hang out, but a lot of the time, the rest of the group will go out and do stuff together, but I won't be included." \\ Example questions in each category [and explanations where appropriate]: & \\ a) **Very general**: & \\ Any advice? & \\ b) **Compatible and likely**: & \\ Can I ask to be included? & \\ c) **Compatible but unlikely**: & \\ How to make new friends in college? & \\ \([\)explanation: it is more likely that the narrator is trying to be more included in the current group of friends, rather than giving up entirely on them and look for replacement.\(]\) & \\ d) **Incompatible (explicit)**: & \\ Any advice for finding new friends for a senior in college? & \\ \([\)Explanation: The narrator is a freshman in college, not a senior. This constitutes a clear factual mismatch between the question and the narrative.\(]\) & \\ e) **Incompatible (implicit)**: & \\ What are some excuses to not hang out with them? & \\ \([\)Explanation: We can imply from the narrative that the narrator wants to hang out with the group. & \\ This is incompatible with a question asking how NOT to do that.\(]\) & \\ \hline \end{tabular}

Table A9: Example narratives and questions shown to the annotators.



* [14] M. C. R. Jones, and J. A. Wheeler, "The physics of the universe: The history of black holes," _Physics Reports_**389**, 2005-2067, doi:10.1016/j.physrep.2005.09.001
* [15] S. D. Ellis, "A new theory for black hole formation in a universe," _Journal of Cosmology_**3**, 1993-2008, doi:10.1088/0954-3899/3/1993.
* [16] G. Farrar and P. N. Phillips, "Black Holes from Black Hole Formation," _Astrophys. J. Suppl._**46**, 199-201, doi:10.1140/S0010-9361(93)90001-X.

[MISSING_PAGE_POST]