Venue: NAACL 2024

# Leveraging Interesting Facts to Enhance User Engagement with Conversational Interfaces

 Nikhita Vedula  Giuseppe Castellucci  Eugene Agichtein

Oleg Rokhlenko   Shervin Malmasi

Amazon.com, Inc.  Seattle, WA, USA

{veduln,giusecas,eugeneag,olegro,malmasi}@amazon.com

###### Abstract

Conversational Task Assistants (CTAs) guide users in performing a multitude of activities, such as making recipes. However, ensuring that interactions remain engaging, interesting, and enjoyable for CTA users is not trivial, especially for time-consuming or challenging tasks. Grounded in psychological theories of _human interest_, we propose to engage users with contextual and interesting statements or _facts_ during interactions with a multi-modal CTA, to reduce fatigue and task abandonment before a task is complete. To operationalize this idea, we train a high-performing classifier (\(82\%\) F1-score) to automatically identify relevant and interesting facts for users. We use it to create an annotated dataset of _task-specific interesting facts1_ for the domain of cooking. Finally, we design and validate a dialogue policy to incorporate the identified relevant and interesting facts into a conversation, to improve user engagement and task completion. Live testing on a leading multi-modal voice assistant shows that \(66\%\) of the presented facts were received positively, leading to a \(40\%\) gain in the user satisfaction rating, and a \(37\%\) increase in conversation length. These findings emphasize that strategically incorporating interesting facts into the CTA experience can promote real-world user participation for guided task interactions.

Footnote 1: www.github.com/wiki18/cta-interesting-facts

## 1 Introduction

Conversational Task Assistants (CTAs) are a class of conversational agents that guide human users step-by-step in performing a multitude of activities, like cooking or home improvement tasks [59, 6]. CTAs help users search and find the right task for their needs, inform about the task execution (e.g., tools to use), and answer questions about the task (e.g., substituting an ingredient).

Besides ensuring that CTAs provide useful, accurate content, it is crucial to encourage users to consistently use CTAs to accomplish long and complex tasks. In this setting, users can easily get bored and abandon tasks, so maintaining interesting and entertaining conversations is key [30, 14]. Understanding and predicting human interest is crucial to engage and retain users [20].

We hypothesize that incorporating relevant, entertaining pieces of information into conversations is a powerful tool to maintain and sustain user attention, satisfaction, and participation. For example, while cooking with a CTA, users are likely to interact with it longer, frequently and repeatedly, if their experience is enriched with intriguing facts about the recipe's history, nutritional benefits of its ingredi

Figure 1: Conversations \(C_{1}\) and \(C_{2}\) between two users and our CTA. \(C_{1}\) shows an interesting fact during search. \(C_{2}\) has two interesting facts during task execution.



dients, or useful tips about its cooking techniques (see Figure 1 for an example). Such information can increase user knowledge, satisfaction and trust in the CTA by establishing it as a knowledgeable domain expert. It transforms users' dialogue with the CTA into more than just task-oriented, resulting in an improved user experience, particularly during long, tedious, or monotonous activities.

We propose a practical approach to enhance user engagement with CTAs. Grounded in socio-psychological theories on human _interest_Berlyne (1949, 1970; Silvia, 2005), we design a novel schema to comprehensively and objectively examine the _interestingness_ of pieces of information (henceforth called _'facts'_) in a CTA setting. We study if such facts can spark and foster user interest, and thus boost user satisfaction and engagement during a conversation with a CTA. We also explore multiple techniques to share interesting facts with users. We perform live testing of our framework in a leading multi-modal voice assistant. \(66\%\) of our facts were received positively by real-world users, resulting in a \(40\%\) gain in average user satisfaction ratings, and a \(37\%\) increase in average conversation length. These results demonstrate the practical benefits of enriching the CTA experience with interesting facts, and showcase our approach's potential to enhance the adoption of CTAs for complex real-world tasks requiring sustained user engagement.

We also create for public release a dataset of \(1,379\) interesting facts associated with cooking, extracted from online sources and cleaned for grammar, style, relevance, interestingness, and appropriateness. We believe that this can be a valuable resource to drive future research in CTAs, and user engagement in task-assistance conversations.

## 2 Related Work

Interest and Interestingness:Our work on enhancing user engagement with CTAs by increasing their level of interest in the conversation is inspired by multiple psychological theories conceptualizing the human behavioral dimensions of interest and interestingness. One of the earliest theories is that of Berlyne (Berlyne, 1949, 1970), who viewed interest as a defining factor of motivation, influenced by several factors including uncertainty, complexity and novelty of the subject at hand. Tomkins (1962); Silvia (2005) defined interest as an appealing effect of an activity on a human caused by novelty, exploration, learning, creativity and affect, leading to increased interaction with the subject matter.

Geng and Hamilton (2006) surveyed multiple features relevant to interestingness such as conciseness, coverage, reliability, novelty, surprise and utility in the context of pattern mining and association rules. Carvalho et al. (2005) studied their correlation with real human interest. Wu and Miao (2013); Constantin et al. (2019) observed a negative correlation between interest and low-arousal emotions like memorability, familiarity, and boredom. Clark et al. (2019); Ceha et al. (2021) employed humor, jokes and fun facts within their conversation agents to demonstrate that providing such entertaining information to users can increase user interest, engagement and retention, and reduce monotony.

In our work, we model several aspects contributing to human interest in Section 3, to identify facts that can interest and engage users interacting with a CTA, and reduce monotony. We show in Section 4 that a CTA without interest enhancing signals can cause boredom and reduce engagement by users.

Conversational Task Assistants:CTAs are a special type of conversational assistants Agichtein et al. (2023), different from general purpose intelligent assistants (e.g., Amazon's Alexa, Apple's Siri). CTAs guide users to perform tasks in a stepwise manner, also answering user questions along the way. CTAs differ from conversational search agents Vtyurina et al. (2017) and task-oriented conversational agents Zhang et al. (2020); Strathearn and Gkatzia (2022), where the agent and not the user performs a task based on user input (e.g., booking a ticket). User satisfaction and engagement has been widely studied in the areas of search, task-oriented dialog systems and virtual assistants Kiseleva et al. (2016); Park et al. (2020); Sekhulic et al. (2021); Lotze et al. (2021); Siro et al. (2022). However, these approaches lack a theoretical backing and have not been studied for CTAs.

The work that bears the most similarity with ours is Wizard of Curiosities (WoC) Vicente et al. (2023), which also adds facts within dialogues between users and CTAs. However, both works differ in the following ways: (i) WoC targets _information-seeking_ traits of humans based on _curiosity_ theory Berlyne (1966); Kidd and Hayden (2015). Our work is more principled and draws upon the broader theories of _interest_ which encompass informativeness and curiosity, and are better suited for studying and improving user engagement Pekrun (2019); 

(ii) WoC does not consider several features that we consider in this work to add facts into the CTA dialogue flow, namely novelty, recency, frequency, interest and emotion (defined in Section 3). (iii) Unlike WoC, we translate the theoretical aspects into a novel annotation schema for interestingness that is more objective, consistent and precise for a CTA (Sections 3.1 and 3.3). (iv) We study several user experience options on how to effectively present users interesting facts during the conversation flow.

## 3 Boosting CTA Engagement with Facts

We now describe our framework to offer interesting facts to users conversing with a multi-modal CTA, having both display and voice components, to entertain users and increase their engagement. We focus on cooking-specific conversations, but as explained in Section 5, we believe that our framework can generalize to any CTA-feasible domain.

### Feature Schema for Interestingness

We first define a feature schema inspired by socio-psychological work on interestingness [24], [197], and use it to curate a dataset of facts that may increase user interest and lower monotony.

**(i) Conciseness:** A fact should not be too long in terms of number of words and listening time.

**(ii) Specificity:** A fact should be understandable in general to a large number of users, without requiring specific background or domain knowledge.

**(iii) Novelty:** A fact should offer new, unique information that may surprise an average user not a knowledge expert in the domain.

**(iv) Relevance:** A fact should be relevant and related to the considered conversation turn.

**(v) Informativeness:** A fact should deliver useful, helpful knowledge or insights about the task.

### Interesting Facts Dataset

We create a dataset of \(1,379\) interesting facts associated with cooking, following the above feature schema. We use an off-the-shelf named entity recognition model to extract a list \({\cal E}\) of entities of the types _ingredient_, _recipe_, and _tool_, from a large corpus of recipes \({\cal R}\).2 We then crawl specific websites3 to extract diverse candidate facts for \({\cal E}\).

Footnote 2: The recipe provider in this work is Whole Foods Recipes.

Footnote 3: Wikipedia, and fun facts websites like www.facts.net

To filter out irrelevant facts, we split each fact into individual sentences. We then fine-tune and use a domain classifier, consisting of a linear layer on top of the 5B Alexa Teacher Model [80], with positive training sentences from \({\cal R}\), and negative instances from out-of-domain online sources.4 We match each of the domain-relevant candidate facts to our list of identified entities \({\cal E}\). We only retain facts if they contain an entity in \({\cal E}\) as either the subject or the object of the fact sentence, based on its dependency parse tree.5 This gives us a set of potential interesting facts \({\cal F}^{\prime}\).

Footnote 4: The model achieves a \(0.91\) F1-score on a 13.8K test set.

Footnote 5: We use www.spacy.io to compute the parse tree.

We next sample a set of \(750\) facts from \({\cal F}^{\prime}\), to be manually annotated for their level of _interestingness_. For each fact, \(2\) expert annotators familiar with this domain give it a binary label with respect to each feature of our schema (Section 3.1), followed by an overall label for interestingness with respect to the entity under consideration. We also ask annotators to choose the feature most important to them to decide if a fact selected by a CTA could be interesting. Having such a structured schema can mitigate the subjectivity and/or biases likely to occur while annotating the inherently subjective element of _interestingness_, giving more objective, consistent, and precise annotations.

We find that annotators prefer novel and easily understood facts. Aggregating their preferences yields this ranked order of feature importance for relevant facts: novelty, specificity, conciseness and informativeness, with a per-feature binary inter-annotator agreement of \(0.68\), \(0.84\), \(0.88\) and \(0.75\) respectively. We assign each feature a normalized weight based on how many times they were chosen as important. We then compute a linearly combined interestingness score for each fact, resulting in a set \({\cal F}\) of \(606\) annotated interesting facts. These facts serve as positive instances, and irrelevant or uninteresting facts from \({\cal F}^{\prime}\) serve as negative instances, to train a multi-label binary classifier on top of a RoBERTa [10] language model. The two labels denote relevance and interestingness to the input text (the current step of the ongoing task).

We use this classifier to automatically label the remaining domain-relevant candidate facts in \({\cal F}^{\prime}\). Each fact is linked to a step in the corpus \(R\), based on the entities present in the task step. To minimize redundancy and ensure that the facts associated with each step are sufficiently diverse, we cluster and filter facts with a high (\(\geq 0.85\)) pairwise cosine similarity based on Sentence-BERT [13] embeddings. Each fact is then 

manually cleaned for grammar, style and appropriateness. We finally obtain a novel dataset of \(1,379\) interesting cooking facts (see dataset details in Table 1), to present to real-time customers conversing and performing a task with a CTA. Each fact has accompanying evidence in the form of the website URL it is sourced from.

```
Data: Facts associated with each task \(max\_facts\gets initialize; min\_turns\_btw\_facts\gets initialize; FunctionshowFactAtStep()://Showuserfactatthisstep? iffectexistsforstepand#factsalready shown<\(max\_facts\)and# turnsetwetw. facts\(\geq min\_turns\_btw\_facts\)and final turnisvoicefriendlythen returnTrue; elsereturnFalse;
```

**Algorithm 1**Presenting interesting facts to a user interacting with a CTA during search.

### Incorporating Interesting Facts into Conversations

The interestingness feature schema described in Section 3.1 only includes _fact-specific_ features, to recognize a given fact as interesting. Recall that a CTA aids users explore or search for tasks, provides information about tasks, and guides users through step-by-step execution instructions. For a satisfying user experience (UX), several _conversational_ dimensions need to be considered to effectively inject interesting facts into user-CTA interactions. We consider alternative dialogue policies:

(i) **Present Facts Before & During a Task**: We offer interesting facts to users both when they are exploring or searching for a task to perform, and during the step-by-step execution of their chosen task. Algorithms 1 and 2 show both these UXs.

(ii) **Select Turns to Present Facts:** Engaging users is crucial, but the goal of a user-CTA interaction is to complete a task (e.g., cook a recipe). We use the below criteria to avoid creating distractions and a poor experience by showing too many facts.

* **Recency:** After presenting a fact, the next one is shown only after a set number of turns.
* **Frequency**: A CTA should not present too many facts per conversation. The recency and frequency parameters are implemented in the showFactAtStep() function in Algorithms 1 and 2, and their values are chosen empirically.
* **Diversity**: Our dataset creation in Section 3.2 ensures that users are shown diverse facts.
* **Voice-friendly**: The word count of all conversation turns with interesting facts is bounded.

(iii) **Proactive Inquiry**: A CTA explicitly asking a user if they want to see an interesting fact or not adds two extra turns in the conversation, but reduces the possibility of distracting or frustrating users with undesirable information. When we present facts to users during their search (step 10 of Algorithm 1, first two turns of Figure 1), we do not seek prior permission from them, to avoid additional dialogue overhead before the task even begins. However, during task execution, our CTA seeks user approval each time before showing interesting facts (steps 9-10 of Algorithm 2). Users 

can choose to not answer or bypass this request, in which case the next step of the task continues.

(iv) **Seek User Feedback on Facts:** We explicitly ask the user _once_ during the conversation if they liked the fact that was shown to them (seeKUserFeedback() function in Algorithm 2). Users can choose to not answer or bypass this question, and continue to execute their task normally.

## 4 Experiments and Results

### Interesting Fact Quality Evaluation

Relevance and Interestingness Detection:In Section 3.2, we proposed a multi-label classifier to detect relevant and interesting facts. We now assess multiple approaches in detecting relevance and interestingness in Table 2, on a test set of \(400\) facts with a \(50\%\) split of positive and negative candidates. Our proposed classifier (roberta-base-rel-int) improves upon the single-label classifier baselines (roberta-base-rel, roberta-base-int) by at least \(5\%\) F1 score points, in detecting if an input fact is relevant and interesting. We also prompt the gpt-3.5-turbo [6] LLM with \(8\) randomly chosen few-shot examples from our training set to recognize relevance and interestingness individually (gpt-3.5-turbo-fs-rel and gpt-3.5-turbo-fs-int), as well as together in the same prompt (last row of the table). Our multi-label classifier is comparable to GPT-3.5 in detecting relevant facts despite being a much smaller model. Moreover, it significantly outperforms GPT-3.5 by \(10\%\) in detecting the interestingness of facts.

Evaluating LLMs for Annotation:We prompt GPT-4 [7] with our well-defined interestingness feature schema from Section 3.1. Similar to the human annotation in Section 3.2, we ask GPT-4 to label (i) each input fact for specificity, novelty, relevance and informativeness; (ii) overall interestingness based only on the schema features; and (iii) the most important feature to decide a fact's interestingness. On \(75\) randomly selected facts from Section 3.2, we observe \(44\%\) agreement with human labels for overall interestingsness, showing that it is non-trivial to replace humans with LLMs for this task. The highest agreement (\(>58\%\)) is observed for 'specificity' and 'relevance'. The lowest agreement of \(31\%\) is seen for the 'novelty' feature, likely due to its inherent subjectivity. GPT-4 chose 'informativeness' as the key interestingness indicator, in contrast to the 'novelty' feature chosen by human annotators in Section 3.2.

Interesting Facts and User Engagement:We first evaluate the quality of the interesting facts in the dataset we created in Section 3.2, and their effect on the user engagement with a CTA. We provide crowd workers with 200 sampled facts,6 and ask them if the fact is (i) interesting to them; and (ii) likely to engage users interacting with a CTA. In this experiment, we do not provide our feature schema from Section 3.1 to the annotators, letting the definition of 'interestingness' remain subjective and open to interpretation. \(89\%\) of the input facts were chosen as both interesting and engaging to CTA users, reinforcing the effectiveness of our feature schema in defining interestingness, and the utility of our interesting fact dataset.

Footnote 6: Average binary inter-annotator agreement = \(0.68\)

### Conversation Quality Evaluation

We randomly select \(50\) user-CTA cooking conversations, having \(63\) turns with interesting facts. We ask human judges if the facts within the conversations might bore, frustrate or distract users from their original goal. The judges negatively labeled \(22\%\) of the turns with facts, possibly because those facts were not appealing enough to justify the additional fact-related turns (examples in Appendix A).

Interesting Fact Placement in Conversations:We compare different strategies of inserting facts during a conversation between users and CTAs. We select 20 seed conversations, and present the following variants to crowd workers:7 (i) showing interesting facts only during search, before the user selects a task; (ii) showing interesting facts only

\begin{table}
\begin{tabular}{l|c|c} \hline Number of interesting facts & 1,379 \\ Number of unique fact providers & 5 \\ Number of unique entities & 420 \\ Mean fact length in words & 13 \\ \hline \end{tabular}
\end{table}
Table 1: Details of our curated interesting facts dataset.

\begin{table}
\begin{tabular}{l|c|c|c} \hline
**Model** & **F1-relevant** & **F1-interstinguess** & **F1-rel-int** \\ \hline roberta-base-rel & 0.76 & N/A & 0.76 \\ roberta-base-int & N/A & 0.67 & 0.67 \\ roberta-base-net-int (Ours) & 0.83 & **0.80** & **0.82** \\ gpt-3.5-turbo-fs-rel & **0.84** & N/A & N/A \\ gpt-3.5-turbo-fs-int & N/A & 0.72 & N/A \\ gpt-3.5-turbo-fs-rel-int & 0.78 & 0.66 & 0.72 \\ \hline \end{tabular}
\end{table}
Table 2: Detecting relevance and interestingness of facts. ‘rel’, ’int’ and ‘fs’ denote relevance, interestingness, and few shot respectively. N/A means that a model does not predict the corresponding label.

 

during task execution; (iii) always showing facts after seeking user approval; (iv) always showing facts without seeking user approval; and (v) our proposed hybrid fact placement approach in Section 3.3. In \(65\%\) of cases, human judges preferred hybrid conversations where facts were shown both before task selection and during execution. Asking user permission before showing a fact was preferred \(60\%\) of the time. This further validates the UX design choices we made in Section 3.3.

### Live Testing

Setup:We test our proposed interesting facts feature experience within a CTA of a leading commercial voice assistant, for live interaction and use by thousands of users in the US. Our CTA was built on top of the base Alexa Prize infrastructure [1]. The design, implementation and evaluation details of our final set up are described in Sections 3.2, 3.3, 4.1 and 4.2. We study two versions of the CTA, i.e., with and without the interesting facts feature, via A/B testing over a period of 6 weeks. We obtain explicit real-user feedback on our system in the following ways: (i) we ask users if they liked the interesting facts shown to them; (ii) users rate their experience conversing with the CTA with a score from 1 to 5; and (iii) users provide additional feedback via a free-form text field at the end of the conversation.

User Satisfaction and Engagement:We analyzed \(450\) user-CTA conversations, of which \(300\) belonged to the control group, and \(150\) were exposed to our interesting facts UX. Users stated that they liked about \(66\%\) of the interesting facts shown to them during the conversation. The average satisfaction rating provided by users to their conversations grew by \(40\%\) (_p_-value < \(0.05\) with a _t_-test) over the set up without the interesting facts.

The average conversation length between users and the CTA grew by \(37\%\) (_p_-value < \(0.05\) with a _t_-test). This is a favorable outcome overall, because while longer conversations can increase the amount of time spent by users to compete their task, shorter conversations are also likely to indicate task abandonment or user disinterest in continuing the conversation. Recall that the goal of our work is to maximize users' engagement with the CTA, since this can make our CTA more appealing for further adoption and usage (Sections 1 and 2). In fact, we noted a rise in the number of users both starting a task, and continuing their guided task interaction, after seeing an interesting fact. We also observed a \(5\%\) increase in the task completion rate (i.e, the users completing all execution steps of their chosen task), and a \(5\%\) increase in the repeat users interacting with the CTA after testing this feature.

Finally, we examined users' next turn responses to the provided interesting facts, excluding those who explicitly mentioned liking the facts. We did not find any users abruptly ending their conversations after being presented with an interesting fact, or interrupting the CTA via voice as it spoke the fact. Manual inspection showed a positive or neutral sentiment in \(>70\%\) of these user responses.

## 5 Discussion

Scope and Generalizability:The goals of our work are two-fold. The first is the generation of factually accurate interesting facts that are novel, specific, concise, informative and relevant to the task at hand. The second, more complex goal of our work is strategically injecting interesting facts into a user's conversation with a CTA, and involves multiple steps (Section 3.3). In this paper, we empirically evaluate our proposed approach only on the domain of cooking. However, we believe that the various steps and considerations involved in our framework, namely, our theoretically backed inter-estignness feature schema, fact curation and cleaning, and conversational presentation techniques can serve as an inspiration or a useful starting point for other applicable conversational task domains to consider, evaluate and expand upon.

Scalability:Though potentially applicable across domains where interesting facts are available, our interesting fact dataset creation approach presented in Section 3.2 may not scale due to the manual annotation effort involved. We investigated the use of LLMs to annotate interesting facts in Section 4.1. We next explore the automatic generation of interesting facts using an LLM (gpt-3.5-turbo). We sample \(50\) entities from set \({\cal E}\) (Section 3.2), and prompt GPT-3.5 to generate \(50\) relevant and interesting facts as per our schema in Section 3.1. The facts generated by GPT-3.5 are suitable for a user conversing with a CTA. owever, we found that \(16\%\) of them are either factually inaccurate, or of questionable accuracy with no supporting evidence available. For instance, the fact: _The Can Opener Museum located in Cincinnati, Ohio, celebrates the history and 

evolution of can openers_ is completely fabricated - such a museum does not exist.

Therefore, directly showing unverified LLM-generated facts to users in an online manner can severely impair users' experience and trust in a commercial CTA that aims to be a reliably and knowledgeably guide users through costly and lengthy tasks. Online fact generation by an LLM during a conversation also increases the CTA's response latency. One solution, which we use in this work, involves offline correction and retrieval of facts before presenting to users. Another solution could be to use feedback dataset as demonstration data to tune an LLM or an explicit dialogue policy.

Using LLMs in commercial settings also poses a practical challenge, as legal requirements mandate appropriate crediting of original information sources. When prompted for the original sources it uses to generate facts, GPT-3.5 responds: _I'm unable to directly access external websites or provide specific sources for facts as my knowledge_. We will thus need to provide the LLM a retrieval or API-based search component to get these sources automatically, which adds a risk of hallucination. We leave the end-to-end automation of our approach with one or more LLMs for future work.

## 6 Conclusions

Drawing on psychological theories on human interest, this work defines what users might find interesting during guided task-based conversations with CTAs, and empirically demonstrates the value of engaging and satisfying users with appealing external knowledge. Engaged users are, in fact, more likely to complete tasks and leverage the power of CTAs for complex task assistance. To achieve this goal, we defined a semi-automatic framework to build a dataset of \(1,379\) contextual and interesting facts for the cooking domain, and investigated different options to effectively inject the facts into user-CTA conversations. Offline human evaluations and live testing on a multi-modal voice assistant showed significant gains in user satisfaction and engagement in a CTA enriched with our interesting facts experience.

### Limitations and Future Work

Our work did not consider the following aspects, which we leave for future work.

Domain Generalization:Based on the theoretical backing of our framework, we fully expect it to generalize across multiple task domains feasible for a CTA. However, we would like to verify this empirically, by generating interesting facts for other task assistant domains such as home improvement, inserting those facts into user-CTA conversations, and addressing any potential domain-specific challenges that may occur, e.g. recognizing and linking complex named entities Malmasi et al. (2022).

Fact Presentation: Our current approach only considers the presentation of interesting facts in the form of simple text. Our multi-modal CTA currently shows the entire fact text on the screen, and reads out the entire fact by voice. We seek to explore varied modes of presentation combining both voice and the display screen, such as diverse text formats, catch summary phrases, or visual aids to accompany the facts.

Utilization of LLMs:We plan to reduce the number of steps and manual effort involved in our approach to increase its scalability and generalizability, by employing one or more LLMs to automate the process in an end-to-end fashion. This involves using a retrieval augmented LLM with explicit hallucination reduction techniques to generate factually accurate and interesting facts based on our feature schema. We will then use the insights we discovered in this work as well as any existing UX specifications or constraints, to either prompt or fine-tune an LLM to decide when, where and how best to naturally incorporate interesting facts into task-specific conversations.

Furthermore, though we explored the ability of LLMs to annotate as well as generate interesting facts, we did not directly present the factually accurate interesting facts generated by LLMs to real-world users interacting with our CTA. Given the disagreement on interestingness between human and GPT-4's annotations in Section 4.2, we would like to investigate the user engagement and user satisfaction obtained when users are presented facts generated and selected by GPT-4, and how this differs compared to the interesting facts preferred by humans annotators.

## Acknowledgements

We would like to express our sincere gratitude to Zhiyu Chen, Jason Choi, Saar Kuzi as well as the anonymous reviewers for their insightful feedback and comments on our work 



## References

* Agichtein et al. (2023) Eugene Agichtein, Michael Johnston, Anna Gottardi, Lavina Vaz, Cris Flagg, Yao Lu, Shaohua Liu, Satvik Sahai, Giuseppe Castellucci, Jason Ingyu Choi, et al. 2023. Advancing conversational task assistance: the second alexa prize taskbot challenge.
* Berlyne (1949) Daniel E Berlyne. 1949. Interest as a psychological concept. _British Journal of Psychology_, 39(4):184.
* Berlyne (1966) Daniel E Berlyne. 1966. Curiosity and exploration: Animals spend much of their time seeking stimuli whose significance raises problems for psychology. _Science_, 153(3731):25-33.
* Berlyne (1970) Daniel E Berlyne. 1970. Novelty, complexity, and hedonic value. _Perception & psychophysics_, 8(5):279-286.
* Carvalho et al. (2005) Deborah R Carvalho, Alex A Freitas, and Nelson Ebecken. 2005. Evaluating the correlation between objective rule interestingness measures and real human interest. In _Knowledge Discovery in Databases: PKDD 2005: 9th European Conference on Principles and Practice of Knowledge Discovery in Databases, Porto, Portugal, October 3-7, 2005. Proceedings__9_, pages 453-461. Springer.
* Ceha et al. (2021) Jessy Ceha, Ken Jen Lee, Elizabeth Nilsen, Joslin Goh, and Edith Law. 2021. Can a humorous conversational agent enhance learning experience and outcomes? In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_, pages 1-14.
* Choi et al. (2022) Jason Ingyu Choi, Saur Kuzi, Nikhita Vedula, Jie Zhao, Giuseppe Castellucci, Marcus Collins, Shervin Mamasi, Oleg Rokhlenko, and Eugene Agichtein. 2022. Wizard of tasks: A novel conversational dataset for solving real-world tasks in conversational settings. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 3514-3529.
* Clark et al. (2019) Leigh Clark, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, Christine Murad, Cosmin Munteanu, et al. 2019. What makes a good conversation? challenges in designing truly conversational agents. In _Proceedings of the 2019 CHI conference on human factors in computing systems_, pages 1-12.
* Constantin et al. (2019) Mihai Gabriel Constantin, Miriam Redi, Gloria Zen, and Bogdan Ionescu. 2019. Computational understanding of visual interestingness beyond semantics: literature survey and analysis of covariates. _ACM Computing Surveys (CSUR)_, 52(2):1-37.
* FitzGerald et al. (2022) Jack FitzGerald, Shankar Ananthakrishnan, Konstantine Arkoudas, Davide Bernardi, Abhishek Bhagia, Claudio Delli Bovi, Jin Cao, Rakesh Chada, Amit Chauhan, Lucxin Chen, et al. 2022. Alexa teacher model: Pretraining and distilling multi-billion-parameter encoders for natural language understanding systems. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_, pages 2893-2902.
* Folstad and Bare Brandtzeeg (2020) Asbjorn Folstad and Peter Bae Brandtzaeg. 2020. Users' experiences with chatbots: findings from a questionnaire study. _Quality and User Experience_, 5(1):3.
* Geng and Hamilton (2006) Liqiang Geng and Howard J Hamilton. 2006. Interestingness measures for data mining: A survey. _ACM Computing Surveys (CSUR)_, 38(3):9-es.
* He et al. (2023) Jessica He, David Piorkowski, Michael Muller, Kristina Primijoin, Stephanie Houde, and Justin D Weisz. 2023. Understanding how task dimensions impact automation preferences with a conversational task assistant. In _ACM CHI Conference on Human Factors in Computing Systems_.
* Kidd and Hayden (2015) Celeste Kidd and Benjamin Y Hayden. 2015. The psychology and neuroscience of curiosity. _Neuron_, 88(3):449-460.
* Kiseleva et al. (2016) Julia Kiseleva, Kyle Williams, Ahmed Hassan Awadaliah, Aidan C Crook, Imed Zitouni, and Tasos Anastassakos. 2016. Predicting user satisfaction with intelligent assistants. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_, pages 45-54.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Lotze et al. (2021) Tom Lotze, Stefan Klut, Mohammad Aliannejadi, and Evangelos Kanoulas. 2021. Ranking clarifying questions based on predicted user engagement. _arXiv preprint arXiv:2103.06192_.
* Malmasr et al. (2022) Shervin Malmasr, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rohllenko. 2022. SemEval-2022 task 11: Multilingual complex named entity recognition (MulticonNER). In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_, pages 1412-1437, Seattle, United States. Association for Computational Linguistics.
* OpenAI (2023a) OpenAI. 2023a. Chatgrt. [https://openai.com/chatgpt](https://openai.com/chatgpt).
* OpenAI (2023b) OpenAI. 2023b. Gpt-4 technical report. _arXiv preprint arXiv: 2303.08774_. [https://openai.com/gpt4](https://openai.com/gpt4).
* Park et al. (2020) Dookun Park, Hao Yuan, Dongmin Kim, Yinglei Zhang, Matsoukas Spyros, Young-Bum Kim, Ruhi Sarikaya, Edward Guo, Yuan Ling, Kevin Quinn, et al. 2020. Large-scale hybrid approach for predicting user satisfaction with conversational agents. _arXiv preprint arXiv:2006.07113_.
* Pekrun (2019) Reinhard Pekrun. 2019. The murky distinction between curiosity and interest: State of the art and future prospects. _Educational Psychology Review_, 31(4):905-914.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. _arXiv preprint arXiv:1908.10084_

Ivan Sekulic, Mohammad Aliannejadi, and Fabio Crestani. 2021. User engagement prediction for clarification in search. In _Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28-April 1, 2021, Proceedings, Part I 43_, pages 619-633. Springer.
* Silvia (2005) Paul J Silva. 2005. What is interesting? exploring the appraisal structure of interest. _Emotion_, 5(1):89.
* Siro et al. (2022) Clemencia Siro, Mohammad Aliannejadi, and Maarten de Rijke. 2022. Understanding user satisfaction with task-oriented dialogue systems. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 2018-2023.
* Strathearn and Gkatzia (2022) Carl Strathearn and Dimitra Gkatzia. 2022. Task2dial: A novel task and dataset for commonsense enhanced task-based dialogue grounded in documents. _arXiv preprint arXiv:2204.01061_.
* Tomkins (1962) Silvan Tomkins. 1962. _Affect imagery consciousness: Volume I: The positive affects_. Springer publishing company.
* Vicente et al. (2023) Frederico Vicente, Rafael Ferreira, David Semedo, and Joao Magalhaes. 2023. The wizard of curiosities: Enriching dialogues with fun facts. In _Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue_, pages 149-155.
* Vtyurina and Fourney (2018) Alexandra Vtyurina and Adam Fourney. 2018. Exploring the role of conversational cues in guided task support with virtual assistants. In _Proceedings of the 2018 CHI conference on human factors in computing systems_, pages 1-7.
* Vyurina et al. (2017) Alexandra Vtyurina, Denis Savenkov, Eugene Agichtein, and Charles LA Clarke. 2017. Exploring conversational search with humans, assistants, and wizards. In _Proceedings of the 2017 chi conference extended abstracts on human factors in computing systems_, pages 2187-2193.
* Wu and Miao (2013) Qiong Wu and Chunyan Miao. 2013. Curiosity: From psychology to computation. _ACM Computing Surveys (CSUR)_, 46(2):1-26.
* Zhang et al. (2020) Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. 2020. Recent advances and challenges in task-oriented dialog systems. _Science China Technological Sciences_, 63(10):2011-2027.

 



## Appendix A Examples of Conversation Turns with Interesting Facts

Table 3 shows some examples of conversation turns containing interesting facts, and their labels annotated by human annotators.

\begin{table}
\begin{tabular}{l|c|c} \hline
**Entity** & **Conversation turn** & **Human annotation** \\ \hline Ingredient: Sweet potato & Did you know that the vibrant colors of sweet potatoes & Interesting \\  & can be extracted and used as a natural dye for fabrics? From facts.net. & \\ \hline Recipe: Smoked Salmon & According to facts.net, crepe-making is & Boring, frustrating \\ Crepes & an art form in Japan. & or distracting \\ \hline Ingredient: Sausage & Here’s something interesting from thefactsite.com: & Interesting \\  & Sauses play a key role in Australian politics! & \\ \hline Tool: Chopsticks & According to Wikipedia, the first chopistics were used not as & Boring, frustrating \\  & eating thesis but for cooking, stirring and serving. & or distracting \\ \hline Recipe: Mooncake & According to monoando.com, half-baked mooncakes must be taken out & Interesting \\  & and cooled for 15 minutes before continuing baking. & \\ \hline Ingredient: Candy & From earththis.com, weird enough, cotton candy was invented & Interesting \\  & in 1897 by a dentist. & Interesting \\ \hline Recipe: Chocolate cake & According to facts.net, the secret to a moist cake is to & Interesting \\  & use ingredients like buttermilk, sour cream, or yogurt, which add moisture. & Interesting \\ \hline Ingredient: Pumpkin spice & Here’s an interesting fact from facts.net: pumpkin spice does not & Interesting \\  & actually contain pumpkins. & \\ \hline Tool: Baking tin & From organicfacts.net, aluminium is considered the best material for & Boring, frustrating \\  & a baking tin as it allows for a quick transfer of heat. & or distracting \\ \hline Ingredient: Baking soda & According to tasty.co, baking soda must be replaced every month, & Interesting \\  & otherwise a bit more than the recipe calls for can be added. & Interesting \\ \hline \end{tabular}
\end{table}
Table 3: Examples of conversation turns annotated for interestingness by human annotators.