EVENT REPORT

Dagstuhl Seminar 19461 on Conversational Search
Seminar Goals and Working Group Outcomes
Avishek Anand
Leibniz Universität Hannover, DEanand@kbs.uni-hannover.de

Lawrence Cavedon
RMIT University - Melbourne, AU
lawrence.cavedon@rmit.edu.au

Matthias Hagen
MLU Halle, DEmatthias.hagen@informatik.uni-halle.deHideo Joho
University of Tsukuba - Ibaraki, JP
hideo@slis.tsukuba.ac.jp

Mark Sanderson
RMIT University - Melbourne, AU
mark.sanderson@rmit.edu.au

Benno Stein
Bauhaus-Universität Weimar, DE
benno.stein@uni-weimar.de

Abstract
In the week of November 10–15, 2019, 44 researchers from the fields of information retrieval
and Web search, natural language processing, human computer interaction, and dialoguesystems met for the Dagstuhl Seminar 19461 “Conversational Search” to share the latest
development in the area of conversational search and discuss its research agenda and future
directions. The clear signal from the seminar is that research opportunities to advance conversational search are available to many areas and that collaboration in an interdisciplinary
community is essential to achieve the goals. This report overviews the program and selected
findings of the working groups.

1

Seminar Motivation and Goals

The conversational search paradigm promises to satisfy information needs using human-friendly
dialogs, be it in spoken or in written form [Burtsev et al., 2017, Culpepper et al., 2018, Radlinski and Craswell, 2017, Spina et al., 2018]. This kind of “information-providing dialogs” and“information-seeking conversations” will increasingly happen en passant and spontaneously, inpart triggered by smart objects surrounding us like intelligent assistants such as Amazon Alexa,
ACM SIGIR Forum

1

Vol. 54 No. 1 June 2020

Apple Siri, Google Assistant, and Microsoft Cortana, domestic appliances, environmental control
devices, toys, or autonomous robots and vehicles. The outlined development marks a paradigm
shift for information technology, and the key question(s) include(s):What does conversational search mean and how to make the most of it—given thepossibilities, restrictions and challenges that come with this paradigm?
Currently, our understanding is too limited to exploit the conversational search paradigm for effectively satisfying the existing diversity of information needs. This first Dagstuhl Seminar on
conversational search brought together leading researchers from relevant communities to understand and to analyze this promising retrieval paradigm and its future from different angles. The
participants discussed issues related to interactivity, result presentation, query clarification, usermodels, and evaluation, but also search behavior that can facilitate a human-machine debate oran argumentation related to the information need in question. Moreover, the participants defined, shaped, and formalized a set of corresponding problems to be addressed, and highlighted
associated challenges that are expected to come in the form of multiple modalities and multiple
users.
The complete report of the seminar can be found on the seminar web page.1 Some working
groups plan to make their individual reports available as arXiv preprints.

2

Seminar Program

The 5-day program of the seminar consisted of six introductory and background sessions, threevisionary talk sessions, one industry talk session, and nine breakout discussion and reporting
sessions. The seminar also involved three social events (two evening ”sessions” in form of a
pub quiz and of a conversational search hands-on experiment, and the traditional excursion on
Wednesday afternoon). The detailed program of the seminar is available online.2
Pre-Seminar Activities
Prior to the seminar, the participants were asked to provide inputs to the following questions:
1. What are your ideas of the “ultimate” conversational search system?
2. Please list, from the perspective of your research field, important open questions or challenges
in conversational search.
3. What are three papers a new PhD student in conversational search should read and why?
From the survey, the following topics initially emerged as interests of the participants. Manyof these topics were discussed at length in the seminar.
• Understanding the nature of information seeking in the context of conversational agents.
• Modelling problems in conversational search.
1
2

https://www.dagstuhl.de/19461
https://www.dagstuhl.de/schedules/19461.pdf

ACM SIGIR Forum2

Vol. 54 No. 1 June 2020• Clarification and explanation.
• Evaluation in conversational search systems.
• Ethics and privacy in conversational systems.
• Extending the problem space beyond the search interface and Q&A.
Another outcome of the above pre-seminar questions was a compilation of a list of recommended
readings to gain a solid understanding of the diverse topics and technologies that are related to
the research on conversational search. The reading list is provided as an Appendix and can also
be found in the seminar report.3
Invited Talks
An important goal of this seminar was to bring a broad range of researchers together to discussestablishing common terminologies among the participants. Therefore, a series of invited talks
was interspersed throughout the seminar program to facilitate the understanding and discussionof conversational search and its potential enabling technologies. The list of talks included thefollowing:
Introductory, Background, and Industry Talks• What have we Learned about Information Seeking Conversations? by Nicholas J. Belkin(Rutgards University, NJ, US)• Personal Knowledge Graphs by Krisztian Balog (University of Stavanger, NOR)
• Beyond Information Retrieval: Socially Interactive Agents by Elisabeth André (Universität
Augsburg, DE)• Conversational Search at Microsoft by Jaime Teevan (Microsoft Corporation - Redmond,
US)• Introduction to NLP by Ido Dagan (Bar-Ilan University – Ramat Gan, IL) and Idan Szpektor(Google Israel - Tel-Aviv, IL)• Dynamic Composition for Domain Exploration Dialogues by Idan Szpektor (Google Israel Tel-Aviv, IL)• Introduction to Deep Learning in NLP by Idan Szpektor (Google Israel - Tel-Aviv, IL)• Computational Argumentation by Henning Wachsmuth (Universität Paderborn, DE)• Searching for Myself: One Naı̈ve Individual’s Human-Centered Audio-Visual Search by
Sharon Oviatt (Monash University - Clayton, AU)• Overview of Dialogue by Phil Cohen (Monash University - Clayton, AU)
3https://www.dagstuhl.de/19461

ACM SIGIR Forum

3

Vol. 54 No. 1 June 2020

• Conversational User Interfaces by Leigh Clark (Swansea University, GB)• A Theoretical Framework for Conversational Search by Filip Radlinski (Google UK - London,GB)• Conversational Search and Recommendation at Spotify by Rosie Jones (Spotify - US)• Conversational Product Search by Ronald M. Kaplan (Stanford University, US)• Conversation about Preferences by Filip Radlinski (Google UK - London, GB)• Lessons We Learnt in Xiaoice by Ruihua Song (Microsoft XiaoIce- Beijing, CN),
Visionary Talks• Conceptual Model of Human-Agent Interactions in Conversational Search by Leif Azzopardi(University of Strathclyde – Glasgow, GB)• Towards an Immersive Wikipedia by Bernd Fröhlich (Bauhaus-Universität Weimar, DE)• Conversational Style Alignment for Conversational Search by Ujwal Gadiraju (Leibniz Universität Hannover, DE)• The Dilemma of the Direct Answer by Martin Potthast (Universität Leipzig, DE)• Conversational Question Answering over Knowledge Graphs by Rishiraj Saha Roy (MPI für
Informatik - Saarbrücken, DE)• Ranking People by Markus Strohmaier (RWTH Aachen, DE)• Demystifying Spoken Conversational Search by Johanne Trippas (RMIT University - Melbourne, AU)• Knowledge-based Conversational Search by Svitlana Vakulenko (WirtschaftsuniversitätWien, AT)
• Clarification in Conversational Search by Hamed Zamani (Microsoft Corporation, US)• Macaw: A General Framework for Conversational Information Seeking by Hamed Zamani(Microsoft Corporation, US) and Nick Craswell (Microsoft Corporation, US)

3

Working Groups

In the afternoon of Day 2, initial working groups were formed based on the inputs to the preseminar questionnaires, introductory and background talks, and discussions among the participants. On Day 3, the grouping was revisited and updated, and, eventually, the following sevengroups were formed to focus on topics such as the definition, evaluation, modelling, explanation, scenarios, applications, and prototypes of conversational search. The summaries and groupmembers were as follows.
ACM SIGIR Forum

4

Vol. 54 No. 1 June 2020

Defining Conversational Search This group aimed to bring structure and common terminology to the different aspects of conversational search systems that characterise the field. After
reviewing existing concepts such as conversational answer retrieval and conversational information
seeking, the group devised a typology of conversational search systems via functional extensions of
information retrieval systems, chatbots, and dialogue systems. The group further elaborated theattributes of conversational search by discussing its dimensions and desirable additional properties. Their report suggests types of systems that should not be confused as conversational searchsystems.
Group Members: Jaime Arguello (University of North Carolina - Chapel Hill, US), Lawrence
Cavedon (RMIT University - Melbourne, AU), Jens Edlund (KTH Royal Institute of Technology - Stockholm, SE), Matthias Hagen (Martin-Luther-Universität Halle-Wittenberg, DE), David
Maxwell (University of Glasgow, GB), Martin Potthast (Universität Leipzig, DE), Filip Radlinski(Google UK - London, GB), Mark Sanderson (RMIT University - Melbourne, AU), Laure Soulier(UPMC - Paris, FR), Benno Stein (Bauhaus-Universität Weimar, DE), Jaime Teevan (Microsoft
Corporation - Redmond, US), Johanne Trippas (RMIT University - Melbourne, AU), and Hamed
Zamani (Microsoft Corporation, US).
Evaluating Conversational Search This group addressed how to evaluate the quality of
conversational search systems. It first described the complexity of conversation between searchsystems and users, followed by a discussion of the motivation and broader tasks as the context
of conversational search that can inform the design of conversational search evaluation. The
group also surveyed 12 recent tasks and datasets that can be exploited for the evaluation of
conversational search. Their report presents several dimensions in the evaluation such as user,
retrieval, and dialog, and suggests that the dimensions might overlap with those of interactive
information retrieval.
As human-to-human conversations differ from human-to-machine conversations, it was argued
that (at least in the near future) we should optimize conversational search systems based onhuman-machine conversations that are grounded in current retrieval systems and technologies.
The different dimensions in which we can evaluate conversational search and conversational spoken
search could be—user-focused, retrieval-focused or dialogue-focused. Lab-based and A/B testing
will typically involve a complete (or simulated) system setup and thus facilitate end-to-end (e2e)evaluation. However, given the highly interactive nature of conversational search, it is unlikelythat a reusable test collection will be developed to support any serious e2e evaluations—testcollections, however, should be able to support component level evaluation.
Group Members: Jaime Arguello (University of North Carolina at Chapel Hill, US), Avishek
Anand (Leibniz Universität Hannover, DE), Leif Azzopardi (University of Strathclyde, GB),
Robert Capra (University of North Carolina at Chapel Hill, US), Leigh Clark (Swansea University, GB), Jens Edlund (KTH Royal Institute of Technology - Stockholm, SE), Norbert Fuhr
(Universität Duisburg-Essen, DE), Ujwal Gadiraju (Leibniz Universität Hannover, DE), Claudia Hauff (Delft University of Technology, NL), and Rishiraj Saha Roy (MPI für Informatik Saarbrücken, DE)

ACM SIGIR Forum

5

Vol. 54 No. 1 June 2020

Modeling Conversational Search This group addressed what should be modeled from thereal world to achieve a successful conversational search and how. They discussed why a rangeof concepts and variables such as capabilities and resources of systems, beliefs, and goals ofusers, history and current status of process, and search topics and tasks should be consideredto advance understanding between systems and users in the context of conversational search.
Their report points out that the options the current search engines present to users can be too
broad in conversational interaction. It suggests that a deeper modeling of users’ beliefs andwants, development of reflective mechanisms, and finding a good balance between macroscopic
and microscopic modeling are promising directions for future research.
Group Members: Elisabeth André (Universität Augsburg, DE), Nicholas J. Belkin (Rutgers University - New Brunswick, US), Phil Cohen (Monash University - Clayton, AU), Ronald M. Kaplan(Stanford University, US), Martin Potthast (Universität Leipzig, DE), Johanne Trippas (RMIT
University - Melbourne, AU), and Arjen P. de Vries (Radboud University Nijmegen, NL).
Argumentation and Explanation Motivated by inevitable influences made to users due to thecourse of actions and choices of search engines, this group explored how the research on argumentation and explanation can mitigate some of the potential biases generated during conversational
search processes, and facilitate users’ decision-making by acknowledging different viewpoints of a
topic. The group proposed a research scheme that consists of three layers: a conversational layer,
a demographics layer, and a topic layer. Their report also explains that argumentation and explanation should be carefully considered when search systems (1) select, (2) arrange, and (3) phrase
the information presented to the users. Creating an annotated corpus with these elements is thenext step in this direction.
Group Members: Khalid Al-Khatib (Bauhaus-Universität Weimar, DE), Ondrej Dusek (Charles
University - Prague, CZ), Benno Stein (Bauhaus-Universität Weimar, DE), Markus Strohmaier(RWTH Aachen, DE), Idan Szpektor (Google Israel - Tel-Aviv, IL), and Henning Wachsmuth(Universität Paderborn, DE).
Scenarios for Conversational Search This group aimed to identify scenarios that invite conversational search, given that natural language conversation might not always be the best way tosearch in some contexts. Their report summarises modalities and tasks of search as two dimensions to define where conversational search might make sense. Modality can be determined by asituation such as driving or cooking, or devices at hand such as a smartwatch or AR/VR systems.
As for the task, the group report explains that the usefulness of conversational search increases
as the level of exploration and complexity increases in tasks. On the other hand, simple information needs, highly ambiguous situations, or very social situations might not provide appropriate
scenarios for conversational search. Proposed scenarios include a mechanic fixing a machine, twopeople searching for a place for dinner, learning about a recent medical diagnosis, and following
up on a news article to learn more.
Group Members: Lawrence Cavedon (RMIT University - Melbourne, AU), Bernd Fröhlich(Bauhaus-Universität Weimar, DE), Hideo Joho (University of Tsukuba - Ibaraki, JP), Ruihua
Song (Microsoft XiaoIce- Beijing, CN), Jaime Teevan (Microsoft Corporation - Redmond, US),ACM SIGIR Forum

6

Vol. 54 No. 1 June 2020

Johanne Trippas (RMIT University - Melbourne, AU), and Emine Yilmaz (University College
London, GB).
Conversational Search for Learning Technologies This group discussed the implication of
conversational search from learning perspectives. The report highlights the importance of searchtechnologies in lifelong learning and education, and the challenges due to th complexity of learning
processes. The group points out that multimodal interaction is particularly useful for educational
and learning goals since it can support students with diverse backgrounds. Based on these discussions, the report suggests several research directions including the extension of the modalities
to speech, writing, touch, gaze, and gesturing, the integration of multimodal inputs/outputs withexisting IR techniques, and the application of multimodal signals to user modelling.
Group Members: Sharon Oviatt (Monash University - Clayton, AU) and Laure Soulier (UPMC Paris, FR).
Common Conversational Community Prototype: A Scholarly Conversational Assistant This group explored potential domain applications that could facilitate research; it proposed developing and operating a prototype conversational search system for scholarly activities
as academic resources that support research into conversational search. Example activities include
finding articles for a new area of interest, planning sessions to attend in a conference, or determining conference PC members. The proposed prototype is expected to serve as a useful searchtool, a means to create datasets, and a platform for community-based evaluation campaigns. The
group also outlined a road map of the development of such a scholarly conversational assistant.
The report describes a set of software platforms, scientific IR tools, open source conversational
agents, and data collections that could be exploited in conversational search work.
Group Members: Krisztian Balog (University of Stavanger, NOR), Lucie Flekova (Technische
Universität Darmstadt, DE), Matthias Hagen (Martin-Luther-Universität Halle-Wittenberg, DE),
Rosie Jones (Spotify, US), Martin Potthast (Leipzig University, DE), Filip Radlinski (Google, UK),
Mark Sanderson (RMIT University, AUS), Svitlana Vakulenko (University of Amsterdam, NL),
and Hamed Zamani (Microsoft, US).

Acknowledgements
The seminar organisers would like to thank all participants for their active contributions. We also
thank the staff of Schloss Dagstuhl for providing a great venue for a successful seminar.

References
Mikhail Burtsev, Aleksandr Chuklin, Julia Kiseleva, and Alexey Borisov. Search-Oriented Conversational AI (SCAI). In Jaap Kamps, Evangelos Kanoulas, Maarten de Rijke, Hui Fang,and Emine Yilmaz, editors, Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR 2017, Amsterdam, The Netherlands, October 1–4, 2017,
ACM SIGIR Forum

7

Vol. 54 No. 1 June 2020

pages 333–334. ACM, 2017. doi: 10.1145/3121050.3121111. URL https://doi.org/10.1145/
3121050.3121111.
J. Shane Culpepper, Fernando Diaz, and Mark D. Smucker. Research Frontiers in Information Retrieval: Report from the Third Strategic Workshop on Information Retrieval in Lorne(SWIRL 2018). SIGIR Forum, 52(1):34–90, 2018. doi: 10.1145/3274784.3274788. URL
https://doi.org/10.1145/3274784.3274788.
Filip Radlinski and Nick Craswell. A Theoretical Framework for Conversational Search. In Ragnar
Nordlie, Nils Pharo, Luanne Freund, Birger Larsen, and Dan Russel, editors, Proceedings of the2017 Conference on Conference Human Information Interaction and Retrieval, CHIIR 2017,
Oslo, Norway, March 7–11, 2017, pages 117–126. ACM, 2017. doi: 10.1145/3020165.3020183.
URL https://doi.org/10.1145/3020165.3020183.
Damiano Spina, Jaime Arguello, Hideo Joho, Julia Kiseleva, and Filip Radlinski. CAIR’18: Second
International Workshop on Conversational Approaches to Information Retrieval at SIGIR 2018.
SIGIR Forum, 52(2):111–116, 2018. doi: 10.1145/3308774.3308793. URL https://doi.org/
10.1145/3308774.3308793.

ACM SIGIR Forum

8

Vol. 54 No. 1 June 2020

Appendix: Recommended Reading List
The publications below were recommended by the seminar participants via a pre-seminar survey.
There are also specialized reading lists available in the respective reports of the working groups.
[1] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny
Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth
Kikin-Gil, and Eric Horvitz. Guidelines for Human-AI Interaction, CHI 2019. https://doi.org/10.1145/3290605.3300233[2] Aliannejadi, Mohammad, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. Asking
clarifying questions in open-domain information-seeking conversations. SIGIR 2019. https://doi.org/10.1145/3331184.3331265[3] Belkin, Nicholas J., Colleen Cool, Adelheit Stein, and Ulrich Thiel. Cases, scripts, andinformation-seeking strategies: On the design of interactive information retrieval systems.
Expert systems with applications, 9 (3), 1995. https://doi.org/10.1016/0957-4174(95)00011-W[4] Timothy Bickmore, Justine Cassell. Social dialogue with embodied conversational agents.
Advances in natural multimodal dialogue systems, 2005. https://doi.org/10.1007/
1-4020-3933-6_2[5] Daniel Braun, Adrian Hernandez-Mendez, Florian Matthes, Manfred Langen. Evaluating
natural language understanding services for conversational question answering systems. SIGdial 2017. https://doi.org/10.18653/v1/W17-5522[6] Andrew Breen, et al. Voice in the User Interface. Interactive Displays: Natural HumanInterface Technologies, 2014, https://doi.org/10.1002/9781118706237.ch3.
[7] Brennan, Susan E., and Eric A. Hulteen. Interaction and feedback in a spoken language
system: A theoretical framework. Knowledge-based systems, 8 (2-3), 1995. https://doi.
org/10.1016/0950-7051(95)98376-H[8] Harry Bunt. Conversational principles in question-answer dialogues. Zur Theorie der Frage,
pages 119-141.
[9] Justine Cassell. Embodied conversational agents. AI Magazine, 22(4), 2001. https://doi.
org/10.1609/aimag.v22i4.1593[10] Justine Cassell, Joseph Sullivan, Elizabeth Churchill, Scott Prevost. Embodied conversational agents. MIT Press, 2000.
[11] Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang,
Luke Zettlemoyer. QuAC: Question Answering in Context. EMNLP 2018. https://dx.
doi.org/10.18653/v1/D18-1241[12] Christakopoulou, Konstantina, Filip Radlinski, and Katja Hofmann. Towards conversational
recommender systems. SIGKDD 2016. https://doi.org/10.1145/2939672.2939746[13] Leigh Clark, Phillip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl, Jens Edlund,
Matthew Aylett, João Cabral, Cosmin Munteanu, Benjamin Cowan. The State of Speech
ACM SIGIR Forum

9

Vol. 54 No. 1 June 2020

in HCI: Trends, Themes and Challenges. Interacting with Computers, 31 (4), 2019. https://doi.org/10.1093/iwc/iwz016[14] Mark Core and James Allen. Coding dialogs with the DAMSL annotation scheme. AAAI
Fall Symposium on Communicative Action in Humans and Machines, 1997.
[15] Paul Grice. Studies in the Way of Words. Harvard University Press, 1989.
[16] Haider, Jutta, and Olof Sundin. Invisible Search and Online Search Engines: The ubiquity
of search in everyday life. Routledge, 2019.
[17] Ben Hixon, Peter Clark, Hannaneh Hajishirzi. Learning knowledge graphs for question
answering through conversational dialog. NAACL 2015. http:s//dx.doi.org/10.3115/
v1/N15-1086[18] Mohit Iyyer, Wen-tau Yih, Ming-Wei Chang. Search-based neural structured learning forsequential question answering. ACL 2017. https://dx.doi.org/10.18653/v1/P17-1167[19] Diane Kelly and Jimmy Lin. Overview of the TREC 2006 ciQA task. SIGIR Forum 41(1),2007. https://doi.org/10.1145/1273221.1273231[20] Liu, Bei, Jianlong Fu, Makoto P. Kato, and Masatoshi Yoshikawa. Beyond narrative description: Generating poetry from images by multi-adversarial training. ACM Multimedia,
2018. https://doi.org/10.1145/3240508.3240587[21] Dominic W. Massaro, Michael M. Cohen, Sharon Daniel, Ronald A Cole. Developing andevaluating conversational agents. Human performance and ergonomics, 1999. https://doi.org/10.1016/B978-012322735-5/50008-7[22] McTear, Michael F. Spoken dialogue technology: enabling the conversational user interface.
ACM Computing Surveys, 34 (1), 2002. https://doi.org/10.1145/505282.505285[23] Oddy, Robert N. Information retrieval through man-machine dialogue. Journal of documentation, 33 (1), 1977. https://doi.org/10.1108/eb026631[24] Filip Radlinski, Nick Craswell. A theoretical framework for conversational search. CHIIR
2017. https://doi.org/10.1145/3020165.3020183[25] Siva Reddy, Danqi Chen, Christopher D. Manning. CoQA: A conversational question answering challenge. TACL, 7, 2019. https://doi.org/10.1162/tacl_a_00266[26] Ren, Gary, Xiaochuan Ni, Manish Malik, and Qifa Ke. Conversational query understanding
using sequence to sequence modeling. The Web Conference, 2018. https://doi.org/10.
1145/3178876.3186083[27] Zsófia Ruttkay, Catherine Pelachaud. From brows to trust: Evaluating embodied conversational agents. Springer Science & Business Media, 2004. https://doi.org/10.1007/
1-4020-2730-3[28] Shum, Heung-Yeung, Xiao-dong He, and Di Li. From Eliza to XiaoIce: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering,
19 (1), 2018. https://doi.org/10.1631/FITEE.1700826

ACM SIGIR Forum

10

Vol. 54 No. 1 June 2020

[29] Adelheit Stein, Elisabeth Maier. Structuring Collaborative Information-Seeking Dialogues.
Knowledge-Based Systems, 8(2-3), 1995. https://doi.org/10.1016/0950-7051(95)98370-L[30] Oriol Vinyals, Quoc Le. A neural conversational model. ICML Deep Learning Workshop,
2015. https://arxiv.org/abs/1506.05869[31] Marylyn Walker, Dianne Litman, Candace Kamm, Alicia Abella. PARADISE: A framework
for evaluating spoken dialogue agents. ACL 1997. https://dx.doi.org/10.3115/976909.
979652[32] Weston, J., Bordes, A., Chopra, S., Rush, A.M., van Merriënboer, B., Joulin, A. and
Mikolov, T. Towards ai-complete question answering: A set of prerequisite toy tasks.
https://arxiv.org/abs/1502.05698[33] Wu, Wei, and Rui Yan. Deep Chit-Chat: Deep Learning for Chit-Chat. SIGIR 2019.
https://doi.org/10.1145/3331184.3331388[34] Zhang, Yongfeng, Xu Chen, Qingyao Ai, Liu Yang, and W. Bruce Croft. Towards conversational search and recommendation: System ask, user respond. CIKM 2018. https://doi.org/10.1145/3269206.3271776[35] Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. A dataset for document grounded
conversations. EMNLP 2018. https://dx.doi.org/10.18653/v1/D18-1076[36] Zhou, Li, Jianfeng Gao, Di Li, and Heung-Yeung Shum. The design and implementationof XiaoIce, an empathetic social chatbot. Computational Linguistics, 2020. https://doi.
org/10.1162/coli_a_00368

ACM SIGIR Forum

11

Vol. 54 No. 1 June 2020