Social Commonsense-Guided Search Query Generation for Open-Domain Knowledge-Powered Conversations
Authors: Revanth Gangi Reddy
Hao Bai
Wentao Yao
Sharath Chandra Etagi Suresh
Heng Ji
ChengXiang Zhai
Venue: EMNLP 2023 Findings (2023)
Keywords: Conversational AI
Knowledge-Powered Dialog
Commonsense Knowledge
TL;DR: To tackle passive conversations, we propose to integrate social commonsense reasoning for the generation of search queries in knowledge-powered conversations.

Abstract: Open-domain dialog involves generating search queries that help obtain relevant knowledge for holding informative conversations. However, it can be challenging to determine what information to retrieve when the user is passive and does not express a clear need or request. To tackle this issue, we present a novel approach that focuses on generating internet search queries that are guided by social commonsense. Specifically, we leverage a commonsense dialog system to establish connections related to the conversation topic, which subsequently guides our query generation. Our proposed framework addresses passive user interactions by integrating topic tracking, commonsense response generation and instruction-driven query generation. Through extensive evaluations, we show that our approach overcomes limitations of existing query generation techniques that rely solely on explicit dialog information, and produces search queries that are more relevant, specific, and compelling, ultimately resulting in more engaging responses.

Social Commonsense-Guided Search Query Generation for Open-Domain Srowledge-Powereee Conversations
Revanth Gangi Reddy Hao Bai Wentao Yao Sharath Chandra Etagi Suresh
Heng Ji ChengXiang Zhai
University of Illinois at Urbana-Champaign (revanth3, haob2, wentaoy4, sce3, hengji, czhai)eillinois.c edu
Abstract
Topic Tracker
Hey. kt's chat
Surcl Noo DECAE)
Conjuning
Open-domain dialog involves generating search queries that help obtain relevant knowledge for holding informative conversations. How- ever. it can be challenging to determine what information to retrieve when user pas- sive and does not express clear need or re- quest. To tackle this issue, we present a novel approach that focuses on generating internet search queries that are guided by social com- monsense, Specifically, we leverage a common- sense dialog system establish connections related to the conversation topic, which sub- sequently guides our query generation. Our proposed framework addresses passive user in- leractions by integrating topic tracking, com- monsense response generation and instruction- driven query generation. Through extensive evaluations, we show that our approach' over- comes limitations of existing query generation techniques that rely solely on explicit dialog information, and produces search queries that are more relevant, specific, and compelling, ul- timately resulting in more engaging responses.
Commonsenaa Responder
taxouttt model?
Cormmneeensa Associatior
Hiso The Conuriy
movie reviews
Favv you alrendy reen
Vee
Talk about reviow
Dialog Context
have heard the movie has good roviews
Baseline Query Generator
Query Generator
What is corguing
Wint are the oesinno the corjaing movie?
Knowledge Selection Response Generation
The Conuring 2013 American supemati nil homer film diretle by
"hhe Conkring seernel souilly erdattering tiniing deo-echool frekkou and one of the smarleal, most viccanllyy effective thillers in recent memory.
Jamss Vien and wittin by Chidd Hayes and Carey Hayes.
Corjuiing from other hamor monies you'ru sour?
apert from The Corpurin??
Baseline Reaponse
Our Response
Our proposed query generation vs baseline query generator. The commonsense responder identifies associations related to the dialog topic (e.g., movies reviews) and provides a latent directive (in the form response) to guide the generation of search queries.
Figure 1:
1 Introduction
Conversational systems have evolved to include personal assistants, task-oriented bots, and open- domain dialog agents for casual conversations. For these agents to maintain engaging and informative discussions, it is crucial to access external knowl- dialog (Dinan
clear guidance for the conversation (Hardy et al., 2021). Nevertheless, in open-domain conversa- tions, users can introduce any topic, and designing a comprehensive algorithm that can produce a rele- vant query in response to random user topic poses unique, complex challenge that has not been ex- plored in prior rescarch.
edge. Holding a knowledge-powere et al.; Komeili et al., 2022; Li et al., 2022; Lai et al., 2023a) typically involves the generation of search query that help gather the most relevant infor- mation continue the conversation. While such queries are more obvious when the user explicitly asks for certain information, a.k.a. conversational information seeking (Zamani ct al., 2022), it un- clear what information should be pursued when users are passive, disengaged, and do not provide
To tackle this challenge, we propose to integrate social commonsense reasoning for the generation of search queries in knowledge-powene tions. Social commonsense (Moore, 2006) refers to the general knowledge about social situations and human behavior used to connect conversation topics and guide discussions. We thus hypothesize that by leveraging deeper understanding of social commonsense and the unspoken cues that guide human conversation, chatbots can become more
conversa-
'Code and models are available here: http:://gthab..commgggiiwwaaaaad qur.rlgerrrcrr-ggeaatttiiio

---

adept at navigating passive conversations. Concretely, we introduce a novel framework that uses a commonsense response as a latent direc- live for an instruction-folloiing query generator. Our approach incorporates the use of topic tracking ($2.1) to first identify the main point of discussion, followed by commonsense-bene based response genera- tion that can associate concepts the main topic to give a latent commonsense directive ($2.2). Finally, we use instruction-drivve query generation ($2.3) to output a search query that adheres to the latent directive within the commonsense response.
ToToic Tracking
Prompt
You are given a diaing betweon user and bot. You need to identlly the main topic being discuseed.
2. Commonsenee Generation
Context
Situation
<Dhalog Contests
A having Irfondll corversstion user- They are talking about <ispics
Toolc
You Instruction you need generate a resporise the
3. Search Query Generation
Prompt
You are glven a short dalog between a user and bat for discussion <ispies. Convort the bot response to search query use for internet.
Context
<Dialog Contects
<Diisig Contasts +Cummoreenae Responsee
Context
Commonsennee
Resppoee
(aaent directive)
Search Query
Our method overcomes the limitations of exist- ing techniques (Shuster et al., 2022a,b; Cai al., 2022: Lai al., 2023b) that solety depend on ex- plicit information present in the conversation to generate search queries. Such an approach is sub- optimal in case of passive conversations, where the human isn't necessarily asking for any specific in- formation. Figure shows an example comparing our approach against a baseline query generation system (Shuster al., 2022b). Our topic tracking identifies "The Conjuring' the subject, with the commonsense responder making the association movie + reviews to output latent commonsense directive that refers to discuss movie reviews. This directive guides the search query generator to out- put a query for the movie reviews, the results of which lead to more engaging bot response com- pared to the baseline, can be seen in Figure 1.
Figure 2: Interaction of different components within our proposed query generation framework.
that track a fixed set of broad high-level topics (e.g.. movies, sport), our objective is to detect uncon- strained, finer-grained topics (such as movie/actor names teams). For fine- fine- -raiined topic tracking, apply an instruction-tuma model (Chung al., 2022) identify the current topic from the dialog context. We utilize the prompt in Figure and rely on instruction-tung models with strong zero-shot abilities (Wei et al., 2021) due to lack of training data for such topic tracking in dialog". An alterna- tive topic tracking approach could follow Shuster et al. (2022a); Adolphs et al. (2022), extracting top- ics relevant entities grounding the final response.
2.2 Commonsense-Basel Directive
Social commonsense-based dialog systems (Kim et al., 2022a,b; Zhou et al., 2021) typically demon- strate a fundamental understanding of handling and responding to specific topics or situations. They involve using external commonsense knowledge graphs such as ConceptNet (Speer et al., 2017) or ATOMIC (Sap et al., 2019) to collect triples for response generation (Zhou al., 2022), or distill- ing such knowledge into language models (LM) through large-scale pretraining (Kim et al., 2022a; Chen et al., 2023) for direct response generation. In this work, we adopt the latter approach, by us- ing pretrained LM to derive the commonsense directive in the form of a response.
2 A Novel Query Generation Framework
In this section, we present our framework for generating search queries by leveraging common- sense reasoning. Our approach consists of three main components: topic tracking to pinpoint the core subject, commonsense-based response gener- ation that relates concepts with the primary topic and provides a latent commonsense directive, and instruction-drivee query generation to produce a search query capable of retrieving relevant infor- mation that follows the commonsense directive. Figure 2 illustrates how these components are inte- grated, with each step described in detail below.
Specifically, we use Cosmo (Kim al., 2022a), which was trained on socially-grounded synthetic dialogues generated by prompting InstructGPT (Ouyang et al., 2022) using contextualized com- monsense knowledge from ATOMIC. Cosmo takes
2.1 Fine-Grained Topic Tracking
Topic tracking (Nakata al., 2002) aims identify the primary subject of the discussion in free-form dialogs, and has been demonstrated (Guo et al., 2018) to improve the coherence of dialog systems. Unlike previous approaches (Khatri et al., 2018)
We note that topic tracking is related to open-domain dialog, and is different from state tracking (Williams et al., 2013) which is specific to task-oriiented dialog.

---

situation narrative and role instruction as input, and generates a response based on the dialog con- text. We also integrate the topic tracking output into the situation narrative definition, as illustrated in Figure 2. Subsequently, Cosmo's s output serves as the latent commonsense directive to guide search query generation, which is discussed next.
set, yielding 20k examples.
Internet Search and Response Generation We obtain search results by scoring passages from the top-3 Bing Search pages using a reranker'. With our primary focus being query generation, we sim- ply prompt ChatGPT to generate the response by incorporating the top search result given the dialog context. We also consider a "no query" baseline that corresponds to generating responses directly from ChatGPT (gpt-3.5-turbo-D0 out internet search. For both search query genera- tion and final response generation, we employ the nucleus sampling method (Holtzman et al., 2019) with P set at 0.9 and a temperature of 0.7. We set tokens to 40 and 100 for search query genera- tion and final response generation respectively.
2.3 Instruction-Driven Query Generation
Given the dialog context, conversation topic and a latent directive in the form of a commonsense re- sponse, we aim to generate a search query to obtain relevant information for continuing the conversa- tion. We utilize an instruction-tuned model (Chung et al., 2022) for query generation, by prompting (sce Figure 2) it to transform the commonsense re- sponse into search query, while incorporating the fine-grained topic to enhance relevance and speci- ficity. Essentially, the commonsense response em- bodies the bot's informational requirement, guiding it to obtain the mentioned information.
version) with-
3.2 Evaluation
In our evaluation, we focus on Wol test set with dia- log turns that had scarch queries annotated for gen- erating responses, while specifically targeting "pas- sive turns" where users don't 't explicitly request in- formation. Using an intent detection model (Khatri et al., 2018), we identify and remove turns related to information or opinion requests, and randomly selected 200 examples for human evaluation.
3 Experiments
3.1 Setup
Dataset We use the Wizard of Internet (Wol) (Komeili et al., 2022) dataset for our experi- ments. Wol is a human- dialog corpus for knowledge-poweree conversations, with one of the speakers having internet access to gather informa- tion for generating responses.
Human Evaluation We conducted a human
study with four experienced NLP students to evalu- ate the quality of generated search queries and re- sponses. Queries were assessed based on relevance, specificity, usefulness, and potential to maintain user engagement in the dialog. Responses were evaluated for engagement, coherence, and informa- tiveness. Detailed guidelines are in the appendix.
Models and Baselines
The topic-tracker is based Flan-T5 large (770M) (Chung al., 2022) while the commonsense response generation uses the 3B version of Cosmo (Kim et al., 2022a). The query generator is also based on the Flan-T5 large model. We compare our query generation approach primar- ily against Blender Bot 3 (Shuster et al., 2022b), a state of the art open- pomm-n conversational agent. We also compare against a version of our approach that does not incorporate the Cosmo response for query generation, termed Flan T5 w/o Cosmo.
Automatic Evaluation Recent studies, like G- EVAL (Liu et al., 2023) and GPTScore (Fu et al., 2023), show that LLMS such as GPT-4 can effec- tively evaluate natural language generations and align well with human assessments. Therefore, we utilize GPT-4 for automatic evaluation, prompting* to provide an overall score (ranging from 1-10) for search queries and final responses. As seen in $3.3, our human study results corroborate GPT-4 evaluations. Additionally, (Hedayatnia et al., 2022) trained on the Alexa Prize Socialbot Grand Challenge (Johnston al., 2023) response selection data (Ram et al., 2018) for re- sponse evaluation.
Finetuming with ChatGPT Annotations Our approach uses an instruction-tume Flan T5 model in zero-shot setting for topic tracking and query generation. To improve performance, we separately finetune the topic-tracker and query generator us- ing ChatGPT annotations (the same prompt Flan T5 is used to obtain silver labels from ChatGPT). To create finetuning data, we choose turns corre- sponding to internet search from the Wol training
ranker model
We use the Ds-marcc--MiniM--L model. "Detailed GPT-4 prompts are in the appendix.

---

Search Query
Final Response
Query Generation Approach
Human
Automatic GPT-4
Human
Automatic
Rel.
Spe.
Use.
Int.
Eng.
Info.
Coh.
Ranker
GPT44
No Query
2.71
1.87
3.11
78.9
66.2
Blender Bot
3.13
2.29
2.61
2.28
35.1
2.85
3.19
2.88
75.4
68.0
Flan T5 w/o Cosmo
3,38
3,21
3.06
3.02
44.3
3.01
3.27
2.92
76.9
67.6
Ours (Zero-shot)
3.59
3,51
3.39*
3.29
49.9*
3.13
3.35
3.00
78.6
70.6
Ours (Finetuned)
4.16*
4.05*
3,98*
3.91
72.2*
3.29
3.31
3.10
80.7
72.1
ChatGPT
4.51
4.49
4.48
4.45
80.7
Table I: Evaluation of different query generation approaches on the Wol dataset, based on the quality of search queries and final responses. For query generation (left), Finetuned refers to leveraging dataset dialogs, while zero- shor corresponds to instruction-tuned. For response generation (right), responses from ChatGPT are conditioned on internet search results obtained using the corresponding queries. The acronyms for human evaluation are: Relevance, Specificity, Usefulness, Interestingness, Engagement, Informativeness, and Coherence. * corresponds statistical significance ([z 3.3 and p < 0.05).
3.3 Results
(Large):
Flan (XXL)
WVO Cosmo
Quality of generated search query Table (left) shows the results of human and automatic evalu- ation of search queries. Mainly, notice that instruction-tuned models outperform Blender Bot 3 significantly, and using Cosmo's commonsense response as directive for guiding query genera- tion with Flan T5 shows consistent improvements. Lastly, substantial enhancements in query quality are observed upon fine-tuning the zero-shot sys- tem with ChatGPT annotations. By computing the Spearman correlation between automatic metrics (GPT-4) and overall score for human evaluations (average of four aspect ratings), we found strong correlation (0.674) between the two measures.
Flan T5 (Large): 33.8
Flan (XXL) 66.2
With Cosmoo
W:O Cosmur 46.5
With Conmoo 53,5
(Large)
WXO Cosmo: 32.5
With Cosimox 67.5
(XXL.)
Figure 3: Examining the impact a larger model for query generation, evaluated by GPT-4's preference.
Relevant
Specific
Overall
W/O Topic Tracker
43.0%
39.5%
39.0%
With Topic Tracker
57.0%
60.5%
61.0%
Table Aspect-wise vs overall relative preference scores, based on GPT-4 evaluation, for search queries generated with and without the topic tracking compo- nent in our framework.
Quality of final responses Table (right) shows results for evaluation of the generated responses. We see that directly generating a response from ChatGPT without internet search can still lead to very coherent response, but is less engaging and very uninformative. Our proposed query genera- tion framework leads consistent improvements across all aspects of the final response, particularly with high engagement scores. Notably, boosting engagement, or the likelihood of continued human- bot interaction, is crucial in passive conversations.
influences the quality of generated queries both with and without the commonsense directive. Fig- ure reveals that incorporating the commonsense directive (a) significantly enhances query quality as model size increases, and (b) leads to greater improvement for larger models (67.5% for XXL vs 53.5% for Large). Hence, a more robust instruction- tuned model effectively leverages the common- sense directive in generating better queries.
Benefit of Topic Tracking Within our frame- work, topic tracking serves to (a) maintain coher- ence between the generated query and the most recent discussion subject, and (b) assist in shaping Cosmo's 's situation narrative (refer to prompts in fig- ure 2). Here, we study the benefit of topic tracking by removing it from the Cosmo and query gener- ator inputs and evaluating the final query quality. Using GPT-4 for automatic preference evaluation, we compare queries generated with and without the
3.4 Analysis
Instruction-Follovinn Capability We study the impact of the query generator's instruction- following capability on utilizing the commonsense directive (i.e., cosmo output) to generate better queries. Using GPT-4 preference evaluation, explore how increasing the query generator's size
*As per Chung et al. (2022), larger instruction- tuned mod- els usually have hetter instruction-ollooiing capability.

---

topic tracker. Table 2 shows results from the study, with GPT-4 finding queries generated by involving topic tracking better, particularly for relevance and specificity.
great support, guidance, and help. We are also grateful the anonymous reviewers, the Blender NLP group and CharmBana team members for their valuable feedback and comments.
Error Categorization We examined 50 low- scoring examples from the human evaluation of search queries produced by the zero-shot approach. The main error categories were: (i) Incorrect Topic (31.4%) topic tracker failed to identify the cur- rent discussion subject, (ii) Trivial Query (29.4%) query was obvious already answered in the conversation history, (iii) Query Instruction Mis- match (23.5%) query generator misunderstands instructions, and outputs conversational questions instead, and (iv) other irrelevant queries (15.7%). After finetuming with ChatGPT annotations, 70.6% of the queries significantly improved, while the rest maintained a more or less similar quality level. The breakdown of examples that continue to score low after finetuning is as follows: 19% result from Incorrect Topic, 41 % from Trivial Queries, 5% from Query Instruction Mismatch, and 35% from other unrelated queries. Notably, finetuning effec- tively reduces 67% of Trivial Query errors, 75% Incorrect Topic errors, and over 90% of Query Instruction Mismatch errors. This suggests that finetuning enhances topic tracking capabilities (re- sulting in fewer Incorrect Topic errors) and ensures better adherence to search query generation instruc- tions (leading to a decrease in Query Instruction Mismatch errors).
Limitations
We expect the following limitations for our ap- proach:
Focusing only on turns involving search: Our methodology primarily targets the gener- ation of search queries, hence we chose only those turns from the Wol dataset that con- tained annotated search queries. A more prag- matic approach would also require search decision module to determine when it is es- sential to seek external information.
Assuming topic continuity in discussion: Our method assumes continuous presence of discussion topic. Nevertheless, situations like topic shifts cause temporary absence of topic, which our approach does con- sider. For instance, when the human suggests, "Let's discuss something else," there no cur- rent topic for the discussion.
Ethical Considerations
We raise the following ethical concerns from lever- aging internet search for open-domain dialog:
Toxicity of retrieved content: There is need for a toxicity filter or content moderation system to ensure that the retrieved content is safe, non-offensive, and free from any form of hate speech, discrimination, or harassment.
4 Conclusion and Future Work
We introduce novel framework that leverages commonsense to enhance the generation of search queries in internet-powered dialog. Our results show that incorporating commonsense-biig based di- rective yields search queries with improved rele- vance, specificity and appeal, fostering user engage- ment in otherwise passive conversations. Future work will use more intricate social narratives by incorporating user preferences from past conver- sations, to align commonsense directives towards individual interests.
Privacy concerns: Using Internet search for dialog systems can raise privacy concerns. as users might be discussing or sharing per- sonal or sensitive information their conver- sations. It is crucial to implement proper data anonymization and encryption techniques to protect users' privacy.
Reliability and credibility of sources: Dia- log systems must be cautious when referring information from the Internet, as the con- tent may not always be accurate or reliable. Verifying the credibility of sources and cross- referencing is essential to ensure that the in- formation provided by the system is reliable and trustworthy.
A Acknowledgement
We would like to thank Amazon Inc. for providing grant partially supporting the work of the team. We thank the Amazon Alexa Prize team for their
'Qualitative examples are provided in the appendix

---

References
open domain dialog: grand challenge. In Alexa Prize SocialBot Grand Challenge 5 Proceedings.
The fifth alexa prize socialbot
Leonard Adolphs, Kurt Shuster, Jack Urbanek, Arthur
Szlam, and Jason Weston, 2022. Reason first, then respond: Modular generation for knowledge-infien infused dialogue. Findings of the Association for Computa- rional Linguistics: EMNLP 2022.
Chandra Khatri, Rahul Goel, Behnam Hedayatnia, An- geliki Metallinou, Raefer Gabriel, and Arindam Man- dal. 2018. Contextual topic modeling for dialogue systems. In SLT2018.
Mingzhu Cai. Siqi Bao, Xin Tian, Huang Hc, Fan Wang. and Hua Wu. 2022. Query enhanced knowledge- intensive conversation via unsupervised joint model- ing. arXiv preprint arXiv:2212.09588.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, et al. 2022a. Soda: Million-scale dialogue distillation with so- cial commonsense contextualization. arXiv preprint arXiv:2212.10465
Maximillian Chen, Alexandros Papangelis,, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkami- Tur. 2023. Places: Prompting language models for social conversation synthesis. arXiv preprint arXiv:2302.0 032669....
Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and Maarten Sap. 2022b. Prosocialdialog: prosocial backbone for conversational agents. arXiv preprint arXiv:2205.1268
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. language models.
2022. Scaling instruction-finettee arXiv preprini arXiv: 10. 11416.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-augmented dialogue generation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8460 8478.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan. Michael Auli, and Jason Weston. Wizard of wikipedia: mhowledge-poweree conversational agents. In International Conference on Learning Representations.
Tuan M. Lai, Giuseppe Castellucci. Saar Kuzi, Heng Ji, and Oleg Rokhlenko. 2023a. External knowledge acquisition for end-to-end document-oriente dialog systems. In Proc. The 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics (EACL22023..
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint
Fenfei Guo, Angeliki Metallinou, Chandra Khatri, Anirudh Raju, Anu Venkatesh, and Ashwin Ram. 2018. Topic-based evaluation for conversational bots. arXiv preprint arxivv1801.0366
Tuan M Lai, Giuseppe Castellucci, Saar Kuzi, Heng Ji, and Oleg Rokhlenko. 2023b. External knowledge acquisition for end-to-end document-oriente dialog systems.
Sha Li, Mahdi Namazifar, Di Jin, Mohit Bansal, Heng Ji, Yang Liu, and Dilck Hakkani-Tur. 2022. Enhancing knowledge selection for grounded dialogues via docu- ment semantic graphs. In Proc. The 2022 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Tech- nologies (NAACL-HLT2022),
Amelia Hardy, Ashwin Paranjape, and Christopher Man- ning. 2021. Efffettive social chathot strategies for increasing user initiative. In Proceedings of the 22nd Annual Meeting of the Special Interest Group Dis- course and Dialogue, pages 99-110, Singapore and Online. Association for Computational Linguistics.
Behnam Hedayatnia, Di Jin, Yang Liu, and Dilek Hakkani-Tüir 2022. systematic evaluation of re- sponse selection for open domain dialogue. In SIG- DIAL 2122
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang 7hu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human align- ment, arXiv preprint XrXiv:2303.166
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text de- generation. In International Conference on Learning Representations.
Chris Moore. 2006. The development of commonsense psychology.
Takayuki Nakata, Shinichi Ando, and Akitoshi Oku- mura. 2002. Topic detection based on dialogue his- tory. In COLING 2002: The 19th International Con- ference on Computational Linguistics.
Michael Johnston, Cris Flagg, Anna Gottardi, Sattvik Sahai, Yao Lu, Samyuth Sagi, Luke Dai, Prasoon Goyal, Behnam Hedayatnia, Lucy Hu, Di Jin, Patrick Lange, Shaohua Liu, Sijia Liu, Daniel Pressel, Hangjie Shi. Zhejia Yang, Chao Zhang, Desheng Zhang, Leslie Ball, Kate Bland, Shui Hu, Osman Ipek, James Jeun, Heather Rocker, Lavina Vaz, Ak- shaya Iyengar, Yang Liu, Arindam Mandal, Dilek Hakkaani-Tis and Reza Ghanadan. 2023. Advancing
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744

---

Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, al. 2018. Conversational ai: The science behind the alexa prize. arXiv preprint 1001::/800003000
knowledge for response generation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1237-1252.
Maarten Sap. Ronan Le Bras, Emily Allaway. Chan- dra Bhagavatula, Nicholas Lourie. Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas machine commonsense for then reasoning. In Proceedings of the AAAI con- ference artificial intelligence, volume 33, pages 3027-3035
Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason We- 2022a. Language models that seek for knowledge: Modular search & generation di- alogue and prompt completion. arXiv preprint
13224
Kurt Shuster, Jing Xu. Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, et al. 2022b. Blenderbot 3: deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.031/ 0318.8388.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5:
An open multilingual graph of gen- eral knowledge. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 31.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M Dai, and Quoe V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXhv:2109.01653
Jason D Williams, Antoine Raux, Deepak Ramachan- dran, and Alan W Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference, pages 404-413.
Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. 2022. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. arXiv preprint 2xXxv22088.03277
Hamed Zamani, Johanne R Trippas. Jeff Dalton, and Filip Radlinski. 2022. Conversational information seeking, arXiv preprint arXiv:2201.088 088808.
Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayat- nia. Seokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, Dilek Hakkani-Tur. 2021. Commonsense- focused dialogues for response generation: An em- pirical study. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 121-132.
Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayatnia, Scokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, and Dilek Hakkani-Tur. 2022. Think before you speak: Explicitly generating implicit commonsense

---

A Experimental Setup
ChatGPT Response Generation
A.1 Wizard of Internet Dataset
You are given a dialog between a bot and user:
The Wizard of Internet (Wol) dataset (Komeili et al.. 2022) comprises vast collection of external knowledge-based conversations, allowing agents to leverage internet search to obtain relevant informa- tion. We utilize a training subset of Wol consisting of 6,029 conversations with 29,371 turns. Upon transforming the dataset, models are employed to sequentially generate three items for each bot ut- terance with human annotated search queries: the ChatGPT topic tracking output, commonsense di- rective from Cosmo, and the ChatGPT search query. From this, 20k instances are sampled to form the finetuning data.
Bot:
User:..
Bot:
Generate single sentence response from the bot that incorporates the most relevant part from the content below.
Content:
<Seacch Content>
Bot:
Figure 4: ChatGPT prompt for generating the bot re- sponse by incorporating search content.
Dialog Context
Bot: Seventeen always does great things for promotions on social media!
User: Yes they do, Their music the best, Their dance chorography even better!
A.2 Models and Baselines
Gold Bot responze: They add un ennotional lone to the chorugraphy. which is awesome.
Flan T5 (Chung et al., 2022) We employ the Flan T5 (Large, 770M) as the model for entity tracking and query generation. Given the context, we incorporate the prompt depicted in Figure 2 into the Flan T5 input using the entity and query created by ChatGPT as the ground truth for finetuning.
Direct ChatGPT
Response: Glad to bear that you enjoy both their music and dance perfor- mancvs!
Blender Bot
Query: seventoon
Response: Yes, Seventeen's name reflacts the unique way they strac- as group 13 members divided tinee units.
without Cusmo (Zero-shot)
Query: what the song seventeen"
Cosmo (Kim et al., 2022a) We use the 3B variant Cosmo model as our social commonsense-bel based response generator. The model has been trained on 1.5 million synthetically generated social dialogs from SODA (Kim et al., 2022a). Cosmo has been shown to be capable of managing conversations across diverse dialog situations.
Respouse: Have you voted for your favorite Seventeen song the list ranking the best all time?
Flan TS with Cosano (Zero-boot)
Query: What is Seventeen's most popular song?
Reosose:: Have you vored your favorite Seventeen song yer? the link to make your voice heard in the ranking of their greatest hits.
Flan TS with Cosmoo (ChatGPTT Finetuned)
Query: "What are some of Seventeen's 's's most popular social modia promo- tions?"
Response: It's great hear that you appreciate Seventeen's is social media presence, have you seen any of our lavorite campuign examples from 2022 such as KPC's #I #UnboringMorrnnng or Reddic's 's Super Bowl Awareness Campuign?
(Shuster et al., 2022b) Blender Bot (BB3) is a unified guage model designed for modular dialogue sys- tems, which appends instruction tokens to the gen- erated text. BB3 employs R2C2 (Shuster et al., 2022b) as the language model to create queries. consisting of a 2.7 pallimer Transformer architecture with 22 encoder and 22 decoder layers, pretrained on pushshift.io Reddit and RoBERTa + CC100en data (Shuster et al., 2022a). Furthermore, BB3 finetunes R2C2 on a variety of query datasets, such as the Wizard of Internet and Feedback on Interactive Talk & Search (FITS) (Xu et al., 2022).
Blender Bot 3
lan-
Dialog Context
Bot: I have heard that carvots probubly originated Persia.
User: didn't know they originated Persia, have always hated them didn't where they from.
Gold Not resposee: What your favorite skyserapor to visit?
Direct ChatGPT
Response: That's interesting to know, even if you don't like them.
Blender Bet
Query: carrots
Response: Despite dislike for them, camros are actually highly nutritious beneficial rooe vegetable.
without Cusao (Zoro-hot)
Query: where did origin carrots come from?
Resposse: Yes, carrots first cultivated East and were hrought to Fumoe by the Arabs in the 10th century. Flan TS with Cosmo (Zero-shot)
of Asia and the Middle
A.3 Qualitative Examples
Query: What are the different theories about the origin of curross? wild
Here, we show some qualitative examples for the scarch queries generated by different approaches, and the corresponding responses generated by in- corporating results obtained using the query. Table 3 shows an example for each error type.
Resposse:
single origin in Central Asia, ancestors probablyy originated in Persia and were selecrively brad over the centuries produce familiar garden vegetable.
with Cosmo (ChatGPT Finefuned)
Query:
some
where carrots originited Persia!
Response: Yes, camois actually have single origin in Central Asia and their wild ancestors probably originated in Persia, which is still the centre of diversity for the wild carrot.

---

Ervor Type (Ratio)
Examples
Context
interesting swedish User: Yeah they have quite a few alhoms think.
Bor: yeah they have studio albums!
User: That's awesome. I'd love to find other bands that sound like them.
Bor: Wormrot metal band that popular
Entity
sure chock our, going to shows, just Cosmo Response
Thanks,
Incorrect Topic
live
always great experience.
(31.4%%
Zero-shot TS with Cosmo
Entiy:: Rock band
Query: What
Wommrof!
Problem: Entily tracker too general and does not track topic transition
T5
Enity: Wormrot (metal band)
Query: What
notable live performances Wormmot?
Context
Bor: Have you been hiking lately?
User: We have been espocially with all the current ismues, being outsiiee is only
AlTrrallPro newsletter? They some great suggestions to User: Yes have actually been that newsletter for years, they also have an amazing app that shows you what around
you sign up
hike
Query Trivial
Casmo Response
Bor: That sounds like great app! would love
sometime.
(29,4%)
Zero-shol with Cosmo
Enilyy: Hilingg
Query: What AllTrailsPro newsletter ahout?
Problen: The generaleed was already answered previously so it's not interesting
Flactuned '55 with Cosmno
Enity: Hiking
Query: What
popular hiking trails the AlTTrailPPr nowsletter!
Context
love Federer Nadal when they paired up and played.
User: Yes that was great. Are you interested in becoming tennis champion?
requires
physical
User: It docs. have been training Miami for years now. How long have you played?
Bor: have had lessons from Miani Tennis instructors and played for two years.
Query
User: That fantastic. have played my whole life. My dream is to one day compete in Wimbledon
Instruction
Cosao Response
Mismatch
amazing. hope one day compete
(23.5%
Zero-shot TS with Cosmo
Enity: Tennis
Query: How long have you been playing tennis?
Problem: The query asking question
Fimelumed IS with Cosmo
Enidy:: Tannis
Query:
instructors
Context
Bor: Hello, welcome to Alexa social bot. What you want to chat?
User: I like listening to NE because like rap, and appreciate the vulnerability in lyrics
Cosmo Response
Bor: I'm glad you enjoy bis music' (hink that
listen and artists who
Query Others
tyries.
Zero-shot T$ with Cosmo
(15.7%)
Listen
Query: What genre of music does NF like?
Problem: Query generated is factually wrong: NF singer, listener.
Finetumed T5 with Cosmo
Enityy: NY
Query: are
most lyrics?
Table 3: Qualitative examples of errors types in the query generation.

---

Human Evaluation Guidelines
Evaluation of Generated Query (200 examples)
Instructions
You will be given short human-bot conversation (ending with user utterance) and a bot-generated search query gather information for the next bot response. You need first read the conversation and then evaluate the search query based the following aspects described next.
Current search systems can handle phrase-based and natural language queries. Hence, you should evaluate queries irrespective of how well formed they might be as long as they have all the important terms within them. For instance, the phrase-based query "shoes for long distance marathon running" and the natural language query- "What are some good shoes for running long distances in marathons?" can be considered to be similar.
Note:
Relevance
This evaluates for how relevant the search query is to the dialog. Ideally, the search query is expected be about the topic of discussion in the last 2-3 turns. The results from the search query should be expected to keep the discussion coherent and not make it go off topic.
Scoring Rubric:
You will have to rate this on likert scale of 1 to 5:
1: Completely Irrelevant
3: Vaguely Relevant
5: Highly Relevant
Specificity
This evaluates for how specific the query is the relevant topic being discussed.
Queries can become more specific based on what fine-grained aspects of the topic the query explores.
For instance, below is an ordering of search queries based on specificity:

---

Adidas running shoes for marathons > running shoes from Adidas What are some good running shoes? shoes
Scoring Rubric:
You will have to rate this on a likert scale of 5:
1: Very generic
3: Somewhat specific
5: Highly specific
Usefulness
This evaluates how useful the search query is for obtaining information continue the conversation. This is based on the expectation of the usefulness of the search results corresponding to the given query.
Note: If you are unsure whether a query can be useful, you can pass it to google search and see whether the top-3 results can be useful.
Scoring Rubric:
You will have to rate this on likert scale of 1 to 5:
definitely not useful
3: somewhat useful definitely useful
Interestingness
This evaluates whether the query can obtain information that can make the dialog very engaging. Such queries, while still keeping the discussion coherent i.e. being on-topic, have the potential to take the conversation to new and interesting directions.
Note: Interestingness is the highest bar for evaluation. A query that is irrelevant is not expected to be useful. A query that is relevant but very generic can still be somewhat useful (based on quality of ranking). However, query that can provide interesting information is usually expected be at least relevant, useful and somewhat specific.
Scoring Rubric:
You will have to rate this on likert scale of 1 to 5:
very routine uninteresting
3: somewhat interesting
5: definitely interesting

---

Evaluation of Generated Responses (200 examples)
Instructions
You will be given a short human-bot conversation (ending with a user utterance) and the next bot utterance. You need to first read the conversation and then evaluate the bot response on the following aspects described below.
Coherence
This evaluates how well the response continues the dialog. Ideally, a coherent response expected be about the latest topic of discussion and show follow the logic flow of the dialog
Scoring Rubric: You will have to rank the responses according to which one more coherent given the dialog context.
Engaging
A response is engaging has the potential to spark a conversation or further dialogue. Evaluation would from the perspective of how likely is that you would continue conversation with the bot, given this response.
Since the evaluation of level of engaging is based on the quality of the information in the response and not the actual response generator, you should ignore any follow-up questions from the bot that are present in the response. Responses with follow-up questions can appear more engaging than those without follow-up questions, hence we don't consider the follow-up questions to be fair to all query generation approaches.
Note:
Scoring Rubric: You will have to rank the responses according to which one is more engaging.
Informative
This evaluates for whether the bot response is knowledgeable and the main points of the response still relevant to the conversation topic. Informative responses involve the use of evidence, examples and provide enough novel content (something not already in the dialog history) to satisfy the user's curiosity.
Scoring Rubric:
You will have to rank the responses according to which one more informative.

---

GPT-4 Automatic Evaluation Prompt
Query Scoring
evalustion of the performance continue conversation.
We would like
of which
seerch for obsaining relevant information
You will be given short humen-bot conversation (ending with user utterance) and generated search cuery from each of the systems, to gather information for the next bot resporse. You need read the human-bot conversation and Judge the overall quality of each of the search queries based on the relevance, speaifiiity usefulness and interestingness of the search
Humon- Bot Conversation:
Context>
System1 Query: <system 1>
System2 Query: <systom 2or
SystemS
<system So
For each of the system quertes, need provide single overall score the range of 1-10, where higher score indicates better overall performance. You should evaluate queries imespective of how well tormed they might be, long as they have all the important terms within them. Note that Interestingness the that can provide interesting information continuing conversation prefarred, which such queries usually expected to least relevant, useful and somewhat specffi. Please ensure that the order which the queries were presented does not affect judgment.
highest bar for evaluation:
(scores range ONLY)
System1:
System2:
SystemS:
Query Preference
would request
performance systems, which
query
information continue
You will be given short human-bot conversation (ending with user utterance) and generated search query from each of the systems, to gather information for bet response. need humen-bot conversetion judge overall quality each the search besed the relevance, specifiitty usefuiness and interestingness search
Human-Ba
<Context>
System1 Query: ssystem 1>
System2 Query:
<system 2>
You should evaiuate queries irrespcctive how well formed they might be, long they have all the important, terms within them. Note that interestingness, that provide interesting information for continuing conversation preferred, which such usually expected samewhat
highest ber for evaluation:
relevant,
which cueries
affect your judgment.
Evalustion
Which
better? (Just output system1 system2)
Output:
Response Scoring
We would to request your evaluation the performance of systems, each of which generates search query for obtaining relevant information continue a conversation.
given humen-bot conversation (ending utterance) generated query response. You to human-boc conversation and judge quality the search queries based relevance, specificity usefuiness interestingness search query.
systems, gather
Human-Bot Bot Conversation:
Contoxt>>
Bot1 Response:
<Response 1>
Bot2 Response:
<Response 2>
Bot5 Response:
<Response S>
ensure that responses, you
single range 1-10, higher performance. responses presented affect your judgment.
Evaluation (scores
1-10 ONLY)
Bot1:
Bot2:
Bot4:
Bot5:


Reviews:
[{"review_id": "lu48H7cLko", "reviewer": "Reviewer_TgDw", "summary": "", "questions": "It is not clear to me what the \"ChatGPT\" results in query generation indicate (the last line of Table 1). The scores are higher than all other entries (both baselines and proposed systems) - is this intended to be an oracle system of some sort? If so, please indicate accordingly.\n\nI am also curious about the statistical significance. Can you clarify whether the comparison is done with both baseline methods, or just the worse of the two baselines?\n\nI am somewhat puzzled by the marking of statistical significance in the final response section of table 1. Only GPT4-Zero Shot cell is marked as significantly improving over the baselines; but GPT4-Fine Tuned has a higher result, so I would expect it to also be significant. Can you double check the significance markings in that section?\n\nI would like to see the distribution of errors after fine tuning, too - can you include that in section 3.4 at the end of the Error Categorization paragraph? I am curious if any particular class of errors is helped by the fine tuning process.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and comments. Here, we address the reviewer’s concerns and answer the questions they have raised.\n\n**Minor improvements in the final response**: It's important to clarify that our primary focus in this study is on search query generation.  However, producing the final response introduces steps like web search, ranking, and prompting ChatGPT to generate a response using the search results. Due to these additional components, the following errors arise:\n\n- Despite a relevant query, the web search and ranking process can occasionally select irrelevant content.\n- ChatGPT often (a) adjusts excessively for irrelevant content to still create a coherent and relevant final response and (b) does not fully utilize pertinent search content, sometimes missing key elements -  this is evidenced by an average ROUGE-L overlap of only 0.25 between the generated response and the search content.\n\nThus, even when there is a significant difference in the quality of generated queries  (supported by both human and machine evaluations), the differences do diminish in the final response due to errors introduced by the steps mentioned above.\n\n**What do “ChatGPT” query generation numbers correspond to?**: We consider ChatGPT as the state-of-the-art model for search query generation, and show numbers for evaluation of ChatGPT generated queries as a measure of upper-bound for model performance. While these numbers serve as a dynamic standard for upper-bound of model performance, we anticipate even better results with GPT-4, especially considering our experiments utilized gpt-3.5-turbo-0301.\n\n**Clarification of statistical significance results**: To determine statistical significance, we compared the zero-shot version of our model, denoted as \"Ours (Zero-shot)\" in Table 1, with all baseline methods. Meanwhile, the finetuned version of our model, labeled as \"Ours (Finetuned)\" in Table 1, is compared against our zero-shot model. These details will be more explicitly outlined in the final version.\n\n**Error Distribution After Finetuning**: After finetuning, of the examples that continue to score low, the breakdown is as follows: 19% result from *Incorrect Topic*, 41% from *Trivial Queries*, 5% from *Query Instruction Mismatch*, and 35% from other unrelated queries. Notably, finetuning effectively reduces 67% of *Trivial Query* errors, 75% of *Incorrect Topic* errors, and over 90% of *Query Instruction Mismatch* errors. This suggests that finetuning enhances topic tracking capabilities (resulting in fewer *Incorrect Topic* errors) and ensures better adherence to search query generation instructions (leading to a decrease in *Query Instruction Mismatch* errors).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I find the paper generally well written, clear, and easy to follow. The system proposed appears to be effective, based on the experimental results shown in table 1. Results show statistical significance over the baseline for query generation (other than the last line, for which I have a question, below).", "reasons_to_reject": "While results in the Final Response section are promising, the paper appears to overemphasize the achievements; there is little statistically significant improvement in this section (with one question about this included below); the main achievement appears to be in matching the coherence obtained from the no-search configuration (first line) while also matching (or perhaps slightly improving upon) the engagement and informativeness of Flan T5 w/o Cosmo. This is fine but the paper as written claims more, which I find misleading. ", "questions_to_authors": "It is not clear to me what the \"ChatGPT\" results in query generation indicate (the last line of Table 1). The scores are higher than all other entries (both baselines and proposed systems) - is this intended to be an oracle system of some sort? If so, please indicate accordingly.\n\nI am also curious about the statistical significance. Can you clarify whether the comparison is done with both baseline methods, or just the worse of the two baselines?\n\nI am somewhat puzzled by the marking of statistical significance in the final response section of table 1. Only GPT4-Zero Shot cell is marked as significantly improving over the baselines; but GPT4-Fine Tuned has a higher result, so I would expect it to also be significant. Can you double check the significance markings in that section?\n\nI would like to see the distribution of errors after fine tuning, too - can you include that in section 3.4 at the end of the Error Categorization paragraph? I am curious if any particular class of errors is helped by the fine tuning process.", "paper_topic_and_main_contributions": "This paper focuses on the problem of increasing user engagement in the interactions with an open domain dialog system by leveraging common sense reasoning techniques (applied to the conversation content). The main contribution of the paper is the overall system proposed, together with the experimental results demonstrating the effectiveness of the system.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZimvohCd8d", "reviewer": "Reviewer_Vk7b", "summary": "", "questions": "A) What type of decoding strategy and parameters were used in each component of the framework?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We woud like to thank the reviewer for the valuable feedback. Here, we address the reviewer’s concerns.\n\n**More discussion on how incorporating social commonsense helps query generation**: We utilize the Flan T5 without Cosmo system as the main comparison point to demonstrate the advantages of integrating commonsense knowledge. Additionally, in Section 3.4, we discuss how the integration of commonsense knowledge affects both larger and smaller query generation models. The introduction section, from lines 50-57, outlines the rationale for incorporating social commonsense. Further, lines 78-86, along with Figure 1, offer a detailed example illustrating the impact of social commonsense on search query formulation. \n\n\n**Decoding Strategy and Parameters not provided**: For both search query generation and final response generation,we employ the Top-P sampling method with P set at 0.9 and a temperature of 0.7. We set max tokens to 40 and 100 for search query generation and final response generation respectively. For ChatGPT, we use the *gpt-3.5-turbo-0301* version. These details will be added into the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The results achieved are encouraging, showing improvements over the considered baselines.\n2) Tries to solve a practical problem of handling passive user interactions.\n3) The evaluation considered both human-judgments and model-based judgments, and the authors also performed an additional error analysis.", "reasons_to_reject": "1) I would have liked to see a more complete discussion about how incorporating the social commonsense influences the query generation procedure, besides the results of not using Cosmo.\n2) There is some under-specification in the parameters used to generate the answers of the different models, hindering reproducibility of the work.", "questions_to_authors": "A) What type of decoding strategy and parameters were used in each component of the framework?", "paper_topic_and_main_contributions": "This paper tackles the problem of generating queries targeting passive users using social commonsense. \nThe paper contributes with a pipeline composed of 3 main parts: 1) a fine-grained topic tracking method using FLAN-T5 to extract the current topic of the conversation; 2) a commonsense model using Cosmo that receives the topic and generates a latent commonsense directive to guide the query generation procedure; and 3) an instruction-driven query generation method, also using FLAN-T5, prompted to convert the response into a search query to be used in a search engine.\nTo evaluate, the authors use the Wizard of Internet dataset and evaluated with both human evaluation, and automatic evaluation using GPT-4.\nThe results showed that the proposed method was better at generating queries and generating a final response in both types of metrics given the considered baselines. The results also showed that incorporating commonsense using Cosmo led to better performance, and that further fine-tuning the model on ChatGPT answers also helps.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well written and easy to follow.\n\nWhen mentioning ChatGPT, please specify which version you are using and if you used Open AI’s API.\n\nRegarding the instructions given to the annotators, it would probably be good to have them in another format that would allow for easier reading instead of just being appended to the document."}}, {"review_id": "ki2waNTwLv", "reviewer": "Reviewer_XjaA", "summary": "", "questions": "Question A\n\nIn Line 190, can you explain the reason for reranking the top-3 search results? And what is the impact of providing all these search results for response generation?\n\nQuestion B\n\nIn Table 1 (right), why do you omit the evaluations on ChatGPT results?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the valuable feedback and suggestions. Here, we address the reviewer’s concerns and questions.\n\n**Model does not outperform ChatGPT**: Our model may not surpass ChatGPT in performance; however, we would like to highlight that we employ a considerably more compact model, Flan T5 Large with 770M parameters, for generating queries. This is complemented by the 3B parameter COSMO model that supplies commonsense knowledge. In real-world dialog systems where swift response times are crucial, it is advantageous to utilize a smaller model for query generation. Larger models like ChatGPT are generally reserved for the final stages of response creation.\n\n**Zero-shot model with Cosmo seems to be slightly better than finetuned one at informativeness of responses**: When directly evaluated for query generation performance, we see that the finetuned model considerably outperforms the zero-shot model. Yet, when assessed based on the final response quality—derived from search results obtained using the generated queries—the gap between the finetuned and zero-shot models narrows. This discrepancy arises because the final response generation includes web search, ranking, and prompting ChatGPT to generate a response with those search results. Even if the query quality differs, the eventual response might not reflect that due to errors in these stages. For instance, even with a relevant query, the web search and ranking system might occasionally pick unrelated content. Moreover, ChatGPT tends to (a) compensate heavily for unrelated content, ensuring a coherent and relevant final response and (b) sometimes overlooks crucial content from the search, omitting vital elements.Therefore, even though the quality of the final response can provide a rough measure of the search query's effectiveness, it's not devoid of potential biases. We will incorporate this discussion into our final draft.\n\n**Finetuned Flan-T5 without commonsense reasoning not compared**: We thank the reviewer for this excellent suggestion. We will add this comparison, with both human and automatic evaluation, into the final version of the paper.\n\n**Question A**: Each search result is derived from the unprocessed content of a webpage, which might be extensive and filled with unnecessary details. To address this, we segment the webpage into passages and employ a re-ranker to identify the most pertinent section within the page.                                                                                                                   \n\n**Question B**: In Table 1 (left), ChatGPT mainly serves as an upper bound for model performance on search query generation. Given constraints in annotator time and budget, Table 1 (right) focuses on evaluating *ChatGPT (No Query)* as a more insightful baseline for final response, rather than assessing the response generated based on the search results obtained by a ChatGPT-generated query.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Providing an informative response to a passive user utterance can increase the engagingness of the chatbot. This paper first discusses the technique for generating a query when the user intent is unclear.\n\n2. The experiment results are convincing with sufficient human and automatic evaluations. The human evaluation considers both query generation and response generation. The authors provide a good example of query generation human evaluation with detailed guidance.", "reasons_to_reject": "1. Although the idea is good, the queries generated by this framework are still inferior to those generated by ChatGPT. This paper fails to state the benefits of using this framework to generate queries instead of utilizing ChatGPT directly.\n\n2. This paper can be improved by more discussions about the generated query. For example, there is no discussion about why the zero-shot model with Cosmo receives the highest informativeness score in the human evaluation (in Table 1) and is even superior to the finetuned model.\n\n3. The effectiveness of commonsense reasoning for a finetuned Flan-T5 remains unanswered. As stated in Line 178, the authors finetune Flan-T5 with ChatGPT results. Given that ChatGPT can produce queries with the highest evaluation scores, the compared baselines did not include the finetuned Flan-T5 without commonsense reasoning. This resulted in less robust ablation experiments.", "questions_to_authors": "Question A\n\nIn Line 190, can you explain the reason for reranking the top-3 search results? And what is the impact of providing all these search results for response generation?\n\nQuestion B\n\nIn Table 1 (right), why do you omit the evaluations on ChatGPT results?", "paper_topic_and_main_contributions": "This paper focuses on an unexplored conversational query generation problem setting that generates queries with ambiguous user intent. To tackle this problem, this paper leverages social commonsense reasoning to guide query generation. The main contribution of this paper is the query generation framework that consists of a finetuned topic identifier, a fixed commonsense-based response generation (i.e., Cosmo), and a finetuned instruction-based query generation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The coherence score of \"Ours (Finetuned)\" in Table 1 is not the highest, but it is highlighted in bold."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on an unexplored conversational query generation problem setting that generates queries with ambiguous user intent. To tackle this problem, this paper leverages social commonsense reasoning to guide query generation. The main contribution of this paper is the query generation framework that consists of a finetuned topic identifier, a fixed commonsense-based response generation (i.e., Cosmo), and a finetuned instruction-based query generation.

The problem is practical and the results achieved are encouraging. Nevertheless, reviewers have the following concerns:
1. Minor improvements in the final response. While the authors provide their response, I do not think this concern has been fully addressed. From the response, it seems that high-quality queries are not that important. 
2. The paper fails to state the benefits of using the proposed framework to generate queries instead of utilizing ChatGPT directly. Besides, there are no efficiency analyses.
3. Finetuned Flan-T5 without commonsense reasoning not compared.
Decision: Accept-Findings