SlowGuess/NAACL_2021_OCR huggingface dataset:
Figure 3a shows the results of pairwise comparison on the usefulness criteria. We find that our model wins over GAN-Utility by a significant margin with humans preferring our model-generated questions  $77\%$  of the time. Our model also beats BART-baseline  $66\%$  of the time further affirming the importance of using missing schema. Finally, our model beats BART+missinfo model  $61\%$  of the time suggesting that the PPLM-based decoder that uses usefulness classifier is able to produce much more useful questions (RQ3). The annotator agreement statistics are provided in appendix.
Ubuntu Table 6 shows the results of human judgments on the model generations of 150 randomly sampled dialog contexts from the Ubuntu test set. In terms of relevance, we find that the transformer and BART baselines produce less relevant
<sup>6</sup>We do not experiment with the GAN-Utility model (since it requires "answers") and the BART+missinfo+WD model (since usefulness labels are not obtained from humans).
Model	Relevancy	Fluency	MissInfo
GAN-Utility	0.9	0.86	0.81
BART	0.94	0.92	0.77
+ missinfo	0.97	0.92	0.87
+ missinfo + PPLM	0.99	0.93	0.89
Reference	0.96	0.83	0.89
Table 5: Human judgment results (0-1) on 300 randomly sampled descriptions from the Amazon test set
Model	Relevancy	Fluency	MissInfo
Transformer	0.74	0.99	0.99
BART	0.69	0.99	0.96
+ missinfo	0.81	0.95	0.98
+ missinfo + PPLM	0.91	0.83	0.99
Reference	0.85	0.83	0.96
Table 6: Human judgment results (0-1) on 150 randomly sampled dialog contexts from Ubuntu test set
(b)
Figure 3: Results of a pairwise comparison (on usefulness criteria) between our model and baseline generated question on (a) 300 randomly sampled product descriptions from the Amazon test set, (b) 150 randomly sampled dialogs from the Ubuntu test set as judged by humans.
questions. With the addition of missing schema (i.e., BART+missinfo), the questions become more relevant and our proposed model obtains the highest relevance score. The reference obtains slightly a lower relevance score which can possibly be explained by the fact that humans sometimes digress from the topic. Under fluency, interestingly, the transformer and BART baselines obtain high scores. With the addition of missing schema, fluency decreases and the score reduce further with the PPLM model. We suspect that the usefulness classifier trained with a negative sampling strategy (as opposed to human labelled data, as in Amazon) contributes to fluency issues. Under missing information, all models perform well which can be explained by the fact that in Ubuntu, the scope of missing information is much larger (since dialog is much more open-ended) than in Amazon.
Figure 3b shows the results of pairwise comparison on usefulness criteria. We find that humans choose our model-generated questions  $85\%$  of time when compared to either transformer or BART generated questions. When compared to BART+missinfo, our model is selected  $71\%$  of the time, further affirming the importance of using the PPLM-based decoder.
Robustness to input information We analyze how a model is robust toward the amount of information present. To measure the amount of informa
tion, we look for context length (description length for Amazon, dialog context length for Ubuntu) and the size of global schema since these two directly control how much knowledge regarding potential missing information is available to the model. We measure the difference in BLEU score between two groups of data samples where context length/size of global schema is either high or low. Figure 5 shows that our model is the least variant toward the information available hence more robust for the Amazon dataset.[7]
Owing to our modular approach for estimating missing information, we seek to analyze whether a question is really asking about missing information in an automatic fashion. This also allows us to explain the reasoning behind a particular generation as we are able to trace back to the particular missing information that is used to generate the question. We run a YAKE extractor on the generated questions to obtain key-phrases. We calculate the ratio between the number of key-phrases in the output that belong to the original missing schema and the total number of key-phrases present in the output. Table 8 shows that when we use our framework of estimating missing information coupled with BART, both models achieve very high missing information overlap, thus suggesting that we can obtain the reasoning behind a generated question reliably by tracing the missing information overlap, as shown in Table 9.
7Ubuntu follows similar trends; figure in appendix.
Amazon
Category	Binoculars & Scopes
Title	Nikon 7239 Action 7x50 EX Extreme All-Terain Binocular
Description	The Monarch ATB 42mm with dielectric high-reflective Multilayer Prism coating binocular features brighter, sharper colors, crisp and drastically improved low-light performance. A new body style provides unparalleled strength and ruggedness in a package ... {mounting, center focused, (Nikon, works, obj), (Canon, works, obj), digital camera, ...}
Missing Schema
GAN-Utility	price?
BART	How is the focus quality?
BART+missinfo	Is it center focused?
BART+missinfo+PPLM	Is it center focused, or do you have to focus each eye individually?
Ubuntu
Dialog history	User A: I'm having trouble installing nvidia drivers for my geforce 6200, could anyone perhaps assist?
User B: i use the drivers from the website, much better
User A: which drivers? from the website?
User B: I used add/remove software from the menu to install nvidia proprietary drivers {{(driver, update, nsubj), (new version, install, nsubj), (machine, reboot, nsubj), ...}}
Missing schema
Transformer	Did you try booting your machine?
BART	where did you download them from?
BART+missinfo	Can you tell the output after you install them?
BART+missinfo+PPLM	Can you try rebooting from the start and removing the software after installation?
Table 7: Model generations for an example product from Amazon and an example dialog context from Ubuntu.
Model	Amazon	Ubuntu
Retrieval	10.5	6.78
GAN-Utility	73.4	-
Transformer	57.2	45.7
BART	60.3	56.9
+ missinfo	97.3	89.2
+ missinfo + PPLM	98.3	90.1
Reference	99.7	93.7
Table 8: Missing information overlap (in %) between missing schema and output generations
Question length We also observe in Table 9 that baseline models tend to generate short and generic questions as compared to our model that often chooses longer schema key- phrases (e.g. bigrams) to generate a more specific question. We further looked into annotated (for usefulness) questions from the Amazon dataset and we observed that  $70\%$  of questions that were annotated as useful are longer than not-useful questions. The average length of gold useful questions is 10.76 words and 8.21 for not-useful questions. The average length of generated questions for BART, BART+MissInfo and BART+MissInfo+PPLM (ours) are 5.6, 6.2, 12.3 respectively. We also find a similar trend in the Ubuntu dataset as well.
Dynamic expansion of global schema We anticipate that even if we build the global schema from the available offline dataset, it is possible that new entries may appear in a real application. We investigate how our framework responds to the dynamic expansion of global schema. We simulate a scenario where we extend the "Laptop Acces
Figure 4: Average BLEU score difference between classes having longer (>200 (median) words) and shorter descriptions; larger (>200 (median) key-phrases) and shorter global schema for the Amazon dataset. Lower differences indicate more invariance toward the available information.
sories" category in the Amazon dataset, with 100 new products (those that appeared on Amazon.com after the latest entry in the dataset). We obtain key-phrases from their product descriptions and include them in the global schema for the category which amounts to a  $21\%$  change in the existing global schema. For 50 random products in the test set from the same category, we found that in 28 out of 50 cases  $(56\%)$ , the model picked a new schema element that is added later. This indicates that our framework is capable of supporting dynamic changes in the global schema and reflecting them in subsequent generations without retraining from scratch.
Most previous work on question generation focused on generating reading comprehension style
questions i.e., questions that ask about information present in a given text (Duan et al., 2017; Zhang and Bansal, 2019). Later, Rao and Daumé III (2018, 2019) introduced the task of clarification question generation in order to ask questions about missing information in a given context. ClarQ (Kumar and Black, 2020) entails clarification questions in a question answering setup. However, unlike our work, these works still suffer from estimating the most useful missing information.
Recent works on conversational question answering also focused on the aspect of question generation or retrieval (Choi et al., 2018; Aliannejadi et al., 2019). Qi et al. (2020a) especially focused on generating information-seeking questions while Majumder et al. (2020) proposed a question generation task in free-form interview-style conversations. In this work, in addition to improving clarification question generation in a community-QA dataset, we are the first to explore a goal-oriented dialog scenario as well.
Representing context and associated global information in a structure format has been shown to improve performance in generation task (Das et al., 2019; Subramanian et al., 2018; Khashabi et al., 2017) in general and summarization (Fan et al., 2019) and story-generation (Yao et al., 2019) in particular. We also derive inspiration from recent works on information extraction from free-form text (Vedula et al., 2019; Stanovsky et al., 2016) and develop a novel framework to estimate missing information from available natural text contexts.
Finally, for question generation, we use BART (Lewis et al., 2019), that is state-of-the-art for many generation tasks such as summarization, dialog generation etc. Furthermore, inspired from recent works that use controlled language generation during decoding (Ghazvininejad et al., 2017; Holtzman et al., 2018), we use Plug-and-Play-LanguageModel (Dathathri et al., 2019) to tune generations during decoding. While similar approaches for controllable generation (Keskar et al., 2019; See et al., 2019) have been proposed, we extend such efforts to enhance the usefulness of the generated clarification questions.
We propose a model for generating useful clarification questions based on the idea that missing information in a context can be identified by taking a difference between the global and the local view.
We show how we can fine-tune a large-scale pretrained model such as BART on such differences to generate questions about missing information. Further, we show how we can tune these generations to make them more useful using PPLM with a usefulness classifier as its attribute model. Thorough analyses reveal that our framework works across domains, shows robustness towards information availability, and responds to the dynamic change in global knowledge. Although we experiment only with Amazon and Ubuntu datasets, our idea is generalizable to scenarios where it is valuable to identify missing information such as conversational recommendation, or eliciting user preferences in a chit-chat, among others.
Acknowledgements We thank everyone in the Natural Language Processing Group at Microsoft Research, Redmond, with special mention to Yizhe Zhang, Bill Dolan, Chris Brockett, and Matthew Richardson for their critical review of this work. We also thank anonymous reviewers for providing valuable feedback. In addition to this, we want to acknowledge human annotators from Amazon Mechanical Turk for data annotation and human evaluation of our systems. BPM is partly supported by a Qualcomm Innovation Fellowship and NSF Award #1750063. Findings and observations are of the authors only and do not necessarily reflect the views of the funding agencies.
We do not foresee any immediate ethical concerns since we assume that our work will be restricted in domain as compared to free-form language generation. We still cautiously advise any developer who wishes to extend our system for their own use-case (beyond e-commerce, goal-oriented conversations) to be careful about curating a global pool of knowledge for data involving sensitive user information. Finally, since we are finetuning a pretrained generative model, we inherit the general risk of generating biased or toxic language, which should be carefully filtered. In general, we expect users to benefit from our system by reducing ambiguity (when information is presented in a terse fashion, e.g. in a conversation) and improving contextual understanding to enable them to take more informed actions (e.g. making a purchase).
Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In SIGIR. ACM.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Ricardo Campos, Vitor Mangaravite, Arian Pasquali, Alipio Jorge, Celia Nunes, and Adam Jatowt. 2020. *Yake!* keyword extraction from single documents using multiple local features. *Information Sciences*, 509.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quac: Question answering in context. In EMNLP.
Leigh Clark, Nadia Pantidi, Orla Cooney, Philip R. Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, Christine Murad, Cosmin Munteanu, Vincent Wade, and Benjamin R. Cowan. 2019. What makes a good conversation?: Challenges in designing truly conversational agents. In CHI.
Rajarshi Das, Tsendsuren Munkhdalai, Xingdi Yuan, Adam Trischler, and Andrew McCallum. 2019. Building dynamic knowledge graphs from text using machine reading comprehension. In ICLR.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. In ICLR.
Marco De Boni and Suresh Manandhar. 2003. An analysis of clarification dialogue for question answering. In NAACL-HLT.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In EMNLP.
Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale seq2seq models to multi-document inputs. In EMNLP-IJCNLP.
Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an interactive poetry generation system. In ACL, System Demonstrations.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In ACL.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858.
Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2017. Learning what is essential in questions. In CoNLL.
Vaibhav Kumar and Alan W. Black. 2020. Clarq: A large-scale and diverse dataset for clarification question generation. In ACL.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In *NAACL-HLT*, pages 110-119.
Ryan Lowe, Nissan Pow, Iulian Vlad Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In SIGDial.
Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian J. McAuley. 2020. Interview: Large-scale modeling of media dialog with discourse patterns and knowledge grounding. In EMNLP.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In SIGIR.
Julian McAuley and Alex Yang. 2016. Addressing complex and subjective product-related queries with customer reviews. In WWW.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318.
Peng Qi, Yuhao Zhang, and Christopher D. Manning. 2020a. Stay hungry, stay focused: Generating informative and specific questions in information-seeking conversations. In EMNLP.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020b. Stanza: A python natural language processing toolkit for many human languages. In ACL Demo.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Tech report.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In ACL.
Sudha Rao and Hal Daumé III. 2019. Answer-based Adversarial Training for Generating Clarification Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 143-155, Minneapolis, Minnesota. Association for Computational Linguistics.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In *NAACL-HLT*.
Gabriel Stanovsky, Jessica Ficler, Ido Dagan, and Yoav Goldberg. 2016. Getting more out of syntax with props. CoRR, abs/1603.01648.
Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg. 2014. Towards natural clarification questions in dialogue systems. In AISB symposium on questions, discourse and dialogue, volume 20.
Sandeep Subramanian, Tong Wang, Xingdi Yuan, Saizheng Zhang, Adam Trischler, and Yoshua Bengio. 2018. Neural models for key phrase extraction and question generation. In ACL MRQA.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeuRIPS.
Nikhita Vedula, Nedim Lipka, Pranav Maneriker, and Srinivasan Parthasarathy. 2019. Towards open intent discovery for conversational text. CoRR, abs/1904.08524.
Lili Yao, Nanyun Peng, Ralph M. Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In AAAI.
Shiyue Zhang and Mohit Bansal. 2019. Addressing semantic drift in question generation for semi-supervised question answering. CoRR, abs/1909.06356.
A Setup and Data
Schema While creating the global schema, we use word embedding $^{8}$  based similarity to perform hierarchical clustering of key-phrases $^{9}$  and group together key-phrases that have cosine similarity greater than a threshold, a hyperparameter set to
0.6. Finally, we order all key-phrase clusters by their frequencies and retain only the top  $60\%$  thus removing low-frequency schema elements.
While creating the missing schema, we do the match based on semantic similarity of key- phrases (even for a tuple we only look at key-phrase similarity) and we consider two key- phrases to be matched if the cosine similarity is above a threshold, that we set as 0.8 since we want to match only highly similar key- phrases.
GloVe embeddings on Amazon and Ubuntu datasets We train 200 dimensional GloVE embeddings on the vocabulary of both Amazon and Ubuntu dataset separately. We set a vocabulary frequency threshold at 50, i.e. we only obtain embeddings for words that appear at least 50 times in the whole corpus.
Datasets Downloadable links to each datasets are provided here: Amazon<sup>10</sup>, Ubuntu<sup>11</sup>.
Collecting human annotations for usefulness scores For the Amazon dataset, Rao and Daumé III (2019) define the usefulness of a question as the degree to which the answer provided by the question would be useful to potential buyers or current users of the product. We use the annotation scheme defined in Rao and Daumé III (2019) to annotate a set of 5000 questions from the amazon dataset. $^{12}$  We show annotators product details (title, category, and description) and a question asked about that product and ask them to give it a usefulness score between 0 to 5. $^{13}$  Each question was annotated by three annotators. We average the three scores to get a single usefulness score per question. We use the YAKE extractor to extract the schema elements for each question and assign the usefulness score of the question to each of its schema elements.
Since our aim is to assign a usefulness score to each missing element of each product in our dataset, we train a usefulness classifier on the manually annotated schema elements. Although our usefulness score is a real value between 0 and 5, we find that training a regression model gives us poor performance. Hence we convert the real value
<sup>8</sup>We train GLoVE embeddings separately on Amazon and Ubuntu  
<sup>9</sup>For triples, we use only their key-phrase to define similarity.
10 https://nijianmo.github.io/amazon/index.html
<sup>11</sup>https://github.com/rkadlec/ubuntu-ranking-dataset-creator
12We use the Amazon Mechanical Turk platform.
<sup>13</sup>Refer Rao and Daumé III (2019) for an exact description of each score.
into a binary value by threshold at 3 (i.e. values below 3 are assigned label 0 and values above 3 are assigned label 1).
Usefulness classification with negative sampling Collecting usefulness annotation on questions, as we do for the Amazon dataset, can be expensive and may not always be possible in different scenarios. Therefore, for the Ubuntu dataset, we experiment with a classifier where instead of using human annotations are true labels, we use a negative sampling strategy. Specifically, we assume that all (context, question) pairs in the Ubuntu dataset can be labelled 1 and any (context, random question) can be labelled 0. We sample a set of 2500 questions from the Ubuntu dataset and them label 1 and sample an equivalent number of negative samples and assign them label 0.
B Training
BART and PPLM For question generation model, we use BART-base (6 encoder layers, 6 decoder layers, 117M parameters)  $^{14}$ . For PPLM usefulness classifier, we use a bag-of-word model, that uses the pretrained subword embedding layers from BART-base model. We average the subword embeddings to obtain a sentence representation and a usefulness score is predicted via a linear layer projection with softmax. We use the PPLM code from official repository  $^{15}$ .
Each BART variant converged in 3 epochs on an average with batch size 4 in a TITAN X (Pascal) GPU that took 12 hours in total. While training, we only observe perplexity on the validation set to employ an early-stopping criteria.
Usefulness Classifier for BART+MissInfo+WD
We train an SVM (support vector machines) classifier on this data. We use word embeddings as our features by training a 200 dimensional GLoVE model trained on individual dataset. We average the word embeddings of all words in a schema element and use it as a feature. We obtain an F1-score of  $80.6\%$  on a held out test set.[16] We use this classifier to predict a usefulness score for each missing schema element of each instances from a class for each dataset, which was required for the BART+MissInfo+WD model.