SlowGuess/NAACL_2021_OCR huggingface dataset:
Bodhisattwa Prasad Majumder†\* Sudha Rao
Michel Galley\* Julian McAuley
$\clubsuit$ Department of Computer Science and Engineering, UC San Diego
{bmajumde, jmcauley}@eng.ucsd.edu
{sudha. rao, mgalley}@microsoft.com
The ability to generate clarification questions i.e., questions that identify useful missing information in a given context, is important in reducing ambiguity. Humans use previous experience with similar contexts to form a global view and compare it to the given context to ascertain what is missing and what is useful in the context. Inspired by this, we propose a model for clarification question generation where we first identify what is missing by taking a difference between the global and the local view and then train a model to identify what is useful and generate a question about it. Our model outperforms several baselines as judged by both automatic metrics and humans.
An important but under-explored aspect of text understanding is the identification of missing information in a given context i.e., information that is essential to accomplish an underlying goal but is currently missing from the text. Identifying such missing information can help to reduce ambiguity in a given context which can aid machine learning models in prediction and generation (De Boni and Manandhar, 2003; Stoyanchev et al., 2014). Rao and Daumé III (2018, 2019) recently proposed the task of clarification question generation as a way to identify such missing information in context. They propose a model for this task which while successful at generating fluent and relevant questions, still falls short in terms of usefulness and identifying missing information. With the advent of large-scale pretrained generative models (Radford et al., 2019; Lewis et al., 2019; Raffel et al., 2019), generating fluent and coherent text is within reach. However, generating clarification questions requires going beyond fluency and relevance. Doing so requires understanding what is missing, which if included could be useful to the consumer of the information.
TITLE: Sony 18x Optical Zoom 330x Digital Zoom Hi8 Camcorder
DESC: Sony Hi-8mm Handycam Vision camcorder 330X digital zoom, Nightshot(TM) Infrared 0 lux system, Special Effects, 2.5" SwivelScreen color LCD and 16:9 recording mode, Laserlink connection. Image Stabilization, remote, built in video light.
QUESTION: Can I manually control the video quality?
Table 1: Product description from amazon.com paired with a clarification question generated by our model.
Humans are naturally good at identifying missing information in a given context. They possibly make use of global knowledge i.e., recollecting previous similar contexts and comparing them to the current one to ascertain what information is missing and if added would be the most useful. Inspired by this, we propose a two-stage framework for the task of clarification question generation. Our model hinges on the concept of a "schema" which we define as the key pieces of information in a text. In the first stage, we find what's missing by taking a difference between the global knowledge's schema and schema of the local context (§3.1). In the second stage we feed this missing schema to a fine-tuned BART (Lewis et al., 2019) model to generate a question which is further made more useful using PPLM (Dathathri et al., 2019) (§3.2).<sup>1</sup>
We test our proposed model on two scenarios (§2): community-QA, where the context is a product description from amazon.com (McAuley and Yang, 2016) (see e.g. Table 1); and dialog where the context is a dialog history from the Ubuntu Chat forum (Lowe et al., 2015). We compare our model to several baselines (§4.2) and evaluate outputs using both automatic metrics and human evaluation to show that our model significantly outperforms baselines in generating useful questions that identify missing information in a given context (§4.4).
† Work done during an internship at Microsoft Research
The code is available at https://github.com/microsoft/clarification-qgen-globalinfo
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4300-4312 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Test-time behaviour of our proposed model for useful clarification question generation based on missing information in a Community-QA (amazon.com) setup. 1. We obtain a local schema from the available context for a product: description and previously asked questions. 2. We obtain the global schema of the category of the product. 3. We estimate the missing schema that is likely to guide clarification question generation. 4. A BART model fine-tuned on (missing schema, question) pairs to generate a question ("Is it true leather?"). 5. A PPLM model with usefulness classifier as its attribute model further tunes the generated question to make it more useful ("Is there room to also carry a cooling pad?").
Furthermore, our analysis reveals reasoning behind generated questions as well as robustness of our model to available contextual information. (§5).
2 Problem Setup and Scenarios
Rao and Daumé III (2018) define the task of clarification question generation as: given a context, generate a question that identifies missing information in the context. We consider two scenarios:
Community-QA Community-driven question-answering has become a common venue for crowdsourcing answers. These forums often have some initial context on which people ask clarification questions. We consider the Amazon question-answer dataset (McAuley and Yang, 2016) where context is a product description and the task is to generate a clarification question that helps a potential buyer better understand the product.
Goal Oriented Dialog With the advent of high quality speech recognition and text generation systems, we are increasingly using dialog as a mode to interact with devices (Clark et al., 2019). However, these dialog systems still struggle when faced with ambiguity and could greatly benefit from having the ability to ask clarification questions. We explore such a goal-oriented dialog scenario using the Ubuntu Dialog Corpus (Lowe et al., 2015) consisting of dialogs between a person facing a technical issue and another person helping them re
solve the issue. Given a context i.e a dialog history, the task is to generate a clarification question that would aid the resolution of the technical issue.
Figure 1 depicts our approach at a high level. We propose a two-stage approach for the task of clarification question generation. In the first stage, we identify the missing information in a given context. For this, we first group together all similar contexts in our data² to form the global schema for each high-level class. Next, we extract the schema of the given context to form the local schema. Finally, we take a difference between the local schema and the global schema (of the class to which the context belongs) to identify the missing schema for the given context. In the second stage, we train a model to generate a question about the most useful information in the missing schema. For this, we fine-tune a BART model (Lewis et al., 2019) on (missing schema, question) pairs and at test time, we use PPLM (Dathathri et al., 2019) with a usefulness classifier as the attribute model to generate a useful question about missing information.
3.1 Identifying Missing Information
Schema Definition Motivated by (Khashabi et al., 2017) who use essential terms from a
See §4.1 for details to combine data splits
question to improve performance of a Question-Answering system, we see the need of identifying important elements in a context to ask a better question. We define schema of sentence  $s$  as set consisting of one or more triples of the form (key-phrase, verb, relation) and/or one or more key-phrases.
$$
s c h e m a _ {s} = \{e l e m e n t \}; w h e r e
$$
$$
\begin{array}{l} e l e m e n t \in \{(\text {k e y - p h r a s e}, \text {v e r b , r e l a t i o n}), \tag {1} \\ \left. \begin{array}{l} \text {k e y - p h r a s e} \end{array} \right\} \\ \end{array}
$$
Schema Extraction Our goal is to extract a schema from a given context. We consider (key-phrase, action verb, relation) as the basic element of our schema. Such triples have been found to be representative of key information in previous work (Vedula et al., 2019). Given a sentence from the context, we first extract bigram and unigram key-phrases using YAKE (Yet-Another-Keyword-Extractor) (Campos et al., 2020) and retain only those that contain at least a noun. We then obtain the dependency parse tree (Qi et al., 2020b) of the sentence and map the key-phrases to tree nodes. Now, to obtain the required triple, we need to associate a verb and a relation to each key-phrase. This procedure is described in Alg 1. At a high-level, we use the path between the key-phrase and the closest verb in the dependency tree to establish a relation between the key-phrase and the verb. In cases where there is no path, we use only the key-phrase as our schema element. Figure 2 shows an example dependency tree for a sentence.
Figure 2: Dependency tree and paths showing how we obtain schema triples for a sentence: "Will this bag hold a gaming laptop and an iPad?" (from Figure 1).
Creating local schema Given a context, we extract a schema for each sentence in the context. The local schema of a context  $c$  is a union of schemata of each sentence  $s$  in the context.
$$
l o c a l \_ s c h e m a _ {c} = \cup_ {s \in c} s c h e m a _ {s} \tag {2}
$$
Algorithm 1 Pseudocode for extracting (key-phrase, verb, relation) triple.
Initialize with empty path (path length  $\infty$ ) for all possible pairs of verbs ( $\in$  {VB, VBG, VBZ}) and key-phrases in the sentence
for Each verb and key-phrase pair do
Search for the key-phrase among the children of the verb in the dependency tree
if A key-phrase is found and path is shorter than the stored path then
Update the path between the key-phrase and the verb pair
if The key-phrase is the immediate child of the verb then
Create the triple (key-phrase, verb, relation) using the relation in the path
Traverse backward from the key-phrase, stop at the first verb, use the relation with its immediate child in the path to create (key-phrase, verb, relation)
Creating global schema We define global schema at the class level where a 'class' is a group of similar contexts. For Amazon, classes consist of groups of similar products and for Ubuntu, classes consist of groups of similar dialogs (see §4.1 for details). The global schema of a class  $K$  is a union of local schemata of all contexts  $c$  belonging to  $K$ .
$$
\text {g l o b a l} _ {-} \operatorname {s c h e m a} _ {K} = \cup_ {c \in K} \operatorname {l o c a l} _ {-} \operatorname {s c h e m a} _ {c} \tag {3}
$$
A naive union of all local schemata can result in a global schema that has a long tail of low-frequency schema elements. Moreover, it may have redundancy where schema elements with similar meaning are expressed differently (e.g.  $OS$  and operating system). We therefore use word embedding based similarity to group together similar key-phrases and retain only the most frequent elements (see appendix).
Creating a missing schema Given a context  $c$ , we first determine the class  $K$  to which the context belongs. We then compute its missing schema by taking the set difference between the global schema of class  $K$  and the local schema of the context  $c$ :
$$
m i s s i n g \_ s c h e m a _ {c} = \text {g l o b a l} _ {K} \backslash \text {l o c a l} _ {c} \tag {4}
$$
More specifically, we start with the elements in the global schema and remove elements that have a semantic match (see appendix) with any element in the local schema to obtain the missing schema.
3In the case of bigram phrases, we merge the tree nodes.
3.2 Generating Useful Questions
Our goal is to generate a useful question about missing information. In §3.1, we explained how we compute the missing schema for a given context; here we describe how we train a model to generate a useful question given the missing schema.
BART-based generation model Our generation model is based on the BART (Lewis et al., 2019) encoder-decoder model, which is also a state-of-the-art model in various generation tasks including dialog generation and summarization. We start with the pretrained base BART model consisting of a six layer encoder and six layer decoder. We fine-tune this model on our data where the inputs are the missing schema and the output is the question. The elements of the missing schema in the input are separated by a special [SEP] token. Since the elements in our input do not have any order, we use the same positional encoding for all input positions. We use a token type embedding layer with three types of tokens: key-phrases, verbs, and relations.
PPLM-based decoder We observed during our human evaluation<sup>4</sup> that a BART model fine-tuned in this manner, in spite of generating questions that ask about missing information, does not always generate useful questions. We therefore propose to integrate the usefulness criteria into our generation model. We use the Plug-and-Play-LanguageModel (PPLM) (Dathathri et al., 2019) during decoding (at test time). The attribute model of the PPLM in our case is a usefulness classifier trained on bags-of-words of questions. In order to train such a classifier, we need usefulness annotations on a set of questions. For the Amazon dataset, we collect usefulness scores (0 or 1) on 5000 questions using human annotation whereas for the Ubuntu dataset we assume positive labels for (true context, question) pairs and negative labels for (random context, question) pairs and use 5000 such pairs to train the usefulness classifier. Details of negative sampling for Ubuntu dataset is in Appendix.
We aim to answer the following research questions (RQ):
1. Is the model that uses missing schema better at identifying missing information compared to models that use the context directly to generate questions?
Train	Validation	Test
Amazon	123,567	4,525	2,361
Ubuntu	102,678	7,864	200
Table 2: Number of data instances in the train, validation and test splits of Amazon and Ubuntu datasets (Both datasets are in English. Links are in appendix)
2. Do large-scale pretrained models help generate better questions?
3. Does the PPLM-based decoder help increase the usefulness of the generated questions?
Amazon The Amazon review dataset (McAuley et al., 2015) consists of descriptions of products on amazon.com and the Amazon question-answering dataset (McAuley and Yang, 2016) consists of questions (and answers) asked about products. Given a product description and  $N$  questions asked about the product, we create  $N$  instances of (context, question) pairs where context consists of the description and previously asked questions (if any). We use the "Electronics" category consisting of 23,686 products. We split this into train, validation and test sets (Table 2). The references for each context are all the questions (average=6) asked about the product. A class is defined as a group of products within a subcategory (e.g. DSLR Camera) as defined in the dataset. We restrict a class to have at most 400 products, and a bigger subcategory is broken into lower-level subcategories (based on the product hierarchy) resulting in 203 classes. While creating global schema, we exclude target questions from validation and test examples. The product descriptions and associated metadata come as inputs during test time. Hence, including them from all splits while creating the global schema does not expose the test and validation targets to the model during training.
Ubuntu The Ubuntu dialog corpus (Lowe et al., 2015) consists of utterances of dialog between two users on the Ubuntu chat forum. Given a dialog, we identify utterances that end with a question mark. We then create data instances of (context, question) where the question is the utterance ending with a question mark and the context consists of all utterances before the question. We consider only those contexts that have at least five utterances and at most ten utterances. Table 2 shows the number of data instances in the train, validation and test splits. Unlike the Amazon dataset, each context has only one reference question. A class is defined as a
4See results of BART+missinfo in Table 5
group of dialogs that address similar topics. Since such class information is not present in the dataset, we use  $k$ -means to cluster dialogs into subsequent classes. Each dialog was represented using a TF-IDF vector. After tuning the number of clusters based on sum of squared distances of dialogs to their closest cluster center, we obtain 26 classes. We follow a similar scheme as with Amazon for not including target questions from validation and test sets while building the global schema.
4.2 Baselines and Ablations
Retrieval We retrieve the question from the train set whose schema overlaps most with the missing schema of the given context.
GAN-Utility The state-of-the-art model for the task of clarification question generation (Rao and Daumé III, 2019) trained on (context, question, answer) triples.
Transformer A transformer (Vaswani et al., 2017) model trained on (context, question) pairs.
BART We finetune a BART model (Lewis et al., 2019) on (context, question) pairs.
BART + missinfo We compare to a BART model fine-tuned on (missing schema, question) pairs.
BART + missinfo + WD This is similar to the "BART + missinfo" baseline with the modification that, at test time only, we use a weighted-decoding (WD) strategy (Ghazvininejad et al., 2017) by redefining the probability of words in the vocabulary using usefulness criteria (more in appendix).
BART + missinfo + PPLM This is our proposed model as described in §3 where we fine-tune the BART model on (missing schema, question) pairs and use a usefulness classifier based PPLM model for decoding at test time.
4.3.1 Automatic Metrics
BLEU-4 (Papineni et al., 2002) evaluates 4-gram precision between model generation and references. at the corpus level; METEOR (Banerjee and Lavie, 2005) additionally uses stem and synonym matches for similarity; and Distinct-2 (Li et al., 2016) measures diversity by calculating the number of distinct bigrams in model generations scaled by the total number of generated tokens.
4.3.2 Human Judgment
Similar to Rao and Daumé III (2019), we conduct a human evaluation on Amazon Mechanical Turk to evaluate model generation on the four criteria below. Each generated output is shown with the context and is evaluated by three annotators.
Relevance We ask "Is the question relevant to the context?" and let annotators choose between Yes (1) and No (0).
Fluency We ask "Is the question grammatically well-formed i.e. a fluent English sentence?" and let annotators choose between Yes (1) and No (0).
Missing Information We ask "Does the question ask for new information currently not included in the context?" and let annotators choose between Yes (1) and No (0).
Usefulness We perform a comparative study where we show annotators two model-generated questions (in a random order) along with the context. For Amazon, we ask "Choose which of the two questions is more useful to a potential buyer of the product". For Ubuntu, we ask "Choose which of the two questions is more useful to the other person in the dialog".
4.4.1 Automatic Metric Results
Amazon Table 3 shows automatic metric results on Amazon. Under BLEU-4 and METEOR, the retrieval model performs the worst suggesting that picking a random question that matches the most with the missing schema does not always yield a good question. This strengthens the need of the second stage of our proposed model i.e. BART + PPLM based learning. GAN-Utility, which is state-of-the-art on Amazon, outperforms the Transformer baseline suggesting that training a larger model (in terms of the number of parameters) does not always yield better questions. BART, on the other hand, outperforms GAN-Utility suggesting the benefit of large-scale pretraining (RQ2). BART+missinfo further outperforms BART showing the value in training on missing schemata instead of training directly on the context (RQ1). A variation of this model that uses weighted decoding performs marginally better on METEOR but slightly worse of BLEU-4. Our final proposed model i.e., BART+missinfo+PPLM performs the best among all baselines across both BLEU-4 and METEOR.
Under diversity (Distinct-2), the retrieval model
We use original hyperparameters & tokenization scheme.
Model	BLEU-4	METEOR	Distinct-2
Retrieval	8.76	9.23	0.92
GAN-Utility	14.23	16.82	0.79
Transformer	12.89	14.56	0.60
BART	15.98	16.78	0.78
+ missinfo	16.87	17.11	0.82
+ missinfo + WD	16.23	17.98	0.84
+ missinfo + PPLM	18.55	18.01	0.86
Reference	-	-	0.95
produces the most diverse questions (as also observed by Rao and Daumé III (2019)) since it selects among human written questions which tend to be more diverse compared to model generated ones. Among other baselines, transformer interestingly has the lowest diversity whereas GAN-Utility and BART come very close to each other. Model ablations that use missing schema produce more diverse questions further strengthening the importance of training on missing schema. Our model i.e., BART+missinfo+PPLM, in spite of outperforming all baselines (except retrieval), is still far from reference questions in terms of diversity, suggesting room for improvement.
Ubuntu Table 4 shows the results of automatic metrics on Ubuntu. The overall BLEU-4 and METEOR scores are much lower compared to Amazon since Ubuntu has only one reference per context. Under BLEU-4 and METEOR scores, similar to Amazon, we find that the retrieval baseline has the lowest scores. Transformer baseline outperforms the retrieval baseline but lags behind BART, again showing the importance of large-scale pretraining. The difference between the BLEU-4 scores of BART+missinfo and our final proposed model is not significant but their METEOR score difference is significant suggesting that our model produces questions that may be lexically different from references but have more semantic overlap with the reference set. Under Distinct-2 scores, we find the same trend as in Amazon, with the retrieval model being the most diverse and our final model outperforming all other baselines.
4.4.2 Human Judgement Results
Amazon Table 5 shows the human judgment results on model generations for 300 randomly
Table 3: Automatic metric results on the full test set of Amazon. The difference between bold and nonBold numbers is statistically significant with  $p < 0.001$ .
Model	BLEU-4	METEOR	Distinct-2
Retrieval	4.89	5.12	0.82
Transformer	6.89	7.45	0.67
BART	8.23	9.67	0.72
+ missinfo	9.54	10.78	0.75
+ missinfo + PPLM	10.02	11.65	0.79
Reference	-	-	0.87
Table 4: Automatic metric results the full test set of Ubuntu. The difference between bold and nonBold numbers is statistically significant with  $p < 0.001$ .
sampled product descriptions from the Amazon test set. Under relevancy and fluency, all models score reasonably with our proposed model producing the most relevant and fluent questions. Under missing information, the BART model, finetuned on context instead of missing schema, has the lowest score. GAN-Utility outperforms BART but significantly lags behind BART+missinfo and BART+missinfo+PPLM reaffirming our finding from the automatic metric results that our idea of feeding missing schema to a learning model helps.
We additionally observe that the human-written questions score lower than model-generated questions under 'fluency' and 'missing information' criteria, mirroring similar observations from (Rao and Daumé III, 2018, 2019). We believe the reason for this is that human-written questions often have typos or are written by non-native speakers (leading to lower fluency). Moreover, humans may miss out on reading full product descriptions causing them to ask about details that are already included in the description (leading to lower missing information scores).