SlowGuess/ACL_2020_OCR huggingface dataset:
Ashutosh Baheti, Alan Ritter
{baheti.3, ritter.1492}osu.edu
Kevin Small
smakevin@amazon.com
Question answering (QA) is an important aspect of open-domain conversational agents, garnering specific research focus in the conversational QA (ConvQA) subtask. One notable limitation of recent ConvQA efforts is the response being answer span extraction from the target corpus, thus ignoring the natural language generation (NLG) aspect of high-quality conversational agents. In this work, we propose a method for situating QA responses within a SEQ2SEQ NLG approach to generate fluent grammatical answer responses while maintaining correctness. From a technical perspective, we use data augmentation to generate training data for an end-to-end system. Specifically, we develop Syntactic Transformations (STs) to produce question-specific candidate answer responses and rank them using a BERT-based classifier (Devlin et al., 2019). Human evaluation on SQuAD 2.0 data (Rajpurkar et al., 2018) demonstrate that the proposed model outperforms baseline CoQA and QuAC models in generating conversational responses. We further show our model's scalability by conducting tests on the CoQA dataset. $^{1}$
Factoid question answering (QA) has recently enjoyed rapid progress due to the increased availability of large crowdsourced datasets (e.g., SQuAD (Rajpurkar et al., 2016), MS MARCO (Bajaj et al., 2016), Natural Questions (Kwiatkowski et al., 2019)) for training neural models and the significant advances in pre-training contextualized representations using massive text corpora (e.g., ELMo (Peters et al., 2018), BERT (Devlin et al., 2019)). Building on these successes, recent work examines conversational QA (ConvQA) systems capable of interacting with users over multiple turns.
Large crowdsourced ConvQA datasets (e.g., CoQA (Reddy et al., 2019), QuAC (Choi et al., 2018)) consist of dialogues between crowd workers who are prompted to ask and answer a sequence of questions regarding a source document. Although these ConvQA datasets support multi-turn QA interactions, the responses have mostly been limited to extracting text spans from the source document and do not readily support abstractive answers (Yatskar, 2019a). While responses copied directly from a Wikipedia article can provide a correct answer to a user question, they do not sound natural in a conversational setting. To address this challenge, we develop SEQ2SEQ models that generate fluent and informative answer responses to conversational questions.
To obtain data needed to train these models, rather than constructing yet-another crowdsourced QA dataset, we transform the answers from an existing QA dataset into fluent responses via data augmentation. Specifically, we synthetically generate supervised training data by converting questions and associated extractive answers from a SQuAD-like QA dataset into fluent responses via Syntactic Transformations (STs). These STs over-generate a large set of candidate responses from which a BERT-based classifier selects the best response as shown in the top half of Figure 1.
While over-generation and selection generates fluent responses in many cases, the brittleness of the off-the-shelf parsers and the syntactic transformation rules prevent direct use in cases that are not well-covered. To mitigate this limitation, we generate a new augmented training dataset using the best response classifier that is used to train end-to-end response generation models based on PointerGenerator Networks (PGN) (See et al., 2017) and pre-trained Transformers using large amounts of dialogue data, DialogGPT (D-GPT) (Zhang et al., 2019). In §3.2 and §3.3, we empirically demon
The code and data are available at https://github.com/abaheti95/QDialogSystem.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 191-207 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Overview of our method of generating conversational responses for a given QA. In the first method, the Syntactic Transformations (STs) over-generate a list of responses (good and bad) using the question's parse tree and the best response classifier selects the most suitable response from the list. Our second method uses this pipeline to augment training data for training a SEQ2SEQ networks PGN or D-GPT (§3.1). The final SEQ2SEQ model is end-to-end, scalable, easier to train, and performs better than the first method exclusively.
strate that our proposed NLG models are capable of generating fluent, abstractive answers on both SQuAD 2.0 and CoQA.
2 Generating Fluent QA Responses
In this section, we describe our approach for constructing a corpus of questions and answers that supports fluent answer generation (top half of Figure 1). We use the framework of overgenerate and rank previously used in the context of question generation (Heilman and Smith, 2010). We first overgenerate answer responses for QA pairs using STs in §2.1. We then rank these responses from best to worst using the response classification models described in §2.2. Later in §3, we describe how we augment existing QA datasets with fluent answer responses using STs and a best response classifier. This augmented QA dataset is used for training the PGN and Transformer models.
2.1 Syntactic Transformations (STs)
The first step is to apply the Syntactic Transformations (STs) to the question's parse tree along with the expert answer phrase to produce multiple candidate responses. For the STs to work effectively accurate question parses are essential. We use the Stanford English lexparser $^2$  (Klein and Manning, 2003), which is trained on WSJ sections 1-21, QuestionBank (Judge et al., 2006), amongst other corpora. However, this parser still fails to recognize  $\sim 20\%$  of the questions (neither SBARQ nor SQ tag is assigned). For such erroneous parse trees, we simply output the expert answer phrase as a single
response. The remaining questions are processed via the following transformations to over-generate a list of candidate answers: (1) Verb modification: change the tense of the main verb based on the auxiliary verb using SimpleNLG (Gatt and Reiter, 2009); (2) Pronoun replacement: substitute the noun phrase with pronouns from a fixed list; (3) Fixing Preposition and Determiner: find the preposition and determiner in the question's parse tree that connects to the answer phrase and add all possible prepositions and determiners if missing. (4) Response Generation: Using Tregex and Tsurgeon (Levy and Andrew, 2006), compile responses by combining components of all previous steps and the answer phrase. In cases where there are multiple options in steps (2) and (3), the number of options can explode and we use the best response classifier (described below) to winnow. An example ST process is shown in Figure 2.
2.2 Response Classification and Baselines
A classification model selects the best response from the list of ST-generated candidates. Given the training dataset,  $D$ , described in §2.3 of  $n$  question-answer tuples  $(q_{i},a_{i})$ , and their list of corresponding responses,  $\{r_{i1},r_{i2},\dots,r_{im_i}\}$ , the goal is to classify each response  $r_{ij}$  as bad or good. The probability of the response being good is later used for ranking. We experiment with two different model objectives described below,
Logistic: We assume that the responses for each  $q_{i}$  are independent of each other. The model  $(F())$  classifies each response separately and assigns 1 (or 0) if  $r_{ij}$  is a good (or bad) response for  $q_{i}$ . The Logistic loss is given by  $\sum_{i=1}^{n} \sum_{j=1}^{m_i} \log(1 + q_{i,j})$ .
2https://nlp.stanford.edu/software/faq.html#z
Figure 2: An example of Syntactic Transformations in action. Question: "what year did the Netherlands rise up against Philip II?" Answer: "1568". Using the question's parse tree we: (1) modify the verb "rise" based on the auxiliary verb "did" (red); (2) add missing prepositions and determiners (sky blue); (3) combine the subject and other components with the answer phrase (green) to generate the candidate  $R_{1}$ . In another candidate  $R_{2}$ , we swap the subject with pronoun "they" (purple). Our transformations can also optionally remove Prepositional-Phrases (PP) as shown in  $R_{2}$  (orange). In the figure, we only show two candidates but in reality the transformations generate many more different candidates, including many implausible ones.
$e^{-y_{ij}*F(q_i,a_i,r_{ij})}$ ), where  $y_{ij}$  is the label for  $r_{ij}$ .
Softmax: We will discuss in §2.3 that annotators are expected to miss a few good responses since good and bad answers are often very similar (may only differ by a single preposition or pronoun). Therefore, we explore a ranking objective that calculates errors based on the margin with which incorrect responses are ranked above correct ones (Collins and Koo, 2005). Without loss of generality, we assume  $r_{i1}$  to be better than all other responses for  $(q_i, a_i)$ . Since the model  $F()$  should rank  $r_{i1}$  higher than all other responses, we use the margin error  $M_{ij}(F) = F(q_i, a_i, r_{i1}) - F(q_i, a_i, r_{ij})$  to define the Softmax loss as  $\sum_{i=1}^{n} \log \left(1 + \sum_{j=2}^{m_i} e^{-M_{ij}(F)}\right)$ .
We experiment with the following feature based and neural models with the two loss functions:
Language Model Baseline: The responses are ranked using the normalized probabilities from a 3-gram LM trained on the Gigaword corpus with modified Kneser-Ney smoothing. $^{3}$  The response with the highest score is classified as 1 and others as 0.
Linear Model: A linear classifier using features inspired by Heilman and Smith (2010) and Wan et al. (2006), who have implemented similar linear models for other sentence pair classification tasks. Specifically, we use the following features:
- Length (Features 1-3): word length of question  $q_{i}$ , answer-phrase  $a_{i}$ , and response  $r_{ij}$
- WH-word (Features 4-12): [0-1 feat.] what, who, whom, whose, when, where, which, why or how is present in the  $q_{i}$
- Negation (Features 13): [0-1 feat.] no, not or none is present in the  $q_i$
- N-gram LM (Features 14-21): 2, 3-gram normalized probability and perplexity of  $q_{i}$  and  $r_{ij}$
- Grammar (Features 22-93): node counts of  $q_{i}$  and  $r_{ij}$  syntactic parse trees
- Word overlap (Features 94-96): three features based on fraction of word overlap between  $q_{i}$  and  $r_{ij}$ . precision =  $\frac{\text{overlap}(q_i, r_{ij})}{|q_i|}$ , recall =  $\frac{\text{overlap}(q_i, r_{ij})}{|r_{ij}|}$  and their harmonic mean
Decomposable Attention: We use the sentence pair classifier from (Parikh et al., 2016), referred as the DA model. It finds attention based word-alignment of the input pair (premise and hypothesis, in our case question  $q_{i}$  and response  $r_{ij}$ ) and aggregates it using feedforward networks. Apart from standard vector embeddings, we also experiment with contextualized ELMo (Peters et al., 2018) embedding with the DA model using the version implemented in AllenNLP (Gardner et al., 2017).
BERT: Lastly, we use the BERT-Base, Uncased model (Devlin et al., 2019) for sentence pair classification. The model takes question  $q_{i}$  and response  $r_{ij}$  separated by the special token [SEP] and predicts if the response is suitable or unsuitable.
In some cases, the number of responses generated by the STs for a question could be as high as  $5000+$ . Therefore, when training the DA model with pre-trained contextualized embeddings such as ELMo or the BERT model in the Softmax loss setting, backpropagation requires computing and storing hidden states for  $5000+$  different responses. To mitigate this issue, we use strided negative-sampling. While training, we first separate all the suitable responses from all the remaining unsuitable responses. We then divide all the responses for  $q_{i}$  into smaller batches of  $K$  or fewer responses. Each batch comprises one suitable response (randomly chosen) and  $K - 1$  sampled from the unsuitable responses. To ensure that all unsuitable responses are used at least once during the training, we shuffle them and then create smaller batches by taking strides of  $K - 1$  size. We use  $K = 150$  for  $\mathbf{DA} + \mathbf{ELMo}$  and  $K = 50$  for BERT when training with the Softmax loss. At test time, we com
<sup>3</sup>http://www.keithv.com/software/giga/
pute logits on the CPU and normalize across all responses.
2.3 Training Data for Response Classification
In this section, we describe the details of the training, validation and testing data used to develop the best response classifier models. To create the supervised data, we choose a sample from the train-set of the SQuAD 2.0 dataset (Rajpurkar et al., 2018). SQuAD 2.0 contains human-generated questions and answer spans selected from Wikipedia paragraphs. Before sampling, we remove all the QA pairs which had answer spans  $>5$  words as they tend to be non-factoid questions and complete sentences in themselves (typically "why" and "how" questions). We also filter out questions that cannot be handled by the parser ( $\sim 20\%$  of them had obvious parser errors). After these filtering, we take a sample of 3000 questions and generate their list of responses using STs (1,561,012 total responses).
Next, we developed an annotation task on Amazon Mechanical Turk to select the best responses for the questions. For each question, we ask the annotators to select a response from the list of responses that correctly answers the question, sounds natural, and seems human-like. Since the list of responses for some questions is as long as  $5000+$ , the annotators can't review all of them before selecting the best one. Hence, we implement a search feature within the responses list such that annotators can type in a partial response in the search box to narrow down the options before selection. To make their job easier, we also sorted responses by length. This encouraged annotators to select relatively short responses which we found to be beneficial, as one would prefer an automatic QA system to be terse. To verify that the annotators didn't cheat this annotation design by selecting the first/shortest option, we also test a Shortest Response Baseline as another baseline response classifier model, where first/shortest response in the list is selected as suitable.
Each question is assigned 5 annotators. Therefore, there can be at most 5 unique annotated responses for each question. This decreases the recall of the gold truth data (since there can be more than 5 good ways of correctly responding to a question). On the other hand, bad annotators may choose a unique yet suboptimal/incorrect response, which decreases the precision of the gold truth.
After annotating the 3000 questions from SQuAD 2.0 sample, we randomly split the data
#q/#a	✓#r	x#r
Train	1756	2028	796174
Val	300	791	172135
Test	700	1833	182963
Table 1: Statistics of the SG training, validation, and test sets curated from the SQuAD 2.0 training data.  $q$  and  $a$  denotes the question and answer from the SQuAD 2.0 sample and  $r$  denotes the responses generated by the STs.  $\# q$  means "number of questions".  $\sqrt{\#r}$  and  $\times \# r$  denotes the number of responses which are labeled 1 and 0 respectively after the human annotation process.
into 2000 train, 300 validation, and 700 test questions. We refer to this as the SQuAD Gold annotated (SG) data. To increase SG training data precision, we assign label 1 only to responses that are marked as best by at least two different annotators. Due to this hard constraint, 244 questions from the training data are removed (i.e. the 5 annotators marked 5 unique responses). On the other hand, to increase the recall of the SG test and validation sets, we retain all annotations. We assign label 0 to all remaining responses (even if some of them are plausible). The resulting SG data split is summarized in Table 1.
Every response may be marked by zero or more annotators. When at least two annotators select the same response from the list we consider it as a match. To compute the annotator agreement score, we divide the number of matches with total number of annotations by each annotator. Using this formula we find average annotator agreement to be 0.665, where each annotator's agreement score is weighted by their number of annotated questions.
2.4 Evaluation of Response Classification
As previously mentioned in §2.3, the SG data doesn't contain all true positives since one cannot exhaustively find and annotate all the good responses when the response list is very long. Additionally, there is a large class imbalance between good and bad responses, making standard evaluation metrics such as precision, recall, F1 score and accuracy potentially misleading. To gather additional insight regarding how well the model ranks correct responses over incorrect ones, we calculate
<sup>4</sup>We found that some bad annotators had a high affinity of choosing the first (or the shortest) response when it was not the best choice in the list. To reduce such annotation errors we add another constraint that the shortest response should be selected by at least 2 different annotators.
Classifier	Loss	P@1	Max-F1	PR-AUC
ShortResp	-	0.324	0.189	-
LangModel	-	0.058	0.012	-
Linear	Log.	0.680	0.159	0.070
Linear	Soft.	0.640	0.387	0.344
DA	Log.	0.467	0.151	0.066
DA+ELMo	Log.	0.694	0.354	0.301
DA	Soft.	0.503	0.383	0.297
DA+ELMo	Soft.	0.716	0.456	0.427
BERT	Log.	0.816	0.490	0.465
BERT	Soft.	0.833	0.526	0.435
Table 2: Best response classifier results on SG test data. "ShortResp" stands for Shortest Response baseline, "LangModel" stands for Language Model baseline, "Linear" stands for Linear model. "Log." and "Soft." in Loss column stands for Logistic and Softmax loss respectively. DA refers to Decomposable Attention model (Parikh et al., 2016). "+"ELMo" refers to adding pre-trained ELMo embeddings to DA model.
Precision@1 (P@1), $^{5}$  Max. F1, $^{6}$  and Area Under the Precision-Recall Curve (PR-AUC). We train all classifier models on the SG training set and evaluate them on SG test data. The resulting evaluation is presented in Table 2.
The results show that the shortest response baseline (ShortResp) performs worse than the ML models (0.14 to 0.51 absolute P@1 difference depending on the model). This verifies that annotation is not dominated by presentation bias where annotators are just selecting the shortest (first in the list) response for each question. The language model baseline (LangModel) performs even worse (0.41 to 0.78 absolute difference), demonstrating that this task is unlikely to have a trivial solution. The feature-based linear model shows good performance when trained with Softmax loss beating many of the neural models in terms of PR-AUC and Max-F1. By inspecting the weight vector, we find that grammar features, specifically the number of prepositions, determiners, and "to"s in the response, are the features with the highest weights. This probably implies that the most important challenge in this task is finding the right prepositions and determiners in the response. Other important features are the response length and the response's 3-gram LM probabilities. The ostensible limitation of feature-based models is failing to recognize correct pronouns for unfamiliar named entities in the questions.
Due to the small size of SG train set, the vanilla
Decomposable Attention (DA) model is unable to learn good representations on its own and accordingly, performs worse than the linear feature-based model. The addition of ELMo embeddings appears to help to cope with this. We find that the DA model with ELMo embeddings is better able to predict the right pronouns for the named entities, presumably due to pre-trained representations. The best neural model in terms of  $\mathrm{P@1}$  is the BERT model fine-tuned with the Softmax loss (last row of Table 2).
3 Data-Augmentation and Generation
SEQ2SEQ models are very effective in generation tasks. However, our 2028 labeled question and response pairs from the SG train set (Table 1) are insufficient for training these large neural models. On the other hand, creating a new large-scale dataset that supports fluent answer generation by crowdsourcing is inefficient and expensive. Therefore, we augment SQuAD 2.0 with responses from the STs+BERT classifier (Table 2) to create a synthetic training dataset for SEQ2SEQ models. We take all the QA pairs from the SQuAD 2.0 train-set which can be handled by the question parser and STs, and rank their candidate responses using the BERT response classifier probabilities trained with Softmax loss (i.e. ranking loss (Collins and Koo, 2005)). Therefore, for each question we select the top ranked responses<sup>7</sup> by setting a threshold on the probabilities obtained from the BERT model. We refer to the resulting dataset as SQuAD-Synthetic (SS) consisting of 59,738  $\langle q, a, r \rangle$  instances.
To increase the size of SS training data, we take the QA pairs from Natural Questions (Kwiatkowski et al., 2019) and Harvesting  $\mathrm{QA}^8$  (Du and Cardie, 2018) and add  $\langle q,a,r\rangle$  instances using the same STs+BERT classifier technique. These new pairs combined with SS result in a dataset of 1,051,938  $\langle q,a,r\rangle$  instances, referred to as the SS+ dataset.
3.1 PGN, D-GPT, Variants and Baselines
Using the resulting SS and  $\mathbf{SS}+$  datasets, we train Pointer generator networks (PGN) (See et al., 2017), DialogPT (D-GPT) (Zhang et al., 2019) and their variants to produce a fluent answer response
7at most three responses per question
HarvestingQA is a QA dataset containing 1M QA pairs generated over 10,000 top-ranking Wikipedia articles. This dataset is noisy as the questions are automatically generated using an LSTM based encoder-decoder model (which makes use of coreference information) and the answers are extracted using a candidate answer extraction module.
$^{5}\mathrm{P}@1$  is the  $\%$  of times the correct response is ranked first  $^{6}\mathrm{Max}$  . F1 is the maximum F1 the model can achieve by choosing the optimal threshold in the PR curve
generator. The input to the generation model is the question and the answer phrase  $\langle q, a \rangle$  and the response  $r$  is the corresponding generation target. PGN: PGNs are widely used SEQ2SEQ models equipped with a copy-attention mechanism capable of copying any word from the input directly into the generated output, making them well equipped to handle rare words and named entities present in questions and answer phrases. We train a 2-layer stacked bi-LSTM PGN using the OpenNMT toolkit (Klein et al., 2017) on the SS and  $\mathbf{SS}+$  data. We additionally explore PGNs with pre-training information by initializing the embedding layer with GloVe vectors (Pennington et al., 2014) and pretraining it with  $\langle q, r \rangle$  pairs from the questions-only subset of the OpenSubtitles corpus $^9$  (Tiedemann, 2009). This corpus contains about 14M question-response pairs in the training set and 10K pairs in the validation set. We name the pre-trained PGN model as PGN-Pre. We also fine-tune PGN-Pre on the SS and  $\mathbf{SS}+$  data to generate two additional variants.
D-GPT: DialogGPT (i.e. dialogue generative pretrained transformer) (Zhang et al., 2019) is a recently released large tunable automatic conversation model trained on 147M Reddit conversation-like exchanges using the GPT-2 model architecture (Radford et al., 2019). We fine-tune D-GPT on our task using the SS and  $\mathbf{SS}+$  datasets. For comparison we also train GPT-2 on our datasets from scratch (i.e. without any pre-training). Finally, to assess the impact of pre-training datasets, we pre-train the GPT-2 on the 14M questions from questions-only subset of the OpenSubtitles data (similar to the PGN-Pre model) to get GPT-2-Pre model. The GPT-2-Pre is later fine-tuned on the SS and  $\mathbf{SS}+$  datasets to get two corresponding variants.