Consistency Training by Synthetic Question Generation for Conversational Question Answering (2024)

Abstract: Efficiently modeling historical information is a critical component in addressing user queries within a conversational question-answering (QA) context, as historical context plays a vital role in clarifying the user’s questions. However, irrelevant history induces noise in the reasoning process, especially for those questions with a considerable historical context. In our novel model-agnostic approach, referred to as CoTaH (Consistency-Trained augmented History), we augment the historical information with synthetic questions and subsequently employ consistency training to train a model that utilizes both real and augmented historical data to implicitly make the reasoning robust to irrelevant history. To the best of our knowledge, this is the first instance of research using synthetic question generation as a form of data augmentation to model conversational QA settings. By citing a common modeling error prevalent in previous research, we introduce a new baseline and compare our model’s performance against it, demonstrating an improvement in results, particularly in later turns of the conversation, when dealing with questions that include a large historical context.

heading: 1 Introduction
text: Humans often seek data through an informationseeking process in which users engage in multiple interactions with machines to acquire information about a particular concept. A prominent example of this phenomenon is the introduction of ChatGPT (OpenAI, 2023). Conversational QuestionAnswering (CQA) systems address user questions within the context of information-seeking interactions. In CQA, unlike conventional question answering, questions are interconnected, relying on previous questions and their corresponding answers (history) to be fully understood without ambiguities. Qiu et al. (2021) showed that filtering irrelevant history can boost the model’s accuracy. How-
ever, it utilizes the gold answers of history instead of the predicted ones, like many previous methods. This setting deviates from the real-world scenario, where models have to rely on their own predictions for previous questions to answer the current question. Our work aligns with the framework of addressing irrelevant history. However, unlike Qiu et al. (2021), our method abstains from utilizing the gold answers of history. Moreover, unlike Qiu et al. (2021), which requires an iterative process to select relevant history, we utilize only one transformer (Vaswani et al., 2017) during prediction, resulting in reduced time and memory. We augment the history of questions in the training set with synthetic questions. Our underlying idea is to maintain the model’s consistency in its reasoning, whether utilizing the original historical data or the augmented version. Baselines like BERT-HAE (Qu et al., 2019a), HAM (Qu et al., 2019b), and GraphFlow (Chen et al., 2020) leverage the gold answers of history in their modeling. Siblini et al. (2021) conducted a re-implementation of BERT-HAE and HAM, and Li et al. (2022) conducted a re-implementation of HAM and GraphFlow using predicted history answers, which resulted in a significant performance decrease. As a result, in this paper, we employ the base transformer of our method as the baseline, as its performance surpasses the re-implementation of the mentioned methods. Our method results in a 1.8% upgrade in overall F1 score compared to this baseline, causing a significant improvement in the scores of questions in the later turns (questions with large historical context). Furthermore, our method introduces a substantial improvement in detecting unanswerable questions compared to the introduced baseline.

heading: 2 Related Works
text: The task of CQA has been introduced to extend question answering to a conversational setting.
630
CoQA (Reddy et al., 2019) and QuAC (Choi et al., 2018) have been proposed as two extractive datasets in the CQA task. BERT-HAE (Qu et al., 2019a) employs a manually defined embedding layer to annotate tokens from previous answers within the document, and Qu et al. (2019b) extends this approach by introducing an ordering to these annotations. GraphFlow (Chen et al., 2020) utilizes a graph made out of document tokens to tackle the problem. FlowQA (Huang et al., 2019) utilizes multiple blocks of Flow and Context Integration to facilitate the transfer of information between the context, the question, and the history. ExCorD (Kim et al., 2021) uses consistency regularization (Laine and Aila, 2017; Xie et al., 2020) to regularize the training by leveraging re-written questions. Qiu et al. (2021) introduces the idea of irrelevant history and its effect on degrading performance, proposing a policy network to select the relevant history before reasoning. However, the mentioned models employ the gold answers from history in their modeling. This approach deviates from real-world scenarios, where systems should rely on their previous predictions to answer current questions (Siblini et al., 2021). Siblini et al. (2021) re-implements BERT-HAE and HAM, and Li et al. (2022) re-implements HAM, GraphFlow, and ExCorD using the model’s predictions, reporting a sharp decrease in performance. FlowQA experiences a performance drop from 64.6% to 59.0% on the development set when gold answers in history are not used (Huang et al., 2019).

heading: 3 Problem Definition
text: To model a CQA setting, at dialog turn k, a model receives a question (qk), a document containing the answer (D), and the history of the question (Hk), which is represented as a set of tuples, such as Hk = {(q0, apred0 ), · · · , (qk−1, apredk−1 )}, where apredj is the model’s prediction for qj . It’s important to note that the model may utilize only some of this information. For instance, we only employ history questions while excluding history answers. The objective is to predict the answer apredk for qk.
apredk = argmax ak P (ak|qk, Hk, D) (1)

heading: 4 Methodology
text: We seek to make the reasoning robust to irrelevant history implicitly by augmenting the dataset. To
this end, for question qk, we augment its history by injecting some synthetic questions. Let H?k be the augmented history. The intuition is that irrespective of whether the reasoning is performed with Hk or H?k , the result should be the same. In other words:
P (ak|qk, Hk, D) = P (ak|qk, H?k , D) (2)
To achieve this goal, we establish a two-stage pipeline. Our pipeline consists of a history augmentation module, whose goal is to augment the history and a question-answering module, whose objective is to consistently train a QA network so that the reasoning is consistent. The overall architecture of our model is depicted in Figure 1.

heading: 4.1 History Augmentation Module
text: This module includes a conversational question generator, denoted as CQGθ, where θ represents the parameter set of the generator, and a question selector, denoted as QS, which is responsible for choosing a set of S synthetic questions generated to augment the history.
Training The first step involves training CQGθ. While there has been research aimed at generating conversational questions (Gu et al., 2021; Pan et al., 2019), for the sake of simplifying the implementation, we employ a straightforward generative transformer for this task. To train this network, we input D, Hk, and ak into the network, intending to generate qk. We train this network using cross-entropy loss in an auto-regressive manner.
Question Generation After training CQGθ, we aim to generate synthetic conversational questions for the training set. Suppose that we want to generate synthetic conversational questions for qk. We iteratively generate synthetic questions between qj and qj+1 for 1 ≤ j ≤ k − 1. Suppose that aj is located in the i-th sentence of the document. We extract noun phrases from sentences i−1, i, and i+1 as potential answers. We make this choice because we want these answers to be similar to the flow of conversation, and if these answers are extracted from local regions, the likelihood increases. Let one of these answers be called asyn. We feed D, Hj+1 (all the questions and answers before asyn), and asyn to CQGθ to obtain the synthetic question of qsyn. We refer to all generated synthetic questions and real questions of history as the pool of questions (Pk) for qk.
Question Filtering & Injection We could set Pk as H?k ; however, Pk contains a multitude of synthetic questions which induces too much noise. Additionally, in the consistency training setting, the noise (perturbation) should be small. Thus, we only select S of synthetic questions from Pk, where S is a hyperparameter. Not all synthetic questions are helpful, necessitating the need to filter out degenerate ones. We want our selected synthetic questions to be similar and relevant to the trend of the conversation. To this end, we compute a score for each synthetic question and only keep the top M synthetic questions with the highest score. To compute the score, each question (real or synthetic) is encoded with LaBSE (Feng et al., 2022). For each synthetic question qsyn which is located between history turns qj and qj+1, the score is computed as Sim(h(qj), h(qsyn))+Sim(h(qj+1), h(qsyn)), where Sim is the cosine similarity function and h(x) is the LaBSE’s encoding of the sentence x. Additionally, sometimes, we generate questions that are too similar to previous or future questions, which are invaluable. Thus, we compare the similarity of the generated question qsyn with questions in {qk} ⋃ Hk and if the similarity is above γ, qsyn is discarded. This situation is depicted in Figure 1, where Pk contains real history questions, depicted in blue, and synthetic questions, depicted
in red and green. Those synthetic questions that have high similarity with {qk} ⋃ Hk are depicted in red. As it can be seen, the two questions “Did she have any children” and “How many children did they have” have high similarity with the question “Did they have children”, and thus, they’re discarded. In addition, we need to set a distribution to guide the selection of S number of generated questions. We conduct experiments using two distributions: uniform and linear. In the uniform setting, the generated questions are selected with the same probability. For the linear, if qsyn is located between qj and qj+1, its probability of being selected (P (qsyn)) is P (qsyn) ∝ j. We opt for the linear distribution, as we believe that closer synthetic questions to the original question might contribute to greater robustness, as questions that are further away are likely less relevant.

heading: 4.2 Question Answering Module
text: For each question qk, as illustrated in Figure 1, we feed qk, Hk, and D to the QA network (QAθ′) to compute the answer distribution. In parallel, we feed qk, H?k , and D to the QA network to compute another answer distribution. As mentioned in Section 4, we need to impose the condition outlined in Equation (2). To achieve this, we employ KL-Divergence between the answer distributions.
Additionally, we use cross-entropy loss to train the QA network for answer prediction. The losses are calculated as per Equation (3), where LCE , LCons, and LT represent the cross-entropy loss, consistency loss, and total loss. λ is a hyperparameter used to determine the ratio of the two losses.
LCE = CE(QAθ′(qk, Hk, D), a gold k ) LCons = DKL(QAθ′(qk, Hk, D), (3)
QAθ′(qk, H ? k , D))
LT = LCE + λLCons
Furthermore, we acknowledge that augmenting the history for all questions may not be optimal, as initial questions in a dialog, due to their little historical context, may not require augmentation for robust reasoning. In this case augmenting their history might add unnecessary noise, potentially degrading performance. Thus, we introduce a threshold named τ and only augment the history of qk if k ≥ τ . According to Miyato et al. (2019), we only pass the gradients through one network. As shown in the Figure 1, the symbol × is used to denote gradient cut. It should be noted that our method is model-agnostic, and any architecture could be used as the QA network.

heading: 5 Setup
text: We utilize the QuAC dataset (Choi et al., 2018), to conduct our experiments on, and data splitting is described in A. We utilize BERT (Devlin et al., 2019) as our base model to conduct experiments following the previous research. For question generation, we adopt Bart-Large (Lewis et al., 2020). Following Choi et al. (2018), we use F1, HEQ-Q, and HEQ-D as our evaluation metrics. F1 measures the overlap between agoldk and a pred k . HEQ-Q and HEQ-D are the ratio of questions and dialogs, for which the model performs better than human (Choi et al., 2018). We run multiple experiments to choose the best set of hyperparameters, resulting in setting S = 2, λ = 2.0, and τ = 6. In Appendix C, the process of choosing all hyperparameters and their analysis is described. For all of our models, we concatenate the question with history questions, feeding them to the network. More details on reproducibility are presented in Appendix E.

heading: 6 Results

heading: 6.1 Question Generation Results
text: The results of question generation are evaluated in Table 1. These scores are obtained from the dev
data. Bleu-1,4 (Papineni et al., 2002), Rouge-L (Lin, 2004), and BERTScore (Zhang et al., 2020) are used for criteria. We use the evaluate library1 to implement these metrics. Find more details in Appendix B.

heading: 6.2 Baselines Performance
text: Table 2 shows the results of our experiments in comparison to other baselines. As stated before, BERT-HAE, HAM, and GraphFlow leverage the gold answers of history. BERT-HAE reimplementation by Siblini et al. (2021), and those of HAM and GraphFlow by Li et al. (2022) are shown in the table as BERT-HAE-Real, HAM-Real, and GraphFlow-Real, respectively, indicating a significant drop in performance. 2 In this scenario, where common baselines experience a substantial decrease, we use a basic BERT model with history concatenation as the baseline, as its performance is superior. We include the results of the reinforced history backtracking model (Qiu et al., 2021) in the table. Since this model’s code is not publicly available, we have been unable to re-implement it with the correct settings and perform a meaningful comparison. However, it’s worth noting that this model utilizes unrealistic settings in two stages: once for history selection and once for question answering, potentially exacerbating the modeling issues even further. We have used “Unrealistic Settings” as a term to indicate that a method uses gold answers from history in its modeling.

heading: 6.3 CoTaH Results Analysis
text: In Table 2, CoTaH-BERT outperforms BERT (Baseline) by 1.8% in the F1 score3. According to Figure 2 in Appendix D, this improvement is mostly due to an improvement in the performance of questions with a large amount of history. This
1https://github.com/huggingface/evaluate 2For a fair comparison, the ExCorD (Kim et al., 2021) model result is not included in this table, as its best-performing model by Kim et al. (2021) and the re-implementation by Li et al. (2022) use RoBERTa (Liu et al., 2019).
3It should be noted that our test set for BERT (Baseline) and CoTaH-BERT is different from previous methods, but it has been drawn from the same distribution.
confirms that our intuition is valid that our method enhances the base model’s ability to answer questions with a large historical context. Moreover, while BERT-HAE outperforms CoTaH-BERT in terms of F1 score, CoTaH-BERT exhibits superior performance in HEQ-D. This highlights the better consistency of our model to maintain its performance throughout the entire dialog, which is achieved through superiority in answering the questions in the later turns.
Avoiding answering unanswerable questions is an indication of language understanding (Zhu et al., 2019). Table 3 shows that CoTaH-BERT brings a considerable improvement in terms of detecting unanswerable questions.

heading: 6.4 Ablation Study
text: Table 4 demonstrates the effectiveness of using the threshold (τ ) in enhancing the model capability, with more details provided in Appendix C. Moreover, the table indicates that question filtering has a tangible effect on improving performance by filtering out degenerate questions with high similarity. Lastly, we observe that using a uniform distribution is more advantageous than a linear one for question selection. We observe a relatively 1% drop in both F1 and HEQ-Q scores with the linear distribution, concluding that our hypothesis has not been true regarding the greater robustness that the linear distribution might pose. We suspect that since the
linear distribution picks more synthetic questions near the original question, it undermines the importance of immediate history, which is potentially more important than distant history, causing the consistency loss to act as a misleader instead of a regularizer in some cases.

heading: 7 Conclusions
text: In this paper, we introduced a novel model-agnostic method to make the reasoning of conversational question-answering models robust to irrelevant history. We coped with this issue by augmenting the history and training the model with consistency training. In our experiments, we didn’t follow the wrong modeling of past research in using the gold answers of history. We examined our method with BERT which exhibited a 1.8% performance boost compared to the baseline model. It was demonstrated that this improvement is primarily attributed to the enhancement of the model’s performance on questions with a substantial historical context, suggesting that our method has been successful in making the reasoning robust for these questions.

heading: 8 Limitations
text: Our model requires a phase of question generation. For synthetic question generation, the history augmentation module could be slow and the speed is directly correlated to the number of questions that one opts to generate. However, question generation is trained only once and all questions are generated in a single run, and all other experiments are conducted by only training the QA module. Moreover, although our model doesn’t need any further computation during evaluation than merely running the QA network, we need two forward passes during the training phase, which makes the training of the QA network a bit more time-consuming than training the baseline model. We have used only the QuAC dataset to report our experiments. This choice was made so that we are able to compare our results with other research, such as Qu et al. (2019a), Qu et al. (2019b), Siblini et al. (2021), and Li et al. (2022), which only use QuAC for their experiments. Thus, other datasets, such as CoQA (Reddy et al., 2019), are not tested in our research. Lastly, our research does not cover experiments on high-performing large language models, like ChatGPT. Brown et al. (2020) reports the results on the QuAC, using GPT-3 (Brown et al., 2020) in zero-shot, one-shot, and few-shot manners. However, these results are substantially inferior compared to other fine-tuning-based models that are mentioned in Table 2. Therefore, further experiments on ChatGPT and other state-of-the-art large language models are needed to better determine the placement of CoTaH and previous baselines in terms of performance.

heading: A Data Splitting
text: Since the test set of QuAC is not publicly available, we divide the development (dev) set into dev/test sets randomly, such that the number of questions in dev and test sets is almost equal. The total number
of dev and test questions is 3678 and 3676, respectively, after splitting. In our splitting, each dialog, with all of its questions, is either attributed to the dev set or the test set, in order to prevent test data leakage. Further, according to Choi et al. (2018), the original dev set of QuAC contains unique documents, meaning that a single document will not be shared among the final dev and test sets, potentially preventing test data leakage.

heading: B Question Generation Considerations
text: Gu et al. (2021) reports better results for the question generation, yet we didn’t aim to optimize BartLarge meticulously as the generated questions have a good quality for our task. The point is that in this research, we only utilize questions alone without considering answers. Thus, if the generated questions have less correlation with answers, it’s tolerable as they are still relevant questions considering the overall flow of the conversation. It should be noted that if a future research wants to incorporate predicted answers into its modeling, it should be more cautious about the quality of the question generation to ensure that the right synthetic questions are generated concerning their answers. Moreover, it should be noted that while it is true we use gold answers from history in the training of CQGθ, this does not threaten the realism of our model. The point is that only the training set of QuAC is used to train CQGθ, and later, the history of the training set is augmented for the use of the QA network. On the other hand, we never augment the history of the dev and test sets for the use of the QA network.

heading: C Hyperparameter Selection & Sensitivity Analysis
text: Initially, we determine M and γ by assessing some examples of the training data, setting M = 10 and γ = 0.8 based on our appraisal. Next, we determine the values of S, λ, and τ by conducting experiments on the dev set. In Table 5, we evaluate the effects of the model’s two main hyperparameters, S and λ, through a grid search with the following values: S ∈ {1, 2, 3} and λ ∈ {1.0, 1.5, 2.0}. Firstly, it is evident that the model performs better when S ∈ {1, 2} compared to when S = 3 overall. This suggests that S = 3 introduces too much noise, which could be detrimental to performance. Furthermore, when λ ∈ {1.5, 2.0}, the performance is better compared to λ = 1.0, indicating that the
introduction of λ is helpful, as simply adding LCE and LKL (or equally setting λ = 1.0) produces inferior performance. For the remaining experiments, we set S = 2 and λ = 2.0 as these settings yield the best F1 and HEQ-Q scores.
After setting the right amount for S and λ, we opt to examine whether the introduction of the threshold (τ ) is effective. Thus, we conduct experiments on three different amounts of this hyperparameter. In Table 6, it’s evident that the right amount of τ has a considerable effect on the performance, confirming our intuition about the functionality of τ . For all tested values of τ within the set {5, 6, 7}, performance has increased compared to the base settings with τ = 0 (or equivalently, using no threshold). Notably, the maximum performance improvement is observed when τ = 6.

heading: D Additional Results
text: In Figure 2, a comparison between the F1 scores of questions for each turn in BERT and CoTaH-BERT on the test set is presented. The score for the k-th turn represents the average F1 score for all questions in the k-th turn across all dialogs in the test set. Questions with a considerable amount of historical context are answered more effectively with our method. For 0 ≤ k ≤ 1, the performances of both
BERT and CoTaH-BERT are nearly equal, which is sensible as these questions contain little historical context and thus have little irrelevant history. However, for most of k > 1 dialog turns, CoTaH-BERT outperforms BERT or it has on par performance with BERT. The performance upgrade is especially evident towards the end of dialogs, where questions contain significant historical context. This finding indicates the superiority of CoTaH-BERT over BERT in establishing greater robustness in answering these questions, by identifying and ignoring the irrelevant history turns.
A case study regarding the performance comparison of CoTaH-BERT and BERT for a question from QuAC dataset (Choi et al., 2018) with a large history is provided in Appendix F.

heading: E Reproducibility
text: The seed for all experiments, except the training of CQGθ, is 1000. All of the experiments to train the QAθ′ are conducted on a single RTX 3070 Ti with 8GB memory, on which each experiment takes approximately 6 hours. CQGθ is trained on a single Tesla T4 from Google Colab. For each model, BERT or CoTaH-BERT, the hyperparameters are optimized on the dev set, and a final model will be trained on the train set with the optimized hyperparameters. Subsequently, a single result on the test set will be reported as depicted in Table 2. The source code can be found on our GitHub page.4

heading: F Case Study
text: In Figure 3, a document sample with its corresponding dialog in the dev set is depicted. In the figure,
4https://github.com/HamedHematian/SynCQG
the ninth turn question, q9, with its history, H9, are shown. The answers of BERT and CoTaH-BERT to q9 are compared, showing that CoTaH-BERT has been successful in answering this question with a full F1 score, while BERT has been unsuccessful. q9 asks about the release date of the album stated in q2. This is a suitable sample for our context, as there are significant irrelevant history turns between q9 and q2. We observe that CoTaH-BERT has been successful in identifying the relevant history by answering the question correctly. However, the BERT model has mistakenly reported another date, which is wrong. As BERT has returned a span containing the word “mixing”, it’s possible that BERT has incorrectly identified the previous turn question, q8, as relevant and has returned a span by text matching encompassing the word “mixing”, and containing merely some random dates.

heading: Future Work
text: It should be noted that if a future research wants to incorporate predicted answers into its modeling, it should be more cautious about the quality of the question generation to ensure that the right synthetic questions are generated concerning their answers. Moreover, it should be noted that while it is true we use gold answers from history in the training of CQGθ, this does not threaten the realism of our model. The point is that only the training set of QuAC is used to train CQGθ, and later, the history of the training set is augmented for the use of the QA network. On the other hand, we never augment the history of the dev and test sets for the use of the QA network