Modeling What-to-ask and How-to-ask for Answer-unaware Conversational Question Generation
Year: 2023

Abstract:
Conversational Question Generation (CQG) is a critical task for machines to assist humans in fulfilling their information needs through conversations. The task is generally cast into two different settings: answer-aware and answerunaware. While the former facilitates the models by exposing the expected answer, the latter is more realistic and receiving growing attentions recently. What-to-ask and how-to-ask are the two main challenges in the answer-unaware setting. To address the first challenge, existing methods mainly select sequential sentences in context as the rationales. We argue that the conversation generated using such naive heuristics may not be natural enough as in reality, the interlocutors often talk about the relevant contents that are not necessarily sequential in context. Additionally, previous methods decide the type of question (boolean/span-based) to be generated implicitly. Modeling the question type explicitly is crucial in this (answerunaware) setting, as the answer which hints the models to generate a boolean or span-based question, is unavailable. To this end, we present SG-CQG, a two-stage CQG framework. For the what-to-ask stage, a sentence is selected as the rationale from a semantic graph that we construct, and extract the answer span from it. For the how-to-ask stage, a classifier determines the target answer type of the question via two explicit control signals before generating and filtering. In addition, we propose Conv-Distinct, a novel evaluation metric for CQG, to evaluate the diversity of the generated conversation from a context. Compared with the existing answerunaware CQG models, the proposed SG-CQG achieves state-of-the-art performance.

heading: 1 Introduction
text: Building systems that can comprehend human speech and provide assistance to humans through
∗ Contribution during the internship at Institute for Infocomm Research.
†Work done when the author was on leave from NTU.
conversations is one of the main objectives in AI. Asking questions during a conversation is a crucial conversational behavior that helps AI agents communicate with humans more effectively (Allen et al., 2007; Li et al., 2016b). This line of research is known as Conversational Question Generation (CQG), which targets generating questions given the context and conversational history (Nakanishi et al., 2019; Pan et al., 2019a; Gu et al., 2021; Do et al., 2022). Compared to traditional single-turn question generation (Pan et al., 2019b), CQG is more challenging as the generated multi-turn questions in a conversation need not only to be coherent but also follow a naturally conversational flow.
Generally, there are two main settings for the CQG task: answer-aware and answer-unaware. In the answer-aware setting, the expected answers of the (to be) generated questions are exposed to the models (Gao et al., 2019; Gu et al., 2021; Shen et al., 2021; Do et al., 2022). In reality, however, the answers are only “future” information that are unknown beforehand. Thus, growing attention has been on the more realistic answer-unaware setting, in which the answers are unknown to the CQG model (Wang et al., 2018; Pan et al., 2019a; Nakanishi et al., 2019; Qi et al., 2020; Do et al., 2022).
Prior studies either attempt to ask the questions first, and compute the reward function to evaluate their answerability (Pan et al., 2019a) or informativeness (Qi et al., 2020); or they extract the answer spans from the context as the what-to-ask first, and generate the questions based on them (Nakanishi et al., 2019; Do et al., 2022). However, it has been argued that the former approach tends to generate repetitive questions (Qi et al., 2020; Do et al., 2022). For the latter approach, Do et al. (2022) recently proposed a selection module to shorten the context and history of the input and achieved stateof-the-art performance. Nonetheless, it simply employs a naive heuristic to select the earliest forward sentence (without traceback) in the context as the
10785
rationale to extract the answer span. Although such heuristics ensure the flow of the generated questions is aligned with the context, we argue that the resulting conversations may not be natural enough, because, in reality, the interlocutors often talk about the relevant parts that may not form a sequential context. Furthermore, previous studies (Gao et al., 2019; Do et al., 2022) trained the models to decide the type of the question (boolean/span-based) to be generated implicitly. We argue that modeling question type explicitly is critical since in this setting, the answer, which hints the models to generate a boolean or span-based question, is unavailable.
To address the above problems, we propose a two-stage CQG framework based on a semantic graph, SG-CQG, which consists of two main components: what-to-ask and how-to-ask. In particular, given the referential context and dialog history, the what-to-ask module (1) constructs a semantic graph, which integrates the information of coreference, co-occurrence, and named entities from the context to capture the keyword chains for the possible “jumping” purpose; (2) traverses the graph to retrieve a relevant sentence as the rationale; and (3) extracts the expected answer span from the selected rationale (Section 3.1). Next, the how-to-ask module decides the question type (boolean/spanbased) via two explicit control signals and conducts question generation and filtering (Section 3.2).
In order to exhaustively assess the quality of the generated question-answer pairs, we propose a set of metrics to measure the diversity, dialog entailment, relevance, flexibility, and context coverage through both standard and human evaluations. Compared with the existing answer-unaware CQG models, our proposed SG-CQG achieves state-ofthe-art performance on the standard benchmark, namely the CoQA dataset (Reddy et al., 2019).
Our contributions can be summarized as follows: (1) We propose SG-CQG, a two-stage framework, which consists of two novel modules: whatto-ask encourages the models to generate coherent conversations; and how-to-ask promotes generating naturally diverse questions. Our codes will be released at https://github.com/ dxlong2000/SG-CQG.
(2) SG-CQG achieves state-of-the-art performance on answer-unaware CQG on CoQA.
(3) To the best of our knowledge, we are the first to propose a set of criteria to comprehensively evaluate the generated conversations. Moreover,
we propose Conv-Distinct to measure the diversity of the generated conversation from a context, which takes the context coverage into account.
(4) We conduct thorough analysis and evaluation of the questions and answers of our generated conversations, which can bring some inspiration for future work on the answer-unaware CQG.

heading: 2 Related Work
text: Our work is closely related to two lines of prior work. Extended related work is in Appendix A.1.

heading: 2.1 Conversational Question Generation
text: Question Generation has gained much attention from the research community over the years (Pan et al., 2019b; Lu and Lu, 2021). Despite such intensive exploration, much less attention has been drawn to Conversational QG or CQG. Generally, CQG has been considered in two main settings: answer-aware and answer-unaware. In the answeraware setting, the expected answers are revealed to models (Gao et al., 2019; Gu et al., 2021; Shen et al., 2021; Do et al., 2022). However, this is not always the case in reality, as the answers are “future information”. The answer-unaware setting; therefore, receives growing interests recently (Wang et al., 2018; Pan et al., 2019a; Nakanishi et al., 2019; Qi et al., 2020; Do et al., 2022).
To tackle the what-to-ask problem, prior studies (Pan et al., 2019a; Do et al., 2022) selected the next sentence in the context as the rationale. Do et al. (2022) extract the target answer span from the rationale, while Pan et al. (2019a) generate the question, and compute a reward function to fine-tune the model by reinforcement learning. The howto-ask challenge was simply formulated as that in the answer-aware setting. In contrast, we attempt to model the rationale selection in a more coherent way by constructing and traversing a semantic graph, which simulates the keyword chains. We further propose control signals to promote diversity and fluency in question generation.

heading: 2.2 Knowledge-grounded Conversation Generation
text: Leveraging graphs to enhance dialog response generation has received growing interest (Moghe et al., 2018; Liu et al., 2019b; Xu et al., 2020, 2021). In particular, Xu et al. (2020) proposed to extract event chains (Mostafazadeh et al., 2016), and utilised them to help determine a sketch of a multi-
turn dialog. Nonetheless, the situation differs significantly when it comes to the CQG task. The responses in the dialog response generation task are normally full sentences with enough relevant mentions. However, in CQG, the questions and answers are mostly short and lack clear keywords, which makes the existing keyword-graph not applicable. We thus present a semantic graph, which incorporates the coreference, co-occurrence, and named entities information from the context.

heading: 3 SG-CQG
text: We formulate the answer-unaware conversational question generation (CQG) task as: given the referential context C = {s1, s2, ..., sm} with si being the i-th sentence in context, and the conversational history Hn = {(q1, a1), (q2, a2), ..., (qn−1, an−1)} with (qi, ai) being the i-th turn of the question-answer pairs, as input Dn = {C,Hn}, the model learns to generate the current question qn and answer an.
Figure 1 demonstrates an overview of our proposed framework. It consists of two main components: (1) A what-to-ask module aims to select a reasonable sentence in the referential context C as the current rationale rn and thereby a span in rn as the target answer an, given Dn. (2) A how-to-ask module aims to generate the question qn, guided by the rationale rn and target answer an.
3.1 What-to-ask Module (WTA) Existing answer-unaware CQG models (Pan et al., 2019a; Do et al., 2022) commonly utilize the next sentence of rn−1 in the context as the current rationale rn. Although such heuristics can guarantee that the flow of the generated questions is consistent with the narrative in context, the generated conversation may not always be as natural as in reality, since human speakers often jump back and forth across the relevant but not sequential contents in context. To facilitate the models in selecting the current rationale and target answer appropriately and further improve the semantic diversity of dialogue flow, we design a what-to-ask module, which consists of two components: semantic graph construction and graph traversal algorithm.
Semantic Graph Construction (SGC) Figure 1 shows an example of our semantic graph. Each node is displayed as a textual span and the index of the sentence it belongs to. To construct the semantic graph G = {V, E}, we first obtain the corefer-
ence clusters from the context C by AllenNLP (Shi and Lin, 2019) and build the set of initial nodes from phrases in the clusters. We then connect all the nodes in the same cluster as a chain: each node in the cluster (except the one that appears last in the context) is connected to the nearest forward one in the context. We denote this type of relation as Coreference. To enhance the connectedness of G, we extract all named entities by spaCy1 and add them as additional nodes if they are not in any clusters. We then connect all the nodes in the same sentence in the context in the same chaining style and name those edges as Same Sentence. Finally, we add a type of Extra edges between all connected subgraphs to make G fully-connected. Since those Extra edges do not bring any semantic relation to the graph, our objective is to minimize the number of those edges. Specifically, we gradually select, and connect two sentences such that their nodes are in different connected components and have the smallest indexes with the smallest difference, until the graph is fully-connected. To connect two sentences, we add an Extra edge between the last phrase in the smaller-index sentence and the first phrase in the remaining sentence. The adding-Extra-edges algorithm is in Appendix A.4.
Graph Traversal Algorithm (GTA) Given the conversational history Hn and the semantic graph G, we create a queue q to store nodes for traversing. We first add the nodes that appear in any previous turn’ rationale to q in the index order 2. We then traverse G by popping the nodes in q until it becomes empty. For each node, we retrieve the sentence that contains it as the rationale rn. If the model can generate a valid question from rn and any answer span extracted from rn, we add all unvisited neighbors of the current node to the beginning of q. A question is considered being valid if it passes the QF module (Section 3.2). Prepending the neighbors to queue is to prioritize the nodes that are connected so that the generated conversation can be formed from a chain of relevant sentences, which consolidates the coherence of the conversation. If the model cannot generate any valid qn by the current node, we add its unvisited neighbors to the end of q. The pseudocode of our proposed Graph Traversal Algorithm is described in Appendix A.2.
1https://spacy.io/ 2The nodes are ordered according to their sentences’ in-
dexes in the original context.
Answer Span Extractor (AE) We follow Do et al. (2022) to design the answer span extractor module. In particular, a T5 model is trained on SQuAD (Rajpurkar et al., 2016) to predict the target answer span (a), given its original sentence in context (r). We use this pretrained model to extract an from rn. Note that we also deselect the answer spans that are the same as those of previous turns.
3.2 How-to-ask Module (HTA) A high ratio of boolean questions in conversational datasets such as CoQA (Reddy et al., 2019) (around 20%) is one of the main challenges for current CQG studies (Gao et al., 2019; Pan et al., 2019a; Gu et al., 2021). To the best of our knowledge; however, there is no up-to-date work which attempts to tackle this challenge. This problem is even worse in the answer-unaware setting since there is no Yes/No answer to be provided to guide the generation of the models. Previous studies (Pan et al., 2019a; Do et al., 2022) simply train the CQG models to let them implicitly decide when to generate the boolean and span-based questions without any explicit modeling of the question type. We argue that explicitly modeling the question type is critical, as the models will gain more control on generating diverse questions, thus making the conversation become more natural. To this end, we introduce two control signals as the additional input to the QG model, and develop a simple mechanism to select the signal for the current turn.
Question Type Classifier (QTC) We design two control signals to guide the QG model:
<BOOLEAN> is prepended to the textual input if we expect the model to generate a boolean question, and <NORMAL> otherwise. To classify which signal should be sent to the QG model, we train a RoBERTa (Liu et al., 2019a) as our Question Type Classifier. This binary clasifier takes the rationale rn and the answer span an generated from what-toask module, the context and the shortened conversational history as the input, and generates the label 0/1 corresponding to <NORMAL>/<BOOLEAN>. We conduct additional experiments to discuss why the control_signals work in Section 6.3.
Rewriting and Filtering (RF) Our RF module serves two purposes. Firstly, following Do et al. (2022), we train a T5 model on CoQA (Reddy et al., 2019) as our CQA model to answer the generated questions. A question is passed this filtering step if the answer generated by the CQA model has a fuzzy matching score greater or equal to 0.8 with the input answer span. Secondly, when invigilating the generated conversations, we observe multiple other errors that the blackbox model encounters, as shown in Table 1. We thus propose extra post-processing heuristics to filter out the gen-
erated questions and try to avoid the following issues: (1) Wrong answer. Unlike Do et al. (2022) that took the extracted spans as the conversational answers, we rewrite the extracted answer spans for the boolean questions by selecting the answers generated from the CQA model; (2) Irrelevant. For each generated question, we remove stopwords and question marks only for filtering purpose, and we check if all the remaining tokens exist in the context C; (3) Uninformative. To remove the turns like (“Who woke up?”, “Justine”), we check validity if no more than 50% of the tokens of rn exist in any previously generated QA pairs; (4) Redundant. Unlike previous studies (Qi et al., 2020; Do et al., 2022) which only considered the redundant information from the generated answers, for each generated question that has more than 3 tokens, we filter it out if it has a fuzzy matching score >= 0.8 with any of the previously generated questions.
Question Generation (QG) We fine-tune a T5 model (Raffel et al., 2020) to generate conversational questions. We concatenate the input Dan = {C,Hn, an, rn, control_signal} in the format: Signal: control_signal Answer: an, rn Context: C [SEP] Hsub, where Hsub ∈ Hn. The model then learns to generate the target question qn. In our experiments, Hsub is the shortened Hn, in which we keep at most three previous turns. It was shown to improve upon training with the whole Hn significantly (Do et al., 2022). The performance of the QG model is in Appendix A.3.

heading: 4 Experimentation

heading: 4.1 Experimental Settings
text: Dataset We use CoQA (Reddy et al., 2019), a large-scale CQA dataset, in our experiments. Each conversation includes a referential context and multiple question-answer pairs, resulting in a total of 127k question-answer pairs. Among them, around 20% of questions are boolean, which makes this dataset become challenging for the CQG task (Pan et al., 2019a; Gu et al., 2021). Since the test set of CoQA is unavailable, we follow Do et al. (2022) to keep the original validation set as our test set and randomly sample 10% of the original training set as our new validation set.
Automatic Evaluation We utilise BERTScore (Zhang et al., 2020) as our dialog entailment metric (BERTScore-entailment), a generalization of Dziri et al. (2019). It considers the generated re-
sponse (question/answer) as the premise, and the utterances in the conversational history as the hypothesis, and measures their similarity score as the topic coherence score. This property is crucial as the questions/answers should focus on the same topic as the previous turn(s). In our experiment, we measure the dialog entailment score with 1, 2, and all previous turn(s). To measure the relevance between the generated conversation and the context, we concatenate the generated QA pairs and compute the BERTScore. It provides how the generated conversation is explicitly relevant to the context.
We observe short conversations with very few generated turns tend to yield very high scores on the available diversity measurement metrics such as Distinct (Li et al., 2016a). Since the conversation is generated from a given context, we argue that how much information from the given context the generated conversation covers should be taken into account. To this end, we introduce Context Coverage (CC) to measure the percentage of the sentences in the context that are the rationales of generated QA pairs. Our proposed Conv-Distinct of a generated conversation is then computed by multiplying the Distinct score of the generated conversation with its CC score, to measure the diversity of the turns generated from a given context:
Conv-Distinct = CC * Distinct (1)
We further provide Jumping Score (JS) to measure the flexibility of the generated conversation. JS is defined as the percentage of turns in which the model jumps back to any previous content of their previous turn (i.e. trace-back). It is worth noting that we do not rank the models based on JS score. Details of proposed metrics are in Appendix A.7.
Human Evaluation Human evaluation is critical to evaluate the quality of the generated conversations since the CQG model may generate reasonable conversations but unmatched well with the provided ground-truth ones. We randomly select 25 contexts in our test set and take the first five generated turns from the output of each model to compare, resulting in 125 samples in total. We hire three annotators who are English native speakers. Each generated question is rated by annotators on a 1-3 scale (3 is the best). We follow Do et al. (2022) to utilize three criteria: (1) Factuality measures the factual correctness and meaning of generated questions, (2) Conversational Alignment measures how aligned the generated questions are with the
history, (3) Answerability measures how answerable the generated questions are by the given context. Given the fact that LMs can generate fluent texts, we omit using Fluency and Grammaticality. We measure the annotators’ agreement by Krippendorff’s alpha (Krippendorff, 2011). Our human rating instructions are in Appendix A.9.
Implementation Details We fine-tune a RoBERTalarge (Liu et al., 2019a) as our binary Question Type Classifier with the pretrained checkpoints from fairseq (Ott et al., 2019) on CoQA. We use a learning rate of 1e-5, a window size of 512, a batch size of 4, and AdamW (Loshchilov and Hutter, 2019) as our optimizer. Our classifier achieves an accuracy of 95.6%. The model is finetuned on a P40 Colab GPU for 10 epochs. Details of the input format are in Appendix A.5.
We initialise SG-CQG with pretrained checkpoints of T5base model (Raffel et al., 2020) from Huggingface (Wolf et al., 2020). We also use AdamW (Loshchilov and Hutter, 2019) as our optimizer with a warmup of 0.1 and an initial learning rate of 1e-4. We train the model for 100k iterations with a standard window size of 512, a batch size of 4, and use a Beam search decoding strategy with a beam size of 4.

heading: 5 Main Results
text: To evaluate the performance of SG-CQG on the answer-unaware CQG task, we employ 4 baselines for comparison, as shown in Table 2. (1) T5base (Raffel et al., 2020), (2) BARTbase (Lewis et al., 2020), (3) GPT-2 (Radford et al., 2019), which are fine-tuned to generate conversational questionanswer pairs end-to-end, and (4) CoHS-CQG (Do et al., 2022) which adopts a strategy to shorten the context and history of the input, achieves the SoTA performance on CoQA in answer-aware and answer-unaware CQG.
Firstly, we observe that SG-CQG outperforms other methods on most of the metrics, except Distinct and BERTScore. The reason is that BART and T5 often generate short QA pairs (the CC scores are 8.62% and 23.33% on average, respectively), and copy more from the context, thus they get higher scores on Distinct and BERTScore. Secondly, the metric Conv-Distinct reasonably penalizes models that generate too short conversations, on which SG-CQG achieves the best results. Thirdly, by allowing the model to jump back and forth across the relevant contents in the context by the semantic graph, SG-CQG outperforms other methods significantly on BERTScore-entailment, which indicates that conversational coherence is indeed im-
proved. Furthermore, SG-CQG achieves the highest JS score, which demonstrates that the whatto-ask module allows our model to be most flexible in selecting rationales compared to the baselines. SG-CQG also achieves a significantly higher Context Coverage (CC) score compared to CoHSCQG. Finally, compared with the results of Oracle, which are from the human-generated conversations, SG-CQG achieves commensurate performance on BERTScore-entailment and BERTScore. It demonstrates that our generated conversations are as closely coherent as human-generated ones.
Question Generation Evaluation We compare the generated conversational questions of our model with 4 baselines: (1) ReDR (Pan et al., 2019a) is an encoder-decoder framework which incorporates a reasoning procedure to better understand what has been asked and what to ask next about the passage; (2) T5base (Raffel et al., 2020); (3) GPT-2 (Radford et al., 2019); (4) CoHS-CQG (Do et al., 2022). For T5, GPT-2 and CoHS-CQG, we extract the generated questions from the generated conversations for comparison. We measure the diversity of the generated questions by Distinct (Li et al., 2016a) and our proposed Conv-Distinct. Table 3 shows evaluation results of the generated conversational questions. We observe that SG-CQG achieves the best performance on Conv-Distinct, which takes the context coverage into account.
Answer Span Extraction Evaluation We further evaluate the generated conversational answers of our model with 4 baselines: (1) T5base (Raffel et al., 2020); (2) BARTbase (Lewis et al., 2020); (3) GPT-2 (Radford et al., 2019); (4) CoHS-CQG (Do et al., 2022). We extract the generated conversational answers from the generated conversations of the models for comparison. We train another T5base model on CoQA for the CQA task (see Appendix A.6) and utilize it to generate the groundtruth answers for the generated questions of the models. We then evaluate the quality of the generated conversational answers by measuring the Exact Match (EM) and F1 scores with the groundtruth ones. Table 4 shows the evaluation results. We observe that the generated conversational answers extracted by SG-CQG achieve the best EM and F1 scores, which are significantly higher than the other baselines.
Human Evaluation The results of the human evaluation are present in Table 5. Generally, SG-
CQG achieves the highest performances on all three proposed metrics with a good overall annotators’ agreement with an alpha of 0.73. In particular, we observe that by integrating the semantic graph into the selection of the rationales, SG-CQG outperforms CoHS-CQG (Do et al., 2022) significantly in the conversational alignment property. Furthermore, SG-CQG improves CoHS-CQG by a gap in the answerability and factuality of the generated questions, which reflects that our RF module with additional post-processing steps works as expected.

heading: 6 Discussion

heading: 6.1 Ablation Studies

heading: Ablation of What-to-ask Module (WTA) To
text: better understand how the what-to-ask module affects our proposed model in generating conversations, we study its ablation named SG-CQG + w/o WTA in Tables 2, 3, 4. In this case, our model becomes an upgraded version of CoHS-CQG (Do et al., 2022). Compared to CoHS-CQG, it achieves higher scores on all metrics except the Context Coverage (CC), which reflects that the quality of the generated conversations is indeed improved. These improvements are expected as the model in this case gains more control over generating boolean questions and has a stricter filtering process. This stricter filtering process also explains why it gets a lower CC score compared to CoHS-CQG.

heading: Ablation of Question Type Classifier (QTC)
text: We conduct an ablation study of the Question Type Classifier (QTC) module. We name this experiment SG-CQG + w/o QTC. Table 2 shows the evaluation results of generated question-answer pairs. Compared with SG-CQG, the performance of SG-CQG + w/o QTC drops slightly on nearly all metrics (except Distinct), which consolidates our hypothesis that explicitly modeling the question type improves the overall coherency of the conversation. Furthermore, Table 3 shows that QTC enhances the diversity of the generated questions, while Table 4 illustrates that QTC improves the quality of the
generated answers.
Ablation of Rewriting and Filtering (RF) SGCQG + w/o RF in Table 2 shows the ablation results of the Rewriting and Filtering (RF) module. As removing the RF module means we do not filter out any generated question, it results in two consequences. Firstly, since for each sentence, the model can generate at least one conversational question, the CC score of SG-CQG + w/o RF is perfect (100%). Second, redundant questions and answers are generated very frequently. As such, removing the RF module reduces the quality of the generated question-answer pairs (Table 2) and questions (Table 3) significantly. Notably, without the RF module, the extracted answer spans by SG-CQG + w/o RF can be very different from the true conversational answers, resulting in very low F1 and EM scores (Table 4). Although the CC score is perfect, the generated question-answer pairs from this experiment are of bad-quality.

heading: 6.2 Case Study
text: We present one conversation generated by our proposed SG-CQG in Table 6. We observe that the rationale of Q2-A2 is the 3-rd sentence in the context, and the rationale of Q3-A3 is the 8-th sentence, which is a forward jump of the model. On the other hand, the rationale of the Q4-A4 is the 7-th sentence, which is a traceback. Such a traceback enhances reasonable coherence between Q3-A3 and Q4-A4. Furthermore, Q5-A5 to Q6-A6 is also a traceback, and especially, Q6 is a boolean question. More case studies are shown in Appendix A.10.

heading: 6.3 Why Do Control Signals Work?
text: Experimental Settings We design the experiments to verify the helpfulness of our two proposed control_signals: <BOOLEAN> and <NORMAL>.
In particular, we train a T5 model (Raffel et al., 2020) in the answer-aware setting. Given the input Dan = {C, Hn, an, rn} with C, Hn, an, rn as the context, ground-truth conversational history, ground-truth answer, and round-truth rationale, respectively, we conduct three experiments in Table 9: original input with Yes/No keyword (With Y/N ), original input without Yes/No keyword (W/o Y/N ), original input without Yes/No and with the ground-truth control_signal (W/o Y/N + control_signal). Note that we train the model with the whole context, and a maximum of three previous history turns, as discussed in Appendix A.3. We measure the performance of the answer-aware CQG model separately on two types of questions: boolean and span-based by ROUGEL (Lin, 2004) and BERTScore (Zhang et al., 2020).
Observations Table 9 shows the experimental results. We derive two main observations. Firstly, without knowing the keyword Yes/No (W/o Y/N ) - this is the case in the answer-unaware setting, the model performs worse. This decrease shows that the Yes/No keyword is indeed helpful in hinting the model towards generating the correct questions. Secondly, by inputting the groundtruth control_signal into the model (W/o Y/N + control_signal), the performance is improved by a large margin compared to (W/o Y/N ). We obtain three implications from the above improvement. Firstly, it consolidates our hypothesis that inputting the ground-truth control_signal is truly helpful. Secondly, by training with the control_signal, the performance of the model is even higher than with Y/N in the span-based cases, which indicates that training the model with control_signal makes it more stable to generate the correct questions. Thirdly, the performance of (W/o Y/N + control_signal) is lower than (With Y/N ) in
boolean cases. The reason is <BOOLEAN> only informs the model to generate a boolean question without informing to generate an Yes or No one.

heading: 7 Conclusion
text: This paper presents SG-CQG, a two-stage framework for the CQG task in the answer-unaware setting. Firstly, the what-to-ask module aims to select a sentence as the rationale by the proposed semantic graph and extract the answer span from it. The how-to-ask module classifies the type of the question before generating and filtering it. Additionally, we propose a set of automatic evaluation criteria for answer-unaware CQG, especially a novel metric, Conv-Distinct, to evaluate the generated conversation from a context. Extensive automatic evaluation and human evaluation show that our method achieves state-of-the-art performances in the answer-unaware setting on CoQA, with a significant improvement in the conversational alignment property compared to previous frameworks. In the future, we will focus on how to reason over our semantic graph to select the rationale, and further improve the performances of how-to-ask module.

heading: Limitations
text: A limitation of our work is that our Graph Traversal Algorithm (Section 3.1) is a heuristic and unlearned algorithm. This leads to a number of nodes after being selected by this algorithm are not suitable for the model to generate conversational questions, and are eventually filtered out by other modules. Future works can focus on more advanced techniques to guide the model to select the nodes such as Graph Neural Networks (Wu et al., 2020). Furthermore, our algorithm to select the relevant turns in the conversational history to generate the conversational questions is a heuristic of selecting a maximum of three previous turns. This heuristic may not be optimal for the model to gather necessary information from history to generate conversational questions in the next turns, as discussed by Do et al. (2022).

heading: Ethical Considerations
text: In this paper, we present a two-stage CQG framework (SG-CQG), which was trained on CoQA (Reddy et al., 2019), a published large-scale dataset for building Conversational Question Answering systems. Our framework is potentially helpful for building chatbot systems, which can serve different
streams such as educational, medical, or commercial purposes.
Through human evaluations, we observe that our proposed method does not generate any discriminatory, insulting responses (questions and answers). We validate the proposed method and baseline models on human evaluation which involves manual labor. We hire three annotators to score 125 generated questions in total. The hourly pay is set to S$15, which is higher than the local statutory minimum wage. Therefore, we do not anticipate any major ethical concerns.

heading: Acknowledgements
text: This research has been supported by the Institute for Infocomm Research of A*STAR (CR-2021001). We would like to thank anonymous reviewers from ARR for their valuable feedback which helped us to improve our paper. We also want to thank Dr. Richeng Duan (A*STAR) for his feedback in the initial stage of the project.

heading: A Appendix

heading: A.1 Extended Related Work
text: Our work is related to two more lines of prior work.

heading: A.1.1 Synthetic Question-Answering (QA) Generation
text: Synthetic QA generation based on pretrained language models (LM) has been studied and demonstrated the helpfulness in improving the downstream Reading Comprehension (RC) task (Alberti et al., 2019; Puri et al., 2020; Shakeri et al., 2020). Alberti et al. (2019) proposed a novel method to generate synthetic data by combining models of question generation and answer extraction, and by filtering the results to ensure roundtrip consistency. However, this work differs from ours since it only considered the task of single-turn QA generation and focused only on extractive QA generation while we focus on multi-turn QA generation and have both span-based and boolean questions. Regarding the filtering technique, this work and Puri et al. (2020) used round-trip filtering method, which is similar to Do et al. (2022) and is a relaxed version of our filtering module. Shakeri et al. (2020) later introduced an end-to-end framework to generate QA data. This work used LM filtering method, which is similar to sample-and-reranking (Holtzman et al., 2020) and ours. In our case (as discussed in (1) Wrong answer error in Section 3.2), to filter QA pairs, we also sample multiple answers from a QA model and select the answers with the highest frequency and confidence score by the model. If the highest frequency one is different from the highest confidence one, we filter our the question.

heading: A.1.2 Dialog Generation Evaluation
text: Dialog evaluation metrics have been studied extensively (Yeh et al., 2021). However, it is worth noting that this task is different from ours, since we prefer evaluating the questions in QA conversations only. In addition, when conducting experiments with reference-free dialog generation metrics like BERT-RUBER (Ghazarian et al., 2019) and HolisticEval (Pang et al., 2020), we observe that these metrics are not suitable for evaluating QA pairs since the questions and answers in QA conversations are normally shorter without many referential details among turns compared to dialog responses.
Previous works (Alberti et al., 2019; Puri et al., 2020; Shakeri et al., 2020) usually evaluated the
generated QA data by training the RC systems with it and examining whether the synthetic data improves the RC systems without actually examining the synthetic data. Recent work (Do et al., 2022) evaluated the QA pairs manually. In addition, (Yue et al., 2022) proposed question value estimator, a novel module to estimate the usefulness of synthetic questions to improve the target-domain QA performance. However, this is not directly relevant to ours since even though the metric can evaluate the usefulness of the generated questions, it does not offer the actual properties of the generated questions. To the best of our knowledge, our work is the first one that proposes a set of criteria to evaluate the question-answer pairs in QA conversations. The performance of models evaluated by our proposed automatic evaluation metrics (Table 2) is positively correlated with human evaluation (Table 5) where we observe that improvements on our metrics are also improvements on human evaluation metrics.

heading: A.2 Graph Traversal Algorithm
text: We present the pseudocode of our Graph Traversal Algorithm, which is described in Section 3.1.
Algorithm 1: Graph Traversal Algorithm Input: G = {V, E},
H = {(q1, a1), ..., (qn−1, an−1)}/∅. Output: Index of the sentence as rn Initialize: I: nodes in rationales of H , q: queue of nodes to visit, Add nodes in I to q in the index order. 1 while q is not empty do 2 cur = q[0] 3 del q[0] 4 if cur is visited twice then 5 continue 6 end 7 rn = retrieve sentence contains q[0] 8 An = answer spans set extracted from rn 9 if successfully generate qn from rn and
any an ∈ A then 10 Add unvisited neighbors of cur to the beginning of q 11 else 12 Add unvisited neighbors of cur to the end of q 13 end 14 end

heading: A.3 Question Generation
text: Given the input Dan = {C, Hn, an, rn, control_signal} in which C, Hn, an, rn, control_signal are the context, conversational history, expected answer, rationale, and the control signal respectively, we fine-tune a T5base model (Raffel et al., 2020) as our question generation model. Do et al. (2022) showed that by training the T5 model with the whole context and the shortened conversational history, the performance of the model is improved. We replicate this experiment by reporting the performance of the T5 model with a different number of the previous history turns in Table 7. We derive the same observation as Do et al. (2022), which is the model performs the best with a maximum of two or three conversational previous turns. As such, we opt for selecting at most 3 previous turns to train our QG model.

heading: A.4 Adding Extra Edges Algorithm
text: We provide the pseudocode for the adding-Extraedges algorithm in Algorithm 2.

heading: A.5 Details of Question Type Classifier
text: In this section, we detail our setting to train and validate the proposed Question Type Classifier. We conduct our experiments on train set, our test set (i.e. CoQA validation set) and our validation set of CoQA (Reddy et al., 2019). For each conversation, we automatically label its questions according to their answers. In particular, a question is labeled as boolean if its answer begins with Yes/No/yes/no/YES/NO, and span-based otherwise. Given the input Dan = {C, Hn, an, rn} with C, Hn, an, rn are the context, groundtruth conversational history, ground-truth answer, round-truth rationale respectively, we construct the input to the classifier as followed. If an ∈ {Yes, No, yes, no, YES, NO}, the input to the classifier is Answer: rn rn Context: C [SEP] Hsub, else, the input is Answer: an rn rn Context: C [SEP] where Hsub is the short-
Algorithm 2: Adding Extra Edges Input: G = {(u, v)} for u, v are nodes in
directed graph that belong to the same sentence. For different sentences, only consider the starting node and the ending node.
Output: The set of newly added edges Initialize: A disjoint set union (DSU) for
checking whether 2 sentences are in the same component.
1 addedEdges = [] 2 pairs = all pairs of 2 sentences 3 sort(pairs) // for prioritizing
those pairs with the minimum index difference
4 for pair in pairs do 5 p1, p2 = pair[0], pair[1] 6 sameComponent = check the connectivity of p1, p2 by DSU 7 if not sameComponent then 8 merge sentences p1 and p2 into the same component by DSU 9 add new edge between the ending
node of sentence p1 with starting node of sentence p2 to addedEdges

heading: 10 end
text: 11 end 12 return addedEdges
ened Hn, in which we keep at most three previous turns, and the output is 0/1 indicating whether the ground-truth question is boolean/span-based. Our classifier achieves an accuracy of 95.6%.

heading: A.6 Details of CQA Model
text: We fine-tuned a T5 (Raffel et al., 2020) as our Conversational Question Answering (CQA) model on CoQA (Reddy et al., 2019). The input to the model follows the format: Question: Q [SEP] Context: C [SEP] H_sub in which Q, C are the question and the context respectively, and H_sub is the shortened conversational history with a maximum of 3 previous turns. Our CQA model achieves 63.65% Exact Match (EM) and 74.08% F1, as we presented in Table 4.

heading: A.7 Evaluation Metrics Discussion
text: One of our core contributions is the set of criteria to evaluate question-answer conversations. In this section, we detail our intuitions as well as computations of the metrics.
A.7.1 Distinct-N (Li et al., 2016a) Distinct-N (Li et al., 2016a) is a N-gram metric to measure the diversity of a sentence. In our experiments, we calculate Distinct-1 score and Distinct-2 score provided by Li et al. (2016a)3.

heading: A.7.2 Context Coverage and Conv-Distinct
text: As we discussed in Section 4, one critical shortcoming when directly applying Distinct-N to evaluate the QA conversations is that the conversations with very few turns tend to attain very high Distinct-N scores. To address this challenge, we introduce Context Coverage (CC) and Conv-Distinct.
Context Coverage (CC) is measured as the percentage of sentences that are rationales. For example, given a context of 6 sentences, among them, 5 sentences are selected as rationales for a generated conversation. Then the CC score of this generated conversation is 5/6 = 0.84.
To compute CC Scores for E2E models, we classify a sentence as a rationale if there is at least one question-answer pair generated from that sentence. As a result, the model of Do et al. (2022) and our SG-CQG can output which sentence is a rationale, and it is straightforward to compute the CC scores. However, the end-to-end outputs of BART (Lewis et al., 2020) and T5 (Raffel et al., 2020) are
3https://github.com/neural-dialogue-metrics/Distinct-N
the question-answer pairs only, it is needed to find which sentence is a rationale of each pair. To do so, we adopt a simple heuristic. For each generated question-answer pair, we classify a sentence as its rationale if that sentence has the longest common substring with the concatenation of its question and answer among all the sentences in the context. By that, we get the set of sentences that are rationales.
Conv-Distinct is defined as the multiplication of the Distinct score of the generated conversation with its CC score. For example, in the above generated conversation, the Distinct-1 score is 60.50. So its Conv-Distinct-1 score is 60.50 * 0.84 = 50.42.
It is worth noting that the diversity in token level is a common property of the dialog which has been discussed in many previous works (Qi et al., 2020; Pang et al., 2020; Adiwardana et al., 2020).

heading: A.7.3 BERTScore−entailment 1, 2, 3
text: BERTScore−entailment is an upgraded version of Dialog−entailment metric (Dziri et al., 2019), which measures the topic coherence property by deep contextual representation. We follow Dziri et al. (2019) to characterize the consistency of dialogue systems as a natural language inference (NLI) problem (Dagan et al., 2006). This property is important for questions and answers in the QA conversation, because the questions should focus on the topic of previous turns, and the answers should focus on their questions. In our experiments, we compute BERTScore−entailment with 1, 2, and all previous turn(s). The BERTScore calculation is adopted from its authors4.

heading: A.7.4 BERTScore (Zhang et al., 2020)
text: We observe that Distinct-N, Conv-Distinct-N and BERTScore-entailment are only to measure the quality of the QA pairs. None of them measures the relationship between QA pairs and the given context. As such, we propose to use BERTScore (Zhang et al., 2020) to measure the similarity of the generated conversation and the given context. It is worth noting that this metric only serves for measuring the similarity between the generated conversation and context only. A generated conversation with a very high similarity score with the given context does not reflect that it is a very good conversation, as in the case of BART (Lewis et al., 2020) in Table 2. We provide this metric to give
4https://github.com/Tiiiger/bert_score
audiences "a sense" of how the generated conversation is explicitly relevant to the given context.

heading: A.7.5 EM & F1 Answerability Measurements
text: The Exact Match (EM) and F1 measurements in Section 5 are to evaluate the answerability and the correctness of our generated questions and answers respectively (i.e. the quality of the generated conversational answers). Since from a context, multiple conversations can be generated, we argue that one critical aspect of a good conversation is the quality of the generated conversational answers, i.e. the conversational questions must be answerable by the given context, and their answers must be exactly the generated conversational answers.

heading: A.7.6 Jumping Score (JS)
text: To further understand the characteristics of each model in generating conversations, we measure its jumping score. We define this score as the percentage of turns in which the model jumps back to any previous content of their previous turn (i.e. traceback). For example, a generated conversation with the indexes of rationales [1,4,3,5,8,6] has the JS score is 2/5 = 0.4. It has 2 turns (over a maximum of 5 jumping back turns) in which the model jumps back, which are the 3−rd turn and 6−th turn. It is worth noting that the JS only shows one of the aspects of the result analysis. We could not say a system with the highest JS is better than others. JS only reflects a kind of flexibility for a what-toask module to some extent. We observe that our proposed SG-CQG achieves the highest JS score, which reflects that our proposed what-to-ask module is the most flexible in terms of selecting the sentences in the context.

heading: A.8 Statistical Significance of Results
text: We compute the Student’s t-test to measure the significant difference between our model’s performance and the best baseline for each evaluation metric with the null hypothesis H0: There is no significant difference, and H1: There is a significant difference. We obtained the p-values as in Table 2:
• Compared to T5: 4.32e-11 (BERT-entailment all), 5.20e-98, (BERT-entailment 1), 2.48e-34 (BERT-entailment 2).
• Compared to CoHS-CQG: 7.62e-188 (CC Score), 5.12e-119 (Conv-Distinct 1), 8.11e-173 (Conv-Distinct 2). The p-values, in this case, are
too small because the improvements are intuitively significant.
We observe that all the p-values are less than .01, which indicates that our improvements on those metrics are significant.

heading: A.9 Human Evaluation Scoring System
text: We describe how we instructed three annotators to point the generated questions based on three criteria: Factuality, Conversational Alignment, and Answerability, as discussed in Section 4, in Figure 2. Following the discussion of Do et al. (2022), it is noted that in the answer-unaware setting, there is no target answer and rationale. However, since the what-to-ask module first seeks for one sentence in the context as the rationale and extracts the target answer from it, we still have the target answer and rationale. We thus define the Score 2, 3 based on the retrieved rationale and extracted target answer.

heading: A.10 Extended Case Studies
text: We present additional samples generated by SGCQG in Table 8. In these samples, the first turn of the conversations is input to the model, and it generates their following turns.

heading: A.11 Supplement: Why Do Control Signals Work?
text: In this section, we supplement the experimental results of the experiments with the control_signal. The results are presented in Table 9, and the discussions are in Section 6.3.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: Future works can focus on more advanced techniques to guide the model to select the nodes such as Graph Neural Networks (Wu et al. , 2020). Furthermore, our algorithm to select the relevant turns in the conversational history to generate the conversational questions is a heuristic of selecting a maximum of three previous turns. This heuristic may not be optimal for the model to gather necessary information from history to generate conversational questions in the next turns, as discussed by Do et al. (2022)