Synthesizing Conversations from Unlabeled Documents using Automatic Response Segmentation (2024)

Abstract: In this paper, we tackle the challenge of inadequate and costly training data that has hindered the development of conversational question answering (ConvQA) systems. Enterprises have a large corpus of diverse internal documents. Instead of relying on a searching engine, a more compelling approach for people to comprehend these documents is to create a dialogue system. In this paper, we propose a robust dialog synthesising method called SynCARS. We learn the segmentation of data for the dialog task instead of using segmenting at sentence boundaries. The synthetic dataset generated by our proposed method achieves superior quality when compared to WikiDialog, as assessed through machine and human evaluations. By employing our inpainted data for ConvQA retrieval system pre-training, we observed a notable improvement in performance across standard benchmark datasets.1

heading: 1 Introduction
text: Conversational Question Answering (ConvQA) is a computational task aimed at modeling the information-seeking processes found in human dialog. The goal of this task is to allow automated systems to understand and respond to questions within a conversational context. Several publicly available datasets, such as QuAC (Choi et al., 2018), CoQA (Reddy et al., 2019), QreCC (Anantha et al., 2021), and OR-QuAC (Qu et al., 2020), have been developed for building ConvQA systems. Despite these resources, the size of the datasets remains relatively limited, posing potential challenges when implementing ConvQA systems in real-world applications.
Meanwhile, a plethora of high-quality documents, including but not limited to sources such
1Our model and dataset are publicly available at https: //github.com/wufanyou/SynCARS.
as Wikipedia and arXiv, are publicly available. Numerous technological roadmaps have been proposed to leverage the vast wealth of information within these documents to construct a ConvQA system. One approach involves utilizing a Large Language Model (LLM) in conjunction with information retrieval techniques, such as New Bing from Microsoft. Alternatively, a well-trained LLM, such as ChatGPT without a plugin, can be employed independently to achieve similar or even superior results. By utilizing information retrieval tools, LLMs can access up-to-date information, albeit at the expense of increased inference time and latency compared to using LLMs alone.
To effectively employ these documents alongside LLMs for the purpose of constructing a Con-
8012
vQA system, there are two potential research directions to explore: the decomposition and synthesis of the documents into a dialog format, or the pursuit of improved question embedding representations for the documents. Additionally, conversational question generation (QG) can be utilized in both of these approaches. Dai et al. (2022) pioneered "dialog inpainting," suggesting every sentence in a document can answer a question, leading to the creation of the WikiDialog dataset. In our study, we introduce SynCARS, a novel approach that leverages ConvQA datasets to produce synthetic conversations from unlabeled documents, yielding superior quality compared to WikiDialog. To summarize, our main contributions can be summarized as follows:
• We identified various challenges present within the WikiDialog dataset, a component of FLAN collection (Longpre et al., 2023) used for instruction fine-tuning.
• We designed a new dataset by merging existing datasets and filtered out inadequate data to address issues present in the WikiDialog dataset. Additionally, we have designed a new answer segmentation technique by introducing a special token pmt . In comparison to the approach by Dai et al. (2022), we obtained a new and compact model specifically tailored for generating dialogues from documents.
• Our generated dataset exhibits significantly higher answer quality and question specificity, as validated through Human and GPT-4 evaluations, when compared to WikiDialog. Furthermore, the question retrieval system trained on our generated data achieves superior results compared to the system trained on WikiDialog and the standard retrieval-only benchmark method.

heading: 2 Related Work
text: Question generation (QG) is a field that seeks to create natural questions using various types of data sources, including structured knowledge bases (Guo et al., 2018; Chen et al., 2023), text (Rus et al., 2010; Du et al., 2017; Nogueira et al., 2019), images (Li et al., 2018), and tables (Bao et al., 2018). Past research efforts in this area have primarily focussed on producing isolated and disconnected questions from a given passage.
Pan et al. (2019) proposed the Conversational Question Generation (CQG) task as an approach
to improve the development of ConvQA systems. This task involves generating the subsequent question by incorporating a passage and a conversation history, thereby requiring a deeper comprehension of the given passage and prior conversation to generate a coherent and relevant question for the next round. Unlike prior QG tasks that only consider the passage, CQG requires an understanding of the previous conversation, making it a more complex task.
Kim et al. (2022) proposed SIMSEEK, a framework that generates ConvQA datasets by modeling the information needs of questioners who may ask incoherent questions due to excessive information. SIMSEEK includes a conversational answer extractor that selects answer candidates from the passage by considering the context of the conversation. However, this method is only suitable for short answers.
In contrast, Dai et al. (2022) introduced dialog inpainting, which assumes that each sentence in a document can be used as an answer to a question. The authors generated a ConvQA dataset called WikiDialog using this approach. This dataset tends to have longer answers as each answer corresponds to a single sentence. This characteristic makes it more suitable for dialog applications. While the proposed method is straightforward and efficient, concerns arise regarding the quality of the WikiDialog dataset. An illustrative example from the WikiDialog dataset is presented in Figure 1. From this example, it is evident that combining certain answers could yield improved responses and questions. Moreover, some questions are overly broad, rendering them less suitable for training a retriever system. In the context of the Open-QA dataset, the "anything else" question serves as a means to transition between topics. However, when examining a brief paragraph, typically containing around six sentences or less, it becomes challenging to discern any significant shifts in the topic.

heading: 3 SynCARS
text: Problem Statement: SynCARS (Synthesizing Conversations using Automatic Response Segmentation) aims to generate a high-quality complete dialog from an informative document. It assumes that at most N continuous sentences where (N > 1) from the document can be treated as an answer to an imagery question.
We build our work on top of the dialog inpaint-
ing (Dai et al., 2022), where each sentence is treated as an answer (equivalent to N = 1 in our assumption). Our idea stems from the observation that not all sentences are equally informative in WikiDialog, the dataset generated by dialog inpainting. Figure 1 shows some examples that demonstrate the limitations of the WikiDialog dataset.
To synthesize better ConvQA datasets, we implemented a simple sentence segmentation mechanism, along with a few modifications to the dialog inpainting method. In the following section, we will introduce these components in more detail.

heading: 3.1 Notations
text: Formally, a complete dialog d is a sequence of speaker questions, answers, and optional context, represented by:
d = (c⊕ q1 ⊕ a1 ⊕ · · · ⊕ qt ⊕ at ⊕ · · · ), (1)
where qt and at are t question and answer in a dialog, respectively. c is the prefix optional context and ⊕ is the sequence joint symbol. We use the same notation for partial dialogs, where unobserved questions are denoted by the · symbol. For example, (c⊕ · ⊕ a1 ⊕ q2 ⊕ a2 ⊕ · ⊕ a3) is a partial dialog where question q1 and q3 are unobserved, and we refer to these as "masked" utterances. Additionally, we use the shorthand dm(1,3) to denote a dialog d with masked utterances at positions 1 and 3.
To complete the partial dialog dm(1,3), we generate predictions for questions 1 and 3, denoted q̂1 and q̂3. The inpainted dialog is then:
Inpaint(dm(1,3)) = (c ⊕ q̂1 ⊕ a1 ⊕ q2 ⊕ a2 ⊕ q̂3 ⊕ a3). (2)
In this scenario, q̂1 and q̂3 are typically questions directed towards the next answer, and our goal is to associate them with all preceding utterances (q and a).

heading: 3.2 Answer Segmentation
text: Each answer at in Eq. (2) can be further decomposed with sentences, denoted by:
at = (s 1 t ⊕ p1t ⊕ · · · ⊕ smt ⊕ pmt · · · ), (3)
where smt is the m-th sentence in answer at, and pmt is its corresponding placeholder. Here we involve placeholder pmt to aid in answer segmentation. Specifically, if pmt is a special token (e.g., empty string in this paper), then we consider that
st and st+1 should be combined as one answer towards a question qt. Considering a similar case in Eq. (2), our inpainted dialog with answer segmentation can be written as follows:
SegInpaint(dm(1,3)) = (c⊕q̂1⊕s11⊕p̂ 1 1⊕s 2 1⊕q2⊕s 1 2⊕q̂3⊕s 1 3). (4)
Our model is capable of generating questions (q̂1 and q̂3) and performing answer segmentation (p̂11) simultaneously. If p̂ 1 1 is the special token that we defined, then q1 is considered as the question to (s11⊕s21). Otherwise, (q̂1⊕s11) and (p̂11⊕s21) form two question-answer pairs. By combining some of those sentences, we can generate more comprehensive responses as well as improved questions.

heading: 3.3 Training
text: To train our model, we utilize a partial dialog and aim to predict two values: qt and pit. This task is similar to the masked language modeling used in BERT (Kenton and Toutanova, 2019), where missing tokens in a passage are reconstructed. However, in our case, we aim to reconstruct a missing utterance in a dialog.
Let us assume that the model is a generative model with parameters θ, which specify a probability distribution Pθ(qt | dm(t)). Our training objective is to minimize the standard cross-entropy loss:
L(θ) = − ∑
d∈D Eqt∼d
[ logPθ(qt | dm(t)) ] (5)
where D is the set of complete dialogs and qt is a randomly sampled question from the dialog D.
Following Dialog Inpainting (Dai et al., 2022), we used T5 (Raffel et al., 2020), a text-to-text encoder-decoder Transformer as our pre-trained model. T5 uses a denoising objective that is slightly different from the original Masked Language Modeling (MLM) used in BERT. We believe that T5’s denoising pre-training objective and encoder-decoder architecture are the most suitable for this task. Figure 2 shows the original texts, inputs and targets during our training.
During training, we randomly masked at least one and at most N continuous questions q within a dialogue or question-answer pairs, as well as all answer segmentation placeholders p. As mentioned earlier in Section 3, our assumption is that N is the maximum number of sentences that can form an answer. To balance the contextual awareness of the model, we decided to randomly add or remove titles of the dialogue or QA during training.

heading: 3.4 Inference
text: Figure 3 illustrates the process of using a trained model to convert a document into a dialogue format. The process begins with a document (d) that contains a set of sentences. The trained model takes this document as input, along with a context (c) that includes the document’s title. The model then generates a dialogue by predicting questions and their corresponding answers based on the input sentences.
The model processes the document iteratively. In each iteration, it considers a fixed number of sentences (denoted as N in the figure) and generates a question that summarizes these sentences. The generated question and its corresponding answer are then added to the dialogue history. This process continues until all the sentences in the document have been processed.
During the inference process, the model maintains a counter to keep track of the number of sentences processed so far. The output of the model is a dialogue consisting of generated questions and their associated answers, which together provide a comprehensive summary of the input document.
By following this approach, the trained model can effectively convert a document into a more accessible and interactive dialogue format, enabling users to quickly grasp the key points of the document through a series of relevant questions and answers.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: The successful implementation of answer segmentation highly depends on the training dataset. Naturally, if the training dataset only contains short answers or single sentences, the answer segmentation will fail. So we perform some basic data analysis in this section. Table 1 shows the distribution of the number of sentences from the selected training datasets. Dialog Inpainting use OR-QuAC (Qu et al., 2020), and QreCC (Anantha et al., 2021) to conduct semi-supervised training, where more than 90% of the answers consist of only one sentence. In order to improve the answer segmentation capacity, we utilized an additional Dolly dataset (Conover et al., 2023), where answers usually have more than one sentence. Based on the statistics of those datasets, we set N = 3 for our experiments reported in this paper.

heading: Dataset Avg # Sen. 1 Sen. 2 Sen. ≥ 3 Sen.
text: We noticed that the WikiDialog dataset often includes a specific follow-up question – "Are there any other interesting aspects about this article?". Figure 1 also shows this behavior. This followup question is a common sentence in the QreCC, and OR-QuAC datasets. For instance, in QreCC, approximately 4% of question-answer pairs and 22.9% of dialogs contain this type of question. The original objective of those "anything else" questions was to indicate shifts in the current topic and to request any new information. However, generating these questions is not ideal because they lack the specificity needed to elicit answers that are meaningfully representative of the content being discussed. Simultaneously, we believe that a short documents should be within a topic. So we decided to cleanup QreCC and OR-QuAC datasets,
using a hand-crafted rule that excludes any question which contain "other interesting". Thus, our synthetic data is less likely to generate questions regarding the shift of topics.
Furthermore, each entry in QreCC and ORQuAC datasets contains two question types, the raw question (RQ) and the rewritten question (WQ). In cases where the raw question includes personal pronouns (such as "she," "they," and "we") and demonstrative pronouns (such as "these," "this," and "that"), a question rewriting model may involve hand-crafted rules to replace those pronouns or rewrite the question entirely. See Qu et al. (2020); Anantha et al. (2021) for more details. In this paper, we chose to use both question types during training as a data augmentation technique. Additionally, to control the desired output question types, we added a prefix to the input as "Type: {question type}" to indicate the current question type for the placeholder.

heading: 4.2 Models
text: In this study, we decided to use FLAN-T5XL (Chung et al., 2022) due to the limitations in our computational resources. Although T5XXL (Raffel et al., 2020), an 11B parameter model, was used in dialog inpainting (Dai et al., 2022), we opted for a smaller model. In Table A.1 (provided in the Appendix), we show that there is not much difference in performance when applying the proposed method to either FLAN-T5-XL or its counterpart T5-V1_1-XL. Both models are effective and efficient compared to the dialog inpainting approach.
To summarize, we initialized our model with FLAN-T5-XL, which has 3 Billion parameters, and fine-tuned it with 8 V100 16GB GPUs. The training process employed a constant learning rate of 10−4, a dropout rate of 0.1, an equivalent batch size of 32, and ran for 3.5K iterations (equivalent to 4 epochs).

heading: 5 Evaluation
text: Our primary focus in this study centers around conducting a human evaluation to compare the outputs generated by our approach with those produced by WikiDialog (WD) using Dialog inpainting method. To carry out this assessment, we utilize Amazon Mechanical Turk (MTurk) as our platform. Each human annotator is compensated at a rate of 0.036 US dollars per question, and we provide them with
identical instructions and sample examples as outlined in Figures 6 to 10 in the work of Dai et al. (2022). A potential issue in Dai et al. (2022) lies in their use of the same raters for evaluations, which could introduce bias due to the evaluators’ perception of the subject matter. To mitigate this concern, our evaluation process involves presenting subjective questions to a minimum of three distinct human evaluators for each dialogue turn. We label the answer for each question as the one agreed upon by at least two human annotators. Given that MTurk offers a more diverse pool of human evaluators, we use two-proportion z-test for evaluations.
We report our findings based on dialogs that correspond to a set of 50 selected passages2. Recent studies suggest using large language models (LLMs) as reference-free metrics for evaluating natural language generation (NLG) tasks. LLMs have the advantage of being applicable to new tasks that lack human-generated reference texts (Liu et al., 2023). Furthermore, Faysse et al. (2023) demonstrate that LLMs are more aligned with human preferences and exhibit consistent performance across a diverse set of generative tasks. In this study, we utilized the OpenAI chat completion API with the GPT-4 model (Brown et al., 2020) to perform the same evaluation as human annotators. We used the exact same rubric as Dai et al. (2022). The rubrics and prompt templates used for both human and GPT-4 evaluations are provided in Table A.4 in the Appendix. The results of this evaluation are presented in Table 2.

heading: 5.1 Quality Comparison
text: Overall, our approach consistently outperforms WikiDialog (WD) in terms of generating more specific questions and better answers, despite our model’s smaller size. To assess the statistical significance of these improvements, we conducted a two-proportion z-test, which is a statistical test used to determine if the proportions of categories in two group variables significantly differ from each other. This means that it is suitable when your variable of interest is categorical and have more than 10 values in each of the populations. If we consider ‘Very’ as an acceptable answer for the question ‘How specific is the question?’, then RQ is significantly better than WD with a p-value of 2.5 × 10−2 according to human assessment and
2These passages are the first 50 passages in the WD dataset and can be found at https://github.com/ google-research/dialog-inpainting

heading: Is the question information seeking?

heading: How well answered is the question?
text: 3.0 × 10−4 according to GPT-4. This indicates that our proposed method excels in asking more specific questions compared to WD.
We also conducted a similar test on the criterion ‘How well answered is the question?’ with ‘Perfectly’ as an acceptable answer choice. In this evaluation, RQ once again outperforms WD, with a pvalue of 4.7×10−4 according to human assessment and 2.0× 10−3 according to GPT-4. This demonstrates that RQ achieves better answer quality than WD. The improved question specificity and answer quality can enhance the utility of the synthesized dataset for downstream tasks, such as information retrieval. While WQ and RQ are both superior to WD in terms of question relevance based on human and GPT-4 evaluations, these differences are not statistically significant. In summary, RQ outperforms WD in 7 out of the 8 cases, demonstrating the superiority of our proposed method compared to the baseline.

heading: 5.2 Question Types
text: Unlike the WikiDialog dataset, we offer rewritten questions (WQ). The expectation is that these rewritten questions are superior based on many proposed criteria, especially since they tend to contain
fewer personal and demonstrative pronouns. This assumption is also validated in Table 2. However, for downstream tasks or real-world scenarios where users provide natural inputs, questions are less often in this rewritten style.

heading: 5.3 GPT-4 vs Human Evaluation
text: In general, from our comparison analysis results, GPT-4 evaluation is aligned with our human evaluation in most of the cases, which supports the argument that GPT-4 evaluation is helpful (Liu et al., 2023; Faysse et al., 2023).
We have also observed that GPT-4 become more "binary thinking" than our human evaluators. GPT4 tends to output the highest ordinal variable while human evaluators seems more conservative. Those evidences could be found from Table 2 that GPT-4 has higher absolute difference between largest ordinal variable and the second largest one compared to the one produced by human.
There is a discrepancy, though no statistical differences, between GPT-4 and humans in determining whether a question is information-seeking or not. The discrepancy can be attributed to the presence of ‘anything else’ questions in WD. When GPT-4 assesses ‘anything else’ questions, it cate-
gorizes them as information-seeking with 100%, while other questions have a 98.2% chance of being considered as information-seeking. In contrast, humans treat 79.6% of ‘anything else’ questions as information-seeking, while 88.0% of the remaining questions are also considered information-seeking. Additionally, the disagreement may arise from the fact that ‘anything else’ questions often lead to more vague or general answers in the training data for GPT-4 causing it to classify such questions as less information-seeking.

heading: 6 Application to Conversational Retrieval
text: As mentioned earlier in the introduction section, there are few ways to utilize the generated dataset, for e.g., use the dataset to train chatbot directly or transform it as a open-domain conversation retrieval task. In this section, we will focus on opendomain conversational retrieval, as datasets such as OR-QuAC (Qu et al., 2020), Trec-CAsT-19 (Dalton et al., 2020), and Trec-CAsT-20 (Dalton et al., 2021) exist in this domain.
A ConvQA system interacts with a user in a multi-turn dialogue, where the user primarily asks questions and the system responds (with occasional exceptions, such as the system asking for clarification). When it is the system’s turn to speak at a given time t, it considers the entire dialogue history, comprising all previous turns, and generates a new utterance as its response. This work focuses on the conversational retriever, showing how to improve it by pre-training on our dataset comparing to WD, leaving improvements to the generator for future work.

heading: 6.1 Dual Encoder
text: In our methodology, we employ a standard dual encoder as described in Ni et al. (2022). The objectives involve optimizing for a combination of factors: maximizing the similarity between a query q and its corresponding positive passage p∗, while simultaneously minimizing the similarity between query q and all of its negative samples N (p). This is achieved through the following loss function:
l(θ) = − log exp(sθ(q,p)/τ)∑ p∈p∗∪N (p) exp(sθ(q,p)/τ) (6) Here, sθ(q,p) represents a standard cosine sim-
ilarity, defined as:
sθ(q,p) = embedθ(q)⊺embedθ(p)
∥embedθ(q)∥∥embedθ(p)∥ (7)
In this context, embedθ refers to an embedding model used to map text into a fixed-dimension embedding vector. We adopt the setup outlined in Dai et al. (2022), utilizing a pre-trained T5-LARGE encoder as the embed model.

heading: 6.2 Datasets
text: The entire WD dataset consists of a total of 11.3 million dialogs. This comprehensive dataset is divided into 100 separate sections, commonly referred to as "dumps." However, due to the sheer size of the WD dataset, utilizing all of it for demonstration purposes can be overwhelming. Therefore, to simplify the demonstration and still provide meaningful insights, we are focusing only on the first five dumps, labeled as #00000 through #00004. These chosen dumps are used to create the RQ and WQ datasets. Notably, each dump from the RQ and WQ datasets represents 1% of the entire WD dataset. For readers interested in a more detailed statistical comparison between our generated datasets (RQ and WQ) and the original WD dataset, please refer to Appendix Table A.2.

heading: 6.3 Two-Stage Training Strategy
text: We employ a two-stage training approach for our dual encoder. Figure 4 provides more details of our training strategy.
In the first phase, we use our generated dataset, comprising WD, RQ, and WQ, to train embedθ. This is accomplished by implementing in-batch negative sampling, whereby the positive passage for a given instance, represented by i, is treated as a
negative sample for all other instances in the same batch that do not equal to i. This in-batch negative sampling strategy is an efficient, straightforward method to train our sentence embedding model. To summarize our initial training phase, we used the aforementioned datasets with a mini-batch size of 8, a learning rate of 1 × 10−4, an iteration step of 500 for each 1% subset, and the accumulated gradient batches parameter set to 32 and AdamW optimizer (Loshchilov and Hutter, 2017).
In the second phase, we fine-tune the model using OR-QuAC training set. For simplicity and in the interest of fair comparison, we’ve chosen not to incorporate the mutli stage hard sampling method outlined in Dai et al. (2022). Instead, we only use the annotated positive sample and negative samples in the dataset. Each model employed here was trained using the following parameters: a minibatch size of 16, a learning rate of 1 × 10−4, an iteration step of 250, and the AdamW optimizer for weight adjustments. These parameters were chosen to ensure the robustness of our models.

heading: 6.4 Results
text: During our experimentation, we discovered that using a small portion of the dataset for training can still improve the performance of the dual encoder. This finding offers valuable insights for optimizing computational efficiency without substantially compromising the effectiveness of the model. For example, in our cases, finetuning on 1% of RQ can lead to an average 0.8% performance improvement
comparing to using the same WD for OR-QuAC dataset. Upon analysis, our generated datasets (RQ and WQ) consistently exhibited superior performance when compared to the original WD dataset in the context of open-domain conversational retrieval tasks, and this superiority is statistically significant.
In addition, we evaluated our pre-trained dualencoder retrievers in comparison to three wellknown retrieval-only benchmarks: BM25-T5QR by Wu et al. (2022), ANCE-Query Rewriter by Yu et al. (2020) and ConvDR by Yu et al. (2021). Our approach outperforms all of these existing baselines. This conclusion not only confirms the quality of our data generation methods but also highlights their potential applicability and utility in enhancing the model’s ability to address the inherent complexities of such tasks. Additionally, as we increase the size of the fine-tuning dataset from 1% to 5% percent, the performance improves further.

heading: 7 Conclusion
text: In this paper, we introduce SynCARS, a novel approach that generates high-quality synthetic conversations from unlabeled documents by leveraging ConvQA datasets. Our method outperforms existing benchmarks in terms of the quality of the generated conversations. To generate high-quality synthetic data, we developed an answer segmentation technique that incorporates a special token and curated a new dataset. Despite using a smaller model and lower compute, human evaluations show that our generated dataset surpasses the WikiDialog dataset in terms of answer quality and question specificity. This demonstrates the effectiveness of the proposed approach. Moreover, the question retrieval system trained on our dataset outperforms both the standard retrieval-only benchmark and the same model trained using the WikiDialog dataset. We believe our contributions will facilitate future progress in the development of document-based conversational systems.

heading: Limitations
text: In our computational setup, we employed machines that were equipped with 8 × V100 GPUs to train the FLAN-T5-XL model. This training process involved using a mixed dataset comprising three public datasets. To give an estimate of the time required, training the FLAN-T5-XL model typically takes approximately one day. These time estimates
highlight the significant computational resources needed for training both models effectively.
For the retrieval experiments, it is important to note that the performance of the in-batch negative sampling strategy is significantly influenced by the mini-batch size. As the size of the mini-batch increases, model performance typically improves, since each iteration introduces more negative samples. However, due to constraints on the computational resources, we had to keep the mini-batch size relatively small.
In addition, we are unable to replicate the experiment conducted by Dai et al. (2022) and train our model using a full dataset (either 100% WD or 100% RQ and WQ) due to our limited computing resources. It is important to note that this limitation may impact the validation of the model’s effectiveness. The absence of evidence in this regard leaves uncertainty about the model’s performance in this context.

heading: Ethics Statement
text: This paper does not present any ethics-related issues. The data and additional resources utilized in this work are open-source and widely used in existing research.

heading: Appendix

heading: A.1 Choice of the base model
text: We conducted an ablation study to check which base model to use. Table A.1 shows the result for GPT-4 evaluation. Using either T5-XL or FLAN-T5-XL, our method produces successful results in the GPT-4 evaluation. We also added LLAMA-7B based on zero-shot setting as baseline method for reference.

heading: A.2 Statistics of the generated dataset
text: The WikiDialog (WD) dataset consists of a total of 11.3 million dialogues. This dataset is divided into 100 separate partitions for easier handling and processing. We use the first five partitions (#00000 to #00004) to generate new dialogs and label them as RQ and WQ datasets. Appendix Table A.2 shows the statistics of our generated dataset compared to the WD dataset. While there are a few entries for which our model cannot generate outputs in the correct format, the number of dialogs in the RQ and WQ datasets for each partition is slightly fewer than that in the WD dataset. The dialogues in our dataset generally consist of a small number of turns, which aligns with our objective of combining multiple responses to create a single, comprehensive answer.

heading: A.3 Question and Answer Overlapping
text: Table A.3 shows the overlap between questions and answers using the ROUGE score. We can conclude that our RQ and WQ datasets have higher Rouge scores compared to WD dataset, although the absolute ROUGE scores still indicate a low level of text overlap. This further demonstrates the superior quality of our RQ and WQ datasets.

heading: A.4 Prompt Template for GPT-4 evaluation
text: In Table A.4, we show the prompt Template for GPT-4 evaluation. The same rubric is also used for human evaluation.

heading: A.4 Additional Generated Examples
text: In Figure A.1 and Figure A.2, we show two more examples to compare performance with WD.

heading: Future Work
text: We believe our contributions will facilitate future progress in the development of document-based conversational systems