Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sources
Authors: Yerin Hwang
Yongil Kim
Hyunkyung Bae
Hwanhee Lee
Jeesoo Bang
Kyomin Jung
Venue: EMNLP 2023 Main (2023)
Keywords: Dialog System
Conversational Question Answering
Dataset Generation

Abstract: To address the data scarcity issue in Conversational question answering (ConvQA), a dialog inpainting method, which utilizes documents to generate ConvQA datasets, has been proposed. However, the original dialog inpainting model is trained solely on the dialog reconstruction task, resulting in the generation of questions with low contextual relevance due to insufficient learning of question-answer alignment. To overcome this limitation, we propose a novel framework called Dialogizer, which has the capability to automatically generate ConvQA datasets with high contextual relevance from textual sources. The framework incorporates two training tasks: question-answer matching (QAM) and topic-aware dialog generation (TDG). Moreover, re-ranking is conducted during the inference phase based on the contextual relevance of the generated questions. Using our framework, we produce four ConvQA datasets by utilizing documents from multiple domains as the primary source. Through automatic evaluation using diverse metrics, as well as human evaluation, we validate that our proposed framework exhibits the ability to generate datasets of higher quality compared to the baseline dialog inpainting model.

Dialogizer: Context-aware Conversational-QA Dataset Generation from
Textual Sources
Yerin Hwang
Yongil Kim 2+ Jeesoo Bang'
Hyunkyung Bae
Hwanhee Lee
Kyomin Jung!.25
IPAL, Scoul National University DDept. of ECE, Scoul National University
LL AI Research 'Chung-Ang University SSNU-LG AI Research Center (dpf1s589, mi miles94, kjungJêsnu. kc. (hkbae, jeesoo. b elgresearr...cc...mmmmm ai,
ac ac.kr kr
Abstract
They recruit additional membere for their live perior- mances and have toured internsionally In 2014 Paul God: frey resigned from the band. Edwards and Ross Godfrey latar formed Skye Ross and released seif-itted album in Saptember 2016
To address the data scarcity issue Conversa- tional question answering (ConvQA), dialog inpainting method, which utilizes documents generate ConvQA datasets, has been pro- posed. However, the original dialog inpainting model is trained solely on the dialog recon- struction task, resulting in the generation of questions with low contextual relevance due to insufficient learning of question-answe align- ment. To overcome this limitation, propose a novel framework called Dialogizer, which has the capability to automatically generate ConvQA datasets with high contextual rele- vance from textual sources. The framework in- corporates two training tasks: question-anse answer matching (QAM) and topic-aware dialog gen- eration (TDG). Moreover, re-ranking is con- ducted during the inference phase based the contextual relevance of the generated questions. Using our framework, we produce four Con- vQA datasets by utilizing documents from mul- tiple domains as the primary source. Through automatic evaluation using diverse metrics, as well as human evaluation, we validate that our proposed framework exhibits the ability to gen- erate datasets of higher quality compared to the baseline dialog inpainting model.
and have loured internationaly.
Dialog Inpainter
Are there any other interesting sspects about this article?
1.249
Dielogizer
When did Paul Goddrey leave the band?
4.629
Paul Goddrey resigned band.
anyone take over?
1.088
Dislogizer
happoned resigned?
4.035
Edwards and Ross Godfrey later formed Rose released self-litled album September 2018.
Figure 1: Example of questions generated by the pro- posed framework, Dialogizer. Dialogizer excels at gen- crating contextually relevant questions compared to the original dialog inpainter. RQUGE (Mohammadshahi et al., 2022) scores are provided on the right side.
the development of dialog systems. In particular, for informatiion-seeking conversational question- answering (ConvQA) tasks (Stede and Schlangen, 2004; Zaib et al., 2022), creating high-qualing domain-specifer dataset is costly because it requires the direct involvement of domain experts in data annotation (Demszky et al., 2021).
1 Introduction
Dialog systems (Huang et al.. 2020b; Ni et al., 2023) are designed to engage in natural language conversations with users, provide relevant infor- mation, answer queries, and simulate human in- teractions. These systems have gained significant attention in both academics and industry owing to their various potential applications, such as online customer service, virtual assistants, and interac- tive chatbots (Jia, 2004; Ghose and Barua, 2013; Nuruzzaman and Hussain, 2020). However, the scarcity of datasets poses major challenge in
Recent work (Dai et al., 2022) proposes a dialog inpainting method to address this challenge by au- tomatically generating ConvQA datasets using pre- existing text datasets. The text dataset is segmented into sentence-level units, which are directly utilized as answers, while the trained dialog inpainter gen- erates questions corresponding these answers complete the conversation. Dialog inpainting has the potential to address the data scareity issue ow- ing to the abundance of online documents authored by domain experts and the capability to convert
"Equal Contribution.
"Corresponding authors.

---

these documents into dialogs with well-defined an- swers. Considering the guaranteed quality of the answers, is crucial to generate questions that are well-aligned with each corresponding answer (Sun et al., 2018). However, we have observed a low contextual relevance in the questions generated by the dialog inpainter trained solely on a dialog re- construction task, as it lacks sufficient training of question-answer alignment. For instance, as illus- trated in Figure 1, the dialog impainter tends to gen- erate questions that exhibit deficiency in answer specificity (c.g., the green case) or are contextually inappropriate (e.g., the blue case). Through ex- perimental analysis, we quantitatively demonstrate that the questions generated by the dialog inpainter have low RQUGE (Mohammadshahi et al., 2022) scores, indicating lack of contextual relevance.
vance than those generated by the vanilla dialog in- painter. Furthermore, we present an ablation study that shows the effectiveness of each methodology of the proposed framework. Our results represent evidence supporting the potential impact of the Di- alogizer in advancing ConvQA research.
Related Works
2.1 Conversational Question-Answering
Conversational question-answering (ConvQA) aims to enable machines to effectively answer mul- tiple questions from users based on a given passage. ConvQA datasets must contain accurate informa- tion regarding specific domains, maintain consis- tency across topics, and facilitate the progression of dialog turns hierarchically (Zhu et al., 2021). How- ever, creating ConvQA datasets is labor-intensive, as evidenced by the manual annotations throughout existing ConvQA datasets such as CoQA (Reddy et al., 2019), CSQA (Saha et al., 2018), and Con- *Questions (Christmann et al., 2019). In this work, we present framework designed to automatically generate high-quality ConvQA datasets and pro- vide empirical evidence that generated datasets may serve valuable resources for ConvQA tasks.
This study introduces Dialogizer as novel framework for generating contextually relevant ConvQA datasets from textual datasets. The frame- work incorporates two training methodologies, in addition to dialog reconstruction, to address the lim- itation of generating contextually-irreeee irrelevant ques- tions. These methodologies include a question- answer matching (QAM) task and a topic-aware dialog generation (TDG) task. In the QAM task, the model is provided with numerous QA pairs and learns to differentiate between matching and non-matching pairs. This enables the model to dis- cern contextual relevance among QA pairs. In the TDG task, we provide the model with keywords ex- tracted from the target answer using a keyword ex- tractor. Then, the model learns to generate answer- specific questions using these keywords along with the given answer sentence and the dialog history. By incorporating these training tasks, the model becomes capable of generating more specific and answer-relevant questions. Furthermore, during the inference process of generating dialog from the passage, re-ranking is conducted using contextual relevance as a metric, ensuring the generation of high-quality questions.
2.2 Dialog Inpainting
To overcome the data scarcity problem in ConvQA, an automatic ConvQA dataset generation frame- work called dialog inpainting (Dai et al., 2022) has recently been developed. Similar to filling in one side of phone call conversation by overhearing the other side, this methodology considers sentences from text documents as one person's utterances to generate the remaining utterances, thereby com- pleting the conversation. The efficiency of dialog inpainting as a ConvQA dataset generation frame- work is demonstrated by its ability to automatically convert text data into dialog format without loss of information, evidenced by the abundance of high-quality documents annotated by experts in do- mains. However, we have experimentally observed that the questions generated via dialog inpainting lack contextual relevance. In this study, we propose Dialogizer as framework for generating contextu- ally relevant ConvQA datasets.
Using Dialogizer, we compile four ConvQA datasets by leveraging source documents from vari- ous domains, such as news and medicine. Through automatic evaluation using multiple reference- free dialog metrics, human evaluation, GPT-4 evalua- tion (Chiang and Lee, 2023; Liu et al., 2023), and an application text retrieval tasks, we experimen- tally demonstrate that our Dialogizer bialogizer-genrrated datasets exhibit higher quality and context rele-
2.3 Reference-free Dialog Metries
Owing to the one-to-many nature of dialog and question generation tasks (Zhao et al., 2017), reference-based natural language generation met-

---

Dialog Reconstrction
Question-Answer Matching
Topic-Aware Dialog Generation
you say?
Did Alia Bhatl redive any awardss?
Most of the club's income from net gete receipts of E3.733.
TRAINING
said it's lovely day. Let's go walk!
Filifare
MATGH
U
MASKED
Best Actress.
Keywords: (6ki), incomm, net gale receipls)
U.
Come ont litte labor, much health.
Who did Sean Combs start the company with??
NOT MATCH
How much of the club's incomne came from net gate receipts?
Well. foal a litil tired.
Sean Combs American rapper, singer, producer.
Autoregressive Generation
Re-ranking (RR) Process
PASSAGE
DIALOG
ranking
INFERENCE
ROUGE
Re
Candisioe Guestoons
1S,, S L
S., S,
for each timeatep
Figure 2: An overview of the proposed Dialogizer framework, In the training phase, in addition to the dialog recon- struction task, two novel tasks are incorporated: Question-Anse Aaawier Matching and Topic-aware Dialog Generation. During the inference phase, autoregressive question generation is performed using textual data to complete the conversation, and for each question generation, re-ranking is conducted based on the contextual relevance.
rics (Papineni et al., 2002; Lin, 2004) have shown a poor association with human judgment (Liu et al., 2016; Lowe et al., 2017: Gupta et al., 2019). Ad- ditionally, these metrics have limitation they can only be applied in situations where refer- ence is available. Therefore, recent studies have fo- cused on developing reference- free metries (Huang et al., 2020a; Gao et al., 2020; Zhang et al., 2021) that exhibit high correlations with human judgment. Hence, we employ various reference-frr metries to evaluate the quality of generated ConvQA datasets. In addition, the reference-free metrics can be used re-ranking criteria to enhance the generation quality (Wang et al.. 2023b). In this study, we employ a reference-free metric, RQUGE, in the re-ranking process to enhance question generation performance.
gizer employs re-ranking through beam search dur- ing the inference phase, taking into account the con- textual relevance of the generated questions ($3.5). Figure 2 provides illustrative overview of our framework.
3.1 Dialog Reconstruction
Dialogizer is basically trained with the Di- alog Reconstruction (DR) task proposed in Dai et al. (2022), which includes the ran- dom masking of one utterance (t)
(in, u2, ..., 14 Ut+1, UT) in the dialog d (M,, U2, ... u1, ..., ur), with the masked utterance denoted by the symbol e. The objective is re- construct the missing utterance uy Specifically, we utilize the T5 (text-to-text transfer transformer) model (Raffel et al., 2020) to implement Dialo- gizer. Dialogizer can be described as generative model characterized by parameters 6, which define probability distribution py of the target utterance 14 given the masked dialog d (t). The following DR training objective is set:
3 Dialogizer
Dialogizer is novel framework designed gener- ate contextually relevant ConvQA datasets of high quality from textual sources. addition to the simple dialog reconstruction (DR) task ($3.1), 1), the framework incorporates two novel training method- ologies: Question-A Arestion-AAnwer Matching (QAM) ($3.2)
(f)
Σ Eurad logpo(ue ddn(1)) dCD
and Topec- Dialog Generation (TDG) ($3.3).
During the inference phase, the textual passage is segmented into sentences that serve as answers, and the trained model autoregressively generates ques- tions relevant to each answer to complete the Con- vQA dataset ($3.4). Furthermore, to consistently generate stable and high-quality questions, Dialo-
where D is a dialog corpus.
The vanilla dialog inpainting model is trained exclusively in the DR task. However, our observa- tions have confirmed that questions generated by this model exhibit low contextual relevance. Given that the answers are already well-defined as they

---

are extracted from the passage, generating ques- tions that align well with the answer crucial concern. Consequently, we identify two primary factors that contribute to this phenomenon. Firstly, since the DR is simply designed to reconstruct masked utterances in the original dialogs, train- ing the model exclusively on the DR task results in insufficient learning of question-answe answer alignment necessary for ConvQA dataset. Second, the origi- nal model's reliance on document title information as prompt during the inference stage poses chal- lenges determining the specific information that must be conveyed within the generated question. In this work, we aim to overcome these challenges and enhance the contextual relevance of questions in ConvQA dataset generation.
answer matches the question", whereas i is set "The answer does not match the question". These outputs serve as reference labels for the model learn and classify alignment within given question- answer pairs.
3.3 Topic-aware Dialog Generation
In typical question-generatio tasks, the primary objective is to determine what to ask and how to ask (Pan et al., 2019; Ghanem et al., 2022). However, when generating questions through this framework, the answers or contents of the ques- tions are predetermined. Therefore, the model must be able to generate good questions by uti- lizing hints from the predetermined what ask. To address this issue, Dai al. (2022) incorporates the document title within the prompt during the inference phase. However, this approach yields unsatisfactory results as document titles are often excessively abstract. Consequently, the model fails to generate answer-relevant questions, instead pro- ducing overly general questions such "What is [document [ifocumentttitle??
3.2 Question-Answer Matching
To address the challenge of insufficient acquisi- tion of question-answer alignment in vanilla dialog inpainting, we incorporate the Question Answer Matching (QAM) task into the Dialogizer training. This task aims to enhance the model's 's ability to in- terpret long-term dependencics in question-answer pairs.
"Are there any other inter- esting aspects about this article?".
To reduce the reliance on document titles and enable the Dialogizer to generate contextually rele- vant questions, we facilitate knowledge acquisition through the Topic-aware Dialog Generation (TDG) task. We hypothesize that incorporating extracted keywords as hints about what ask would enhance the generation of more specific and answer-relevant questions. To ensure that the extracted keywords are effectively incorporated into the prompt dur- ing the inference phase, we introduce a TDG task during the training phase to train the utilization of extracted keywords.
Similar to the BERT (Devlin et al., 2018) pre- training technique known as next-senience predic- tion, this task focuses the binary classification of matching QA pairs. During the training phase, we utilize a negative sampling method to extract negative answers for specific questions in ConvQA dialogs. This method involves randomly sampling from the same dialog, which is challenging due to the similarity of context. Accordingly, the ob- jective of QAM is to train the model to classify positive answers as MATCH and negative answers NOT MATCH, as shown in Figure? 2.
The TDG loss computed as
Therefore, the QAM training objective is to min- imize the following loss function:
C (0) = Σ
Eggaa (logps(at dm)(),, ki)]
Loau (0) = Σ Ba [oggo((p gt of))
dED
dCD
where kÜ denotes the keywords extracted from the answer. k, is obtained by applying a keyword ex- tractor the answer corresponding to Je and sub- sequently utilized in the prompt with the format " K eyword k,".
+ Hogpo(tN lt: }
where D is a corpus of ConvQA dialogs, a is randomly sampled question from dialog d, af is positive answer corresponding question qt, a; is a negative answer randomly sampled from the dialog d, and (p and ly represent positive and negative target texts, respectively. Given the T5 architecture's representation of input and output as text strings, the target output tp is set to "The
Ultimately, we train our Dialogizer model by ag- gregating the losses using the following approach:
L L dasss Laass + Xppo Lypo

---

Source Dataset
Framework
RQUGE!
USR-DR (c)
USR-DR
OReSScoreLaM
QReSScoreaa
GPT2
baseline
2.7579
0,9270
0,6455
0,4655
0.5077
0,6046
Wikipedia
Dialogizer'
3,8303
0,9883
0.9416
0.5303
0.5893
0.6570
baseline
2.5406
0.9430
0.7170
0,4802
0.4589
0.3958
Pubmed
Dialogizer'
3,4380
0.9908
0.9416
0.5343
0.4982
0.5200
baseline
2.2645
0,8650
0,3380
0,4212
0.5211
0.4326
CC-news
Dialogizer'
3.5748
0,9644
0.8339
0.4887
0.6630
0.4696
buseline
2,4185
0,9592
0,3961
0,4430
0,4007
0.2542
Elsevier
Dialogizer'
3,7803
0,9887
0.8670
0.5402
0.6081
0.4367
Table I: Automatic evaluation results on datasets generated using the baseline dialog inpainting framework and our proposed Dialogizer framework, based on four source datasets. f: utilized for re- re-ranking. WiikiDilog2.. "PubmedDialog. OCC-newsDialog. "ElsevierDialog
3.4 Inference: Autoregressive Generation
model selects the most relevant question to both the passage and the answer based on RQUGE.
The inference process of the Dialogizer frame- work comprises transforming a passage into a Con- vQA dialog. In the inference phase, our Dialo- gizer model fills in the masked utterances in the partial dialog (Sp: ,, $1, $2, ..... a ST) for given passage (51, $2, ..., ST) with prompt 8y using the document title. We then add the keyword prompt "K "Kewword: k(s)"" before the mask token, where k(st) represents the keywords extracted from the answer utterance 8t- The red block in Figure represents the keyword prompt. Additionally, the model generates utterances cuttoregrresivell to avoid discrepancies with the DR approach that gen- erates one utterance at time.
Experiments
We deploy the Dialogizer framework to generate four datasets, validating the quality of the datasets through diverse evaluations. The experimental de- tails are provided in Appendix C.
4.1 Model implementation
We evaluate Dialogizer's performance by compar- ing it to the original dialog inpainting as base- line, ensuring identical conditions. The baseline is implemented using the same backbone model and training dataset as Dialogizer for fair comparison. Both models utilize T5-base (Raffel al., 2020) their backbone. We train both frameworks on two open-domain dialog datasets, namely Daily Dialog (Li et al., 2017) and Task Masker (Byrne et al., 2019), along with two Conv ConvQA datasets, OR-QUAC (Qu et al., 2020) and QReCC (Anantha et al., 2020). For both models, we use four datasets for the dialog reconstruction task. For Dialogzier, also these two ConvQA datasets for QAM and TDG tasks during training. During the TDG training, keyword extraction is performed using the T5-based model developed by Pęzik al. (2022).
Namely, Dialogizer generates â1 with k(s))", ,,, and contin- ues generating ú2 with (sp, ú1, "Keyword k(s2)", ,2)2) ",,)"" to fill in all masks auto-regressively, thereby completing the partial dialog to (sp, ún, $1,,.., ..., uy, s,). The overall inference process can be shown at the bottom of Figure 2.
(sp, "Keyword
3.5 Re-ranking with contextual relevance
In addition, incorporate re-ranking (RR) in the inference phase to improve the contextual relevance of the generated questions. Relying solely on cor- pus statistics to generate the most likely output in the decoding stage does not guarantee contextual quality. To ensure the quality of the generated ques- tions, we opt for re-ranking process based on con- textual relevance. Mohammadshahi et al. (2022) have shown that re-ranking with RQUGE increases contextual relevance and enhances correlation with human judgment in sentence evaluation. Therefore. we utilize RQUGE re-rank candidate questions. Specifically, the model first generates set of k- candidate questions using beam search. Then, the
4.2 Generated Datasets
Using Dialogizer, we generate four ConvQA datasets for use in experiments. These datasets are developed by leveraging four source-text datasets from diverse domains: Wikipedia, PubMed, CC- News (Hamborg et al., 2017), and Elsevier OA CC-By (Kershaw and Koeling, 2020). Each dataset named after corresponding source dataset, namely WikiDialog2, PubmedDialog, CC- newsDialog, and ElsevierDialog. Detailed statistics

---

Human evaluation
GPT-4 Evaluation
Contextual Relevance
0.58
0.23
0.19
0.61
0.22
0.66
Well-ormed
8.75
0.00
0.22
0.37
0.99
8,04
Overall
Quality
0.16
0.03
0.21
0.11
015
0.14
Dielogizer
Baseline dialog inpainter
Figure 3: The human evaluation result comparing two datasets, cach generated by the baseline dialog inpainting and our proposed Dialgozier. Ours obtained positive evaluations across all three criteria (contextual relevance, fell-formednsss, and overall quality), surpassing the baseline.
and sample dialogs for each dataset be found in Appendices A and H.
all metrics. When evaluating the results for five metrics excluding RQUGE, which used for re- ranking, Dialogizer exhibits average performance improvements 18.23% for Wikidialog2, 17.52% for PubmedDialog, 23.66% for CC-newsDialog and 38.80% for ElsevierDialog, The results vali- date that Dialogizer generates datasets of superior quality and enhanced contextual relevance com- pared to the baseline. The exceptional performance of Dialogizer across diverse source datasets from various domains further amplifies its value as a ConvQA dataset-generation framework.
4.3 Automatic Evaluation
Evaluation Metrics For quantitative compari- son, we assess the generated datasets using di- verse reference-free metrics that gauge distinct as- pects of the dialog or generated questions. First, RQUGE is a reference-free metric for question generation that measures the quality of a given can- didate question based on its corresponding answer and relevant passage. Similarly, QRelScore (Wang al., 2022) context-aware evaluation metric for question generation that measures word- and sentence-level relevance without additional train- ing or human supervision. This metric consists of QRelScoreLRM and QRelScoreGRG: the former handles complex reasoning by calculating word- level similarity, while the latter measures factual consistency comparing confidence in generating context. USR-DR (Mehri and Eskenazi, 2020) is dialog evaluation metrie specifically developed to evaluate the aspects of context maintenance, inter- est, and knowledge utilization through a retrieval task. This metric evaluates a dialog using retrieval results from the Ubuntu dialog corpus (Lowe et al., 2015), resulting in two categories: USR-DR(c) in- corporates history and facts, while (SR-DR(f)) re- lies just fact information for context. Pang et al. (2020) also proposed GPTT2 based dialog metric that evaluates context coherence between sentences a dialog.
4.4 Human and GPT-4 Evaluation
To comprehensively evaluate the quality of the questions generated by the baseline and Dialo- gizer, we randomly sample 100 dialogs from the generated datasets and conduct a relative compar- ison through both human and GPT-4. For both evaluations, we use three criteria, i.c., contex- tual relevance, well-formedness, and overall qual- ity, according to the characteristics of the Con- vQA task and existing research (Liang and Li, 2021). Contextual relevance measures the rele- vance of the question the context and answer, folleoommedeess assesses whether the question well-formed, and overall quality measures the overall quality of the context and question-answer pair. Human evaluation involves three crowd work- ers per question. Figure shows that Dialogizer outperforms the baseline across all criteria. Addi- tionally, when utilizing GPT-4 as NLG evalua- tor (Wang et al., 2023a), model consistently out- performs the baseline, confirming its superior per- formance. More detailed information regarding hu- man and GPT-4 evaluations anter-annotator agree- ment, payment details, instructions, and prompts can found in Appendices and G.
Results We perform automatic evaluations by comparing the quality of the datasets produced by the bascline model and Dialogizer using four source datasets demonstrate the effectiveness of the Dialogizer in generating ConvQA datasets. We present the main results in Table 1. Our find- ings demonstrate that the datasets generated by Dialogizer surpass those created by baseline across

---

048
0254
3.0
038
541
A22
(a) NDCG MS MARCO
(b) MARCO
(C) RECALL MS MARCO
nosetiiue
piologizer?
Stoniogtzer
Figure 4: Average values for text retrieval benchmark of Sentence Trasnformer (ST) (Reimers and Gurevych, 2019) models trained on dialog datasets generated by three frameworks: baseline dialog inpainter, Dialogizer without e-ranking(t).. and Dialogizer (Higher is better). Shading indicates standard deviation across five seeds.
DR
QAM
TDG
RR
RQUGE
USR-DR (c)
USR-DR (f)
QRelScore
QReSecoreac:
GPT2
2.7579
0.9270
0.6455
0.4655
0.5077
0.6046
2.7818
0,9437
0.7308
0.4840
0.5643
0.6224
/
2.8732
0,9454
0.7381
0.4823
0.5283
0.6191
2.9228
0.9472
0.7437
0.5003
0,5859
0.6232
3,6590
0,9585
0,7592
0.4895
0.5749
0.6257
3,7095
0,9667
0.8141
0.5165
0,5923
0.6338
/
/
3,7200
0.9753
0.8373
0.5210
0,5793
0.6394
/
/
3.8303
0.9883
0.9416
0.5303
0,5893
0.6570
Table 2: Results of an ablation study examining the impact of QAM, TDG, and Re-ranking (RR) components in the Dialogizer framework on the improvement of automatic evaluation performance. Wikipedia is used source dataset.
4.5 Application Text Retrieval
The retrieval model trained on passage-questee pairs generated by Dialogizer consistently outper- forms those trained using the baseline terms NDCG (Järvelin and Kekäläinen, 2017), Mean Av- erage Precision (MAP), and RECALL. Moreover, when Dialogizer incorporates re-ranking during the inference phase, it generates more relevant ques- tion pairs within the passage, further improving performance. These experimental results confirm that Dialogizer exhibits exceptional proficiency generating questions relevant to a given passage. Experimental details and other benchmark results can be found in Appendix D.
In this section, we verify the alignment of the ques- tions generated by framework with the pas- sage through zero-shot text retrieval task. The coherence of query-passage pairs during training is crucial for improving performance retrieving relevant information without explicit task training, especially in zero-shot scenarios. To gauge the alignment, we train a text retrieval model using the generated questions as queries paired with the orig- inal passage and assess its performance through zero-shot text retrieval benchmark. We consider three experimental variations: baseline dialog in- painting, Dialogizer without re-ranking during the inference phase, and Dialogizer with re-ranking for detailed understanding. These three methods are applied to the Wiki corpus to construct passage- query pair datasets and generate 10k questions as queries for each method. These queries are then matched with their corresponding original passages construct training set for text retrieval.
5
Analysis
5.1 Ablation Study
To gain deeper insight into our method, we con- duet ablation study comparing the quality of datasets generated by applying each methodology Wikipedia. As shown in Table 2, the Dialo- gizer model trained with TDG and QAM generates more contextually relevant questions, indicating that both proposed additional tasks effectively fa-
Experimental results for the Ms-Marco bench- mark (Nguyen et al., 2016) are shown in Figure

---

Question Type
Proportion
CONCEPT
38.49%
Dalogizer
Asking definition explanation concepe
How the coets eolimsted for the MODS and Xport
EXAMPLE
32.91%
Asking for examples concept
Costs were estimated by measuring and valuing relevant resources requiedd to perform the MODS and Xpert lests.
VERIFICATION
15.5458
Seeking confirmation regarding truthfulness of concept
Dislog zer
JUDGEMENTAL
8.57%
How the diegnostic accuracy the Xport tests obtainod?
Requesting the answerer's own opinioes
COMPARISON
4.49%
Disgnostic sccuracy were obtained from systematic reviews involving HIV Intected pationts..
Asking for comparison between maltiple concepti.
Table 3: The distribution of question types in Wikidia- log2, accompanied by explanations of their respective descriptions.
CASE 2
Dillgizar
Who designed Mudgoe railay station?
4,4207
Dislogzzer
was Mudgee railway station butl??
2788
cilitate question-answer answer alignment learning. While TDG contributes slightly more to overall perfor- mance improvement, there are slight variations depending on the evaluation metric focused on assessing the different aspects of dialog quality, For instance, the RQGUE and USR-DR metries, which evaluate relevance by considering context, questions, and answers, indicate that TDG yields greater performance enhancement. In contrast, QRelScore, which evaluates question-ane answer pairs, demonstrates the effectiveness of QAM. The results obtained from training with both TDG and QAM simultaneously indicate that the two approaches synergistically contribute to generating more con- textually relevant data. Furthermore, when per- forming re-ranking based on RQGUE scores dur- ing inference, performance improvements are ob- served across all scenarios and metrics.
Didlgioer
Who buil Mudgee railaay station?
Dislogzer
was bull?
4,2294
Dalogizer
Who was the architecl of Mudgae railway stalion?
4.4819
was designed by John Whitton bull from 1883 to 1884.
Figure 5: Case study. CASE 1 Partial turns of the sam- ple dialog from PubmedDialog. CASE 2 Exploring diversity in generated questions with beam size 5.
5.3 Case Study
We present sample application of Dialogizer in specific domain and the diversity of multiple can- didate questions generated using beam search. Case 1 in Figure 5 presents a sample dialog ex- tracted from PubmedDialog, Creating information- seeking dialog datasets in the medical domain is challenging because of the scarcity experts. However, despite the absence of medical domain data in its training set, Dialogizer has demonstrated its ability to automatically generate high-quality medical dialog datasets. Case 2 in Figure 5 depicts sample candidate questions generated with beam size of 5. These questions focus on various aspects of the answer and exhibit diverse expressions.
5.2 Question Types
We analyze the question-type distribution of open- ended questions in Wikidialog2. By referring to the 18 question types specified by Olney et al. (2012), we construct question type ontology by merg- ing ambiguous types (Cao and Wang, 2021). Ta- ble 3 shows that Wikidialog2 encompasses di- verse range of types, including 38% concepts, 1 examples, and 14% verifications. Regarding the ConvQA characteristic of information seeking, we note that there are relatively fewer judgemental questions that inquire about the respondent's 's opin- ion. Furthermore, the original passage is seg- mented into sentence units construct answers, is inferred that fewer comparison-type questions that require comparing multiple concepts within a single question-ansse -nnweer pair. The experimental details can be found in the appendix E.
Conclusion
This study proposes Dialogizer, framework that enables the automatic generation of high-quality dialog datasets from textual sources. Dialogizer is trained with the tasks of dialog reconstruction, question answer -nnserr ment, and topic-aware dialog generation for spe-
matching for contextual align-

---

cific question creation. Our experimental results demonstrate that Dialogizer produces contextually relevant ConvQA datasets. The proposed frame- work holds promise for advancing ConvQA re- search and its practical applications.
& Evaluation (IITP) grant funded by the Korea governmen((MSIT) (NO.2021-0-013433 Artificial Intelligence Graduate School Program (Seoul Na- tional University) & 100.2021-0-20066 Artificial Intelligence Innovation Hub (Artificial Intelligence Institutc, Scoul National University)], the BK21 FOUR program of the Education and Research Pro- gram for Future ICT Pioneers, Scoul National Uni- versity in 2023, and the National Research Foun- dation of Korea (NRF) grant funded by the Korea government (No. 202 IRI A2C20088855 K. Jung with ASRI, Scoul National University, Korea.
Limitations
Our framework demonstrates improved perfor- mance compared to the bascline in terms of Con- vQA dataset generation; however, it is computa- tionally more expensive due to the inclusion of the beam search during the inference phase. When ting the beam size to 5, the inference time increased by approximately 2.4 times compared to greedy de- coding. Since it is dataset generation framework, the inference time may not be critical in real-world scenarios. However, when aiming to generate large number of datasets for purposes such data augmentation, the inference time should also be taken into consideration as a significant factor.
In addition, unlike the DR task, which can be applied to all dialogs datasets, the novel tasks pro- posed in this study, QAM and TDG, require the ConvQA dataset during the training process. This is because these tasks aim to effectively train the model in acquiring comprehensive understanding of guestion-answee alignment, thus posing the limi- tation that well-matched question-answer pairs are necessary.
Ethics Statement
To verify that the generated datasets do not contain any potential ethical problems, crowd workers were instructed to ascertain that the generated datasets do not include offensive, sexist, or racist comments; toxic language; any instances of sexual behavior. The crowd workers were fairly compensated for their work. Additionally, a detailed description, interface to collect human evaluations, and further details of payment can found Appendix F. Additionally, we utilize the GPT-4 model from official website of OpenAl* for GPT-4 evaluation. All models and datasets used in the experiments are from the publicly accessible website or Github repositories.
Acknowledgements
This work was supported by LG Al Research. This work was partly supported by Institute of Infor- mation & communications Technology Planning
"https://openai

---

References
Bilal Ghanem, Lauren Lutz Coleman, Julia Rivard Dex- ter, Speneer Mcintosh von der Ohe, and Alona Fyshe. 2022. Question generation for reading comprehen- sion assessment by modeling how and what ask. arXiv preprint weXiv:2204.02900
Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre. Stephen Pulman, and Srinivas Chappidi. 2020. Open-domaie question answering goes conversational via question rewriting. arXiv preprint w0xiv:2000.04800
Supratip Ghose and Jagat Joyti Barua. 2013. Toward the implementation of topic specific dialogue based natural language chatbot sor. 2013 international conference on informatics, electronics and vision (ICIEV), pages 1-5. IEEE.
Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In Ad- vances in Information Retrieval: 38th European Con- ference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings 38, pages 716-722. Springer.
undergraduate advi-
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. arXiv preprint arXiv: 10000.........
Bill Byme, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, and Young Kim. 2019. Taskmaster- I: Toward a realistic and diverse dialog dataset, arXiv preprint arXiv: (009.0/99000533 05358.
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017. news-please: A generic news crawler and extractor. In 15th International Sympo- sium of Information Science (ISI 2017), pages 218- 223.
Shuyang Cao and Lu Wang. 2021. Controllable open- ended question generation with a new question type ontology. arXiv preprint wex::ii:10.001111
Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020a. Grade: Automatic graph-enhanced coherence metric for evaluating open-domain dialogue systems. arXiv preprint arXiv:2010.03994.
Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evalua- tions? arXiv preprint wxx::2305.00197.
Minlie Huang. Xiaoyan Zhu, and Jianfeng Gao. 2020b. Challenges in building intelligent open-domain di- alog systems. ACM Transactions on Information Systems (TOIS), 38(3):132..
Philipp Christmann, Rishiraj Saha Roy, Abdalghani Abujahal, Jyotsna Jyotsna Singh, and Gerhard Weikum. 2019.
Look before you hop: Conversational question an- swering over knowledge graphs using, judicious con- text expansion. In Proceedings of the 28th ACM International Conference Information and Knowl- edge Management, pages 729-738.
Kalervo Järvelin and Jaana Kekäläänen. 2017. Ir eval- uation methods for retrieving highly relevant doe- uments. ACM SIGIR Forum, volume 51, pages 243-250. ACM New York, NY, USA.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020. Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07180.
Jiyou Jia. 2004. The study of the application of a web- based chatbot system on the teaching of for- eign languages. In Society for Information Technol- ogy & Teacher Education International Conference, pages 1201-1207. Association for the Advancement of Computing in Education (AACE).
Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Y Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, and Kelvin Guu. 2022. Dialog inpainting: Turning documents into dialogs. In International Conference on Machine Learning, pages 4558 4586. PMLR.
Daniel Kershaw and Rob Koeling, 2020. Elsevier oa cc-by corpus. arXiv preprint 20Xiv22088.00774
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint
Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. 2021. Measuring conversational uptake: A case study on student-teacher interactions. arXiv preprint 106.:2206600873
Richard Landis and Gary Koch. 1977. An appli- cation of hierarchical kappa-type statistics in the as- sessment of majority agreement among multiple ob- servers. Biometrics, pages 363-374.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. Xiviv preprint ar1ivv1810.04805.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-tura dialogue dataset. wrXiv preprint 171.:31710.03957.
Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020. Dialogue response ranking triniigg with scale human feedback data, arXiv preprint arXi: 06978.
Hongru Liang and Huaqing Li. 2021. Towards standard criteria for human evaluation of chatbots: A survey. arXiv preprint arXiv:2105.

---

Chin- Lin. 2004. Rouge: package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Liangming Pan, Wengiang Lei, Tat-Seng Chua, and Min- Yen Kan. 2019. Recent advances in neural question generation. arXiv preprint arXiv: 1905.08949
Chia- Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation met- rics for dialogue response generation. arXiv preprint arXiv: 1oXiv:1003.080 1603.08023.
Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, Kewei Tu, et al. 2020. Towards holistic and automatic evaluation of open-domain dialogue generation.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguisties, pages 311-318.
Yang Liu, Dan Iter. Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human align- ment. arXiv preprint arXvv:2333. 16634.
Piotr Pęzik, Agnieszka Mikotajezyk. Adam Wawrzynski, Bartlomiej Nitoñ, and Maciej Ogrodniczuk. 2022. Keyword extraction from short texts with text-to-text transfer transformer. In Recent Challenges in Intelligent Information and Database Systems: 14th Asian Conference, ACHIS 2022, Ho Chi Minh City, Vietnam, November 28-30, 2022, Proceedings, pages 530-542. Springer.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint
1rXiv:1711.05101.
Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W Bruce Croft, and Mohit lyyer. 2020. Open-retrieval con- versational question answering. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. pages 539-548.
Ryan Lowe, Michael Noseworthy, Julian V Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. arXiv preprint 1rXiv:1788.07144 700.00714.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485 5551.
Ryan Lowe, Nissan Pow, Tulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: large dataset for research in unstructured multi-turn dia- logue systems. arXiv preprint arXiv: 08909.
Shikib Mehri and Maxine Eskenazi. 2020. Usr: An unsupervised and reference free evaluation metric for dialog generation. arXiv preprint ar 2rXiv:2005.0044
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266 2266
Alireza Mohammadshahi, Thomas Scialom, Majid Yaz- dani, Pouya Yanki, Angela Fan, James Henderson, and Marzich Saeidi. 2022. Rquge: Reference- free metric for evaluating question generation by answer- ing the question. arXiv preprint arXiv:2211.01482
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 1908. 10084.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. choice, 2640:660.
Amrita Saha, Vardaan Pahuja, Mitesh Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. Com- plex sequential question answering: Towards learn- ing to converse over linked question answer pairs with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2023. Recent advances in deep learn- ing based dialogue systems: A systematic survey. Artificial intelligence review, 56(4):305--311
Manfred Stede and David Schlangen. 2004. Information-secking chat: Dialogues driven by topic-structure. Proceedings of Catalog (the 8th workshop on the semantics and pragmatics of dialogue; SemDial04).
Mohammad Nuruzzaman and Omar Khadeer Hussain. 2020. Intellibot: A dialogue-based chatbot for the insurance industry. Knowledge-Bass Based Systems, 105810..
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma. and Shi Wang. 2018. Answer-focused and position- aware neural question generation. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3930-3939.
Andrew M Olney, Arthur C Graesser, and Natalie K Person. 2012. Question generation from concept maps. Dialogue & Discourse, 3(2)755-99.

---

David Wadden, Shanchuan Lin, Kyle Lo. Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannanch Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. arXiv preprint arXiv:2004. 200.:2004.497.
Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is chatgpt good nlg evaluator? preliminary study, arXiv preprint wxxi:2303.04044.
Xiaoqiang Wang, Bang Liu, Siliang Tang, and Lingfei Wu. 2022. Qrelscore: Better evaluating gener- ated questions with deeper understanding of context- aware relevance. arXiv preprint arXiv:2204. 13921.
Yihe Wang, Yitong Li, Yasheng Wang, Mi, Pingyi Zhou, Jin Liu, Xin Jiang, and Qun Liu. 2023b. His- tory, present and future: Enhancing dialogue genera- tion with few-shot history-future prompt. In ICASSP 2023-2023 IEEE International Conference on Acous- tics. Speech and Signal Processing (ICASSP), pages 1-5, IEEE
Munazza Zaib, Emma Zhang, Quan Sheng, nan Mahmood, and Yang Zhang. 2022. Conversa- tional question answering: A survey. Knowledge and Information Systems, 6-(12)3115--3155
Chen Zhang, Yiming Chen, Luis Fernando D'Haro, Yan Zhang, Thomas Friedrichs. Grandee Lee, and Haizhou Li. 2021. Dynaeval: Unifying turn and dialogue level evaluation. arXiv preprint 10X.v:2106.01112.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-leve diversity for neural dialog models using conditional variational autoencoders. arXiv preprint IrXiv:1703. 10960.
Fengbin Zhu, Wenqiang Lei, Chao Wang. Jianming Zheng. Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint 10X.v:2101.0077.

---

A Generated Datasets Statistics
220M. The model trains with batch size 8 with gradient accumulation step size and takes about 10 hours per epoch.
Table 4 shows the statistical information for the four datasets generated using Dialogizer, namely WikiDialog2, PubmedDialog, CC-newsDialog, and ElsevierDialog.
We use Adam (Loshchilov and Hutter, 2017) optimizer with 31 = 0.9, 32 0.999, e = le - 8. The max gradient norm for gradient clipping is 1.0. In order to find the best-performing model, we conducted experiments
Dutaset
Source dataset
of Dialogs
Average Turns
hyper-parameter com- binations with 3 epoch steps: AQ^u (0.05, 0.1, 0.5), XTa : (0.05, 0.1, 0.5), per latr__gpuubch____ee baze _ learning_rales (le 4, 5e 5. 2e - 5), warmup_sted : (0, 500). The hyper- parameter was manually tuned, and the best- performing model is with AQAM 0.1, AppO 0.1, per per_gpu_ http_batch_eeeeeeeeeeee
Wikliaalog2
Wikipedia
113,678
9.85
PubmedDialog
Pubined-wriing
22,811
9.24
CC-newsDialog
OC-nows
69,846
10.49
: (1, 2), initial
ElsevierDialogg
Elsevier
32,053
11,37
Table 4: Statistics of the four ConvQA datasets gener- ated using the Dialogizer.
2, initial lertddlllaninngggge rate 5e - 5, and warmup_stn step O. We repeatedly con- ducted all experiments for four seed numbers.
B Reproductability checklists
B.1 Dataset and Source code
D Application to Text Retrieval Details
We provide our experiment source code along with configuration code as supplementary materials. We will publicly release the generated datasets and the full codes with weight parameters.
D.1 Experiment Details
Figure 6 provides detailed explanation of the ex- perimental setup for text retrieval. The baseline di- alog inpainter and Dialogizer model are employed to perform inference the Wiki corpora, result- ing in the creation of the WikiDialog dataset used for training the retrieval model. For Dialogizer, two versions of the WikiDialog dataset are gen- crated based on the re-ranking criterion. Subse- quently, the text retrieval model is trained using the three generated datasets: WikiDialog baseline. WikiDialog Dialogizert and WikiDialog_Diaoogicea Dialogizer- We utilize the Sentence Transformer (ST) (Reimers and Gurevych. 2019) as the retrieval model and train it using the cosine similarity score as the ranking score through the MultipeNNeguiveeRanknngg..
B.2 Computing Resources
Xeon 4210R (2.40 GHz) with RXT A6000 used for the experiments. We use four GPUs for our experiments. All codes are implemented on Python 3.7.13 and PyTorch 1.10.1.
Dialogizer Training Details
C.1 Training dataset
The statistics of the four training datasets used for baseline dialog inpainter and Dialogizer training can be found in Table 5.
Adam W optimizer and perform We also use tuning, as same in the Dialogizer training. The best-performing model is with balch_size 32, initial initial Jearning_rate le 4, warmup_st _ttp 0, and nam_epoche 10. We repeatedly conduct all experiments for five seed numbers and report average values and standard deviation values.
Dataset
of Dialogs
Average # Turns
Daily Dialog
13,118
7.85
Task Master
54,255
19.34
OR-QuAC 5,644
14.36
QReCC
12,219
11.94
Table 5: Statistics of the four training datasets.
D.2 Metrics
We use three metrics for the text retrieval exper- iment: NDCG, MAP, and Recall. First, NDCG (Normalized Discounted Cumulative Gain) is a widely used metric that measures the quality of a ranked list of documents, considering both rele- vance and ranking position. MAP (Mean Aver-
C.2 Training configuration
We use T5-base model as our Dialogizer model. The number of parameters our model is about
hhtpss//hugginggfaeccoottssssa

---

age Precision) focuses on precision different ranks and calculates the average precision across all queries, providing insights into the ability of a system to retrieve relevant documents. Finally, Recall measures the system's ss ability to retrieve all relevant documents, indicating the completeness of the retrieval process.
Inter-Annotator Agreement (IAA) We measure the AnnotaAno Agreement (IAA) among three crowd workers in the human evaluation process. We observe that Krippendorff's a and Cohen's kappa score indicate "substantial" or "moderate" agreement according to the referenced guide- line (Landis and Koch, 1977). The Krippendorff's and Cohen's kappa values for each of the three criteria as follows:
D.3 Benchmarks
addition the Ms-Marco benchmark discussed the main text, we conduct experiments on three additional benchmarks: Scifact (Wadden et al., 2020). Nfcorpus (Boteva et al., 2016), and Sci- docs (Cohan et al., 2020).
Contextual Relevance
Kripependorff alpha: 0.617 (Substantial)
A A-A2 Cohen's 's kappa score: 0.661 (Substantial)
A 1-A3 Cohen's kappa score: 0.613 (Substantial)
A2-A3 Cohen's ' kappa score: 0.537 (Moderate)
Well-formed
D.4 Results
Kripependorff alpha: 0.654(Susstantia))
The results for additional benchmarks in the text re- trieval experiment can be found Figure 7. Similar to MS-MARCO, across all benchmarks, STDialogizer
Al-A2 Cohen's kappa score: 0.599 (Moderate)
A Al-A3 Cohen's kappa score: 0.632 (Substantial)
A2-A3 Cohen's kappa score: 0.522 (Moderate)
demonstrates superior results in all metrics com- pared to STaaseline. indicating that our methodology generates more coherent questions with the pas- sage. Additionally, the effectiveness of re-ranking is further enhanced in these cases.
Overall Quality
Kripependorff alpha: 0.630 (Substantial)
A A1-A2 Cohen's kappa score: 0.599 (Moderate)
A A1-A3 Cohen's 's kappa score: 0.646 (Substantial)
A2-A3 Cohen's ' kappa score: 0.547 (Moderate)
(A1 and A3 stands for Annotatorl, Annotator2,
E Question Types Experiment Details
and Annotator3)
The question type analysis experiment is conducted on the RoBERTa-base (Liu et al., 2019) model, and the training dataset is created by Cao and Wang (2021). The model is trained using the Adam opti- mizer (Kingma and Ba, 2014), and the loss function used is CrossEnntoppyLoss The best-performing classifier model is with per_gpu_balch _ize 4, learning_rate le 5, and num_epoche 5.
G
GPT-4 Evaluation Details
The prompt used for GPT-4 evaluation is devised by referencing Liu et al. (2023), and the input template can be found in Table 6.
H Generated Dialog Examples
Sample dialogs for the four datasets created us- ing Dialogizer (WikiDialog2, PubmedDialog, CC- newsDialog, and ElsevierDialog) can be found in Table 7-14.
F Human Evaluation Details
The recruitment process for the three crowd work- ers for the purpose of human evaluation was conducted via the university's is online community, specifically targeting individuals who possessed flu- ency in the English language. The crowd workers were provided with task definitions, instructions, and examples, as illustrated in Figure 8 and 9. Fur- thermore, they were notified that this evaluation is intended for academic purposes. After conducting the sample evaluation and calculating the required time, the crowd workers were fairly compensated to ensure a minimum hourly wage of S12 or higher, as calculated by the coworkers.

---

DailyDialog
TaskMaster
Training
Training
OR_QuAC
QReCC
Baseline
Dialogizer
Inference
Inference without re-ranking
Wiki corpors
Inference
WikiDlalog by Baseline
WikiDialog by Dialogizer
WikiDialog by Dialogizer
Finetuning
Finetuning
Finetuning
ST Baseline
ST Dialogizer
ST Dialogizer
Zero-shot Evaluation
Scifact
Nfcorpus
Mamacro
Scidos
Dataset
Model
Figure 6: The overview of the text retrieval experiment.

---

0.5
0.0
(a) NDCG Scifact
(b) MAP Seifact
(c) RECALL Scifact
0.0
860
1802
350
170
362
122
365
(d) NDCG- Nfcorpus
MAP -fccrruss
RECALL Nfcorpus
1.40
0.34
420
(g) NDCG Scidocs
(h) MAP- Scidoes
RECALL
Scidcos
ST posstine
ST Biningizer?
niatoggtzr
Figure Average values for text retrieval benchmarks of Sentence Trasnformer (ST) (Reimers and Gurevych, 2019) models trained on dialog datasets generated by three frameworks: baseline dialog inpainter, Dialogizer without reranking(t), and Dialogizer (Higher is better). Shading indicates a standard deviation across five seeds.

---

Conversational Question Anwering Evaluation
Instruction
is task to evaluate the quality a conversational question answering dataset. You will given [context, candidate questions, answer], your task compare the quality the candidate questions based on four criteria: contextual rolevance, well formness, fluency, overall quality. Please read the instructions carefully and make sure you understand them before proceeding with task. you have any questions, feel free before continuing with the Please select equal only when is difficult judige.
Contextual whether question relevant answer/context
Woll-formednes:: whether the question is well-formed
Overall quality: overall quality of question
Context: Another spring flowed the embankment the Robles Street exit
101. Other unreinforced masonry buildings, some more than century old, in the city's historic downtown area also had extensive damage, However, none the buildings callapped. There wiongful dealh lavesui filed the rellsl of (he women killed in the earthquake against Mary Mastagni, and several trusts which owned the Acorn Building. The jury found Mastagni negligent in the care maintonance in violation of city ordinances. the jury awarded nearly $2 million the plaintiffs.
even partial retrofitting
Acorn Building, due to not rotrofitting building,
Question A: there wrongful death lawsult filed against Mary Mastagni and several trusts which owned the Acom Building?
Question B: What happened after the 2003 San Simeon earthquakee?
Answer: There was wrongful lawsuit filed by the relatives women killed earthquake against Mary Mastagni, and several trusts which owned the Acorn Building.
Figure 8: Interface of human evaluation comparing the two methods: vanilla dialog inpainting and Dialogizer. (1/2)

---

Which question more relevant to the given answer?
Undecided
Question
Equal
Question
Which question well-formed??
Undecided
Question A
Equal
Question
Which question has better qerall-quality?
Undecided
Question
Equal
Question
Does the whole text (passage, questions, answer) has any potential risk? Potential risk means: (1. offensive, senist, toxic words, sexual behavior.)
Yes
No
you answered 'Yes,, please provide brief explanation.
Next Question
order proceed to the next question, you must answer all the questions.
Figure 9: Interface of human evaluation comparing the methods: vanilla dialog inpainting and Dialogizer. (2/2)

---

This is task to evaluate the quality of conversational question answering dataset. You will be given [context, two candidate questions, answer], and your task is to compare the quality of the candidate questions based on four criteria: contextual relevance, well- formedness, fluency, overall quality. For each criteria, answer which question is belter.
Contextual Relevance: whether the question relevant to the answer/context
Well-formednes:: whether the question well-formed
Overall Quality: overall quality the question
Context:
Question A:
Question B:
Answer:
Choose the question which is more relevant to the given answer.
options: [Question A, Equal, Question B]
Choose the question which more well-formed?
options: [Question A, Equal, Question B]
Choose the question which has better overall-quality.
options: [Question A, Equal, Question
Table 6: The template of the prompt used for GPT- evaluation.

---

USER:
Where is Grevillea rudis found?
AGENT:
Grevillea rudis Grevillea rudis is a shrub of the genus "Grevillea" native area along the west
coast in the Wheatbelt region of Western Australia.
USER:
How tall is the shrub?
AGENT:
The loose, spreading erect shrub typically grows a height of and has non-glaucous branchlets.
USER:
How leaves of the shrub Grevillea rudis look like?
AGENT:
has simple flat, spathulate, irregularly lobed leaves with blade is long and wide. How often does the shrub Grevillea rudis bloom?
USER:
AGENT:
It blooms sporadically throughout the year and produces a terminal raceme regular inflorescence with
cream or yellow flowers and white or cream styles.
USER:
What kind of fruit does the shrub Grevillea rudis produce?
AGENT:
Later it forms obovoid ellipsoidal glandular hairy fruit that is long.
USER:
How does the shrub Grevillea rudis regenerate?
AGENT:
will regenerate from seed only.
Table The first sample dialog extracted from WikiDialog2. The utterances generated by Dialogizer are expressed in bold.
USER:
Why does Freeview use DVB-T ODFM standard?
AGENT:
Freeview uses DVB-T ODFM standard for terrestrial transmission, as established 2001
NZS6610:2001,, to avoid the multipath problem caused by New Zealand's rugged topography. What is the rival US standard that 8-VSB modulation?
USER:
AGENT:
ATSC, the rival US standard that 8-VSB modulation, had number of first gen.
USER:
What the first gen demodulators?
AGENT:
demodulators that couldn't handle multipath well, it chosen.
USER:
What is terrestrial Freeview broadcast in?
AGENT:
Terrestrial Freeview is broadcast in H.264, which unlike H.262 has expensive transmission patent
licensing tax for free TV and subscription use.
USER:
Did people who took part in the Auckland digital trial using terrestrial I1.262 receivers have to change their receivers to more expensive H.264 models in order to receive terrestrial Freeview?
AGENT:
People who took part in the Auckland digital trial using terrestrial H.262 receivers needed to change their receivers to more expensive H.264 models in order to receive terrestrial Freeview.
Table 8: The second sample dialog extracted from WikiDialog2. The utterances generated by Dialogizer are expressed in bold.

---

USER:
How many hospitalizations does Orthostatic Hypotension cause?
AGENT:
Orthostatic_ Orthostati__hyootensiio hypotension causes = 80000 hospitalizations in the United States
USER:
What are treatment options for Orthostatic Hypotension?
AGENT:
Treatments for Orthostatic hyyoottni__yoottniioon include Fludrocorissoee, mineralocoricoid analog that promotes
sodium reabsorption and midodrine,
udrenergic agonist direct vasoconstrictor
USER:
How safe are both medications for Orthostatic Hy Hypotension?
AGENT:
Although both iedications are used to treat Orthostatic cethostatic_hyypeee hypotension. few studies have compared their relative safety
USER:
Are there any other interesting aspects about this article?
AGENT:
compared incidence rates bospitalizations all causes for congestive heart failure between users Fludrocortisone and users of midodrine retrospective cohort study of Tennessee Medicuid adult enrollees
(199--2009)
USER:
How were the adjusted incidence rate ratios calculated?
AGENT:
Adjusted incidence rate ratios were calculated using negative binomial regression models.
Table The first sample dialog extracted from PubmedDialog, The utterances generated by Dialogizer are expressed in bold.
USER:
What osteoarthritis?
AGENT:
Ostecarthritis is degenerative joint disorder of articular cartilage and the most common type of
arthritis elderly.
USER:
Is hydroxychloroquine used to treat osteoarthritis?
AGENT:
There are only reports regarding the use of hydroxychloroquine treatment Ostecarthritis
USER:
What the effect of hydroxychloroquine symptoms of knee- Osteoarttttrt
AGENT:
To investigate the effects of hydrroccchloroquinee on the symptoms of mild to moderate knee, Ostenarthritss
Kellgren and Lawrence grade II and III performed double-blind placebo-controlled study in patients.
USER:
How many hydroxychlorqquine pills did the 44 patients receive?
AGENT:
The patients were randomly assigned two groups one group received hydroxychloraquine pills
(200 twice daily and the other group received placebo pills.
Table 10: The second sample dialog extracted from PubmedDialog. The utterances generated by Dialogizer are expressed in bold.

---

USER:
How do write Bug-free software?
AGENT:
Writing bug-free software is practically impossible, due to the inpracticality of predicting every way which code might be executed.
USER:
the hidden flaws the underlying programming language that exploited by hackers?
What
AGENT:
But even if developers go above and beyond to avoid flaws that cun be exploited by backers, uttackers can often advantage vulnerabilities design underlying programming language. What happened the recent Black Hat Europe conference?
USER:
AGENT:
At the recent Black Hat Europe conference, I0Active security services revealed it had identified flaws in five major, interpreted programming languages that could he used hy hackers crafting an attack.
USER:
What are the interpreted programming languages vulnerabilities?
AGENT:
"With regards to the interpreted programming languages vulnerabilittis, software developers may unknowingly
include code in an application that can be used way that the designer did not foresee," it writes.
Table II: The first sample dialog extracted from CC-newsDialog. The utterances generated by Dialogizer are expressed in bold.
USER:
How many doctors are freedd; strike continues?
AGENT:
Kenyan has relcased doctors who are officials the medies' union and who were
jailed earlier this week for not calling off a strike by doctors working public institutions.
USER:
When did the court free the seven?
AGENT:
The decision free the seven was made Wednesday by three judges of the appellate court.
USER:
How did the public react to the release of the seven?
AGENT:
About 1,000 doctors outside the court celebrated the officials' release and held peaceful march to Parliament and Nairobi's Freedom Park.
USER:
Why are the doctors on strike?
AGENT:
More than 5,000 doctors from public hospitals strike over pay and to protest Kenya's dilapidated health care system.
USER:
What did the health minister say about the release?
AGENT:
Health minister Dr. Cleopa Mailu told the Senate committee for health that he had agreed to the
release officials.
USER:
What did the union say about the release of the officials?
AGENT:
The union had said that negotiations would be held until the freed.
Table The second sample dialog extracted from CC-newsDialog, The utterances generated by Dialogizer are expressed in bold.

---

USER:
What biomarker for anticoagulation?
AGENT:
There is clinical need for lahoratory biomarker to identify patients who, following
unprovoked venous thrombosis (VTE), at low VTE recurrence risk and can discontinue anticoagulation after limited treatment duration (3-6 m).
USER:
What a secondary analysis the ExACT study?
AGENT:
This secondary analysis the ExACT study aimed evaluate whether quantitation peripheral
blood endothelial progenitor cells (EPCs) could improve prediction VTE recurrence risk.
USER:
Was the FxACT study non-blinded, multicentre RCT?
AGENT:
The ExACT study was non- ml-bbinded, multicentre RCT comparing extended vs discontinued anticcagulation following a first unprovoked VTE.
USER:
Who was eligible for the study?
AGENT:
Adult patients eligible if they completed months anticnagulation and remained anticoagulated.
USER:
What the primary outcome?
AGENT:
The primary outcome was time to first recurrent VTE from randomisation.
USER:
How long did the study follow up?
AGENT:
Blood samples were taken at buseline and results correlated with climical outcome over 2 years follow
Table 13:
The first sample dialog extracted from EllsvierDDilogg The utterances generated by Dialogizer are expressed in bold.
AGENT:
B cells constitute on essential line of defense from pathogenic infections through the generation of class switched antibody--serettte cells (ASCs) in germinal centers.
germinal center reactions?
AGENT:
Although this process known to be regulated by follicular helper T (TTH) cells, the mechanism by which B cells initially seed germinal center reactions remains clusive.
USER
What is the role NKT cells in inmunity?
AGENT:
found NKT colls, a population innate-like T lymphocytes. are critical for induction of cell immumity vinal infection.
USER:
How B cells priming by resident macrophages work?
AGENT:
The positioning NKT cells interfollicular areas of lymph nodes facilitutes
priming by resident macrophages and the localized delivery of innate signals to unfigen-experienced B cells.
USER:
How many plodduiug cells NKT cells?
AGENT:
Indeed, NKT cells secrete early wave
constitute up
producing cells.
during the initial stages of infecctio.
USER:
How is the requirement of this innate immunity conserved in Maka-virus-infecter macaques?
AGENT:
Importantly, requirement of this innate immunity arm appears evolutionarily conserved because early NKT
and IL-4 gene signatures also positivoly correlate with the levels of neutralizing antibodics in Zika-vinusinnfeteed macaques.
Table 14: The second sample dialog extracted from ElsevierDialog. The utterances generated by Dialogizer are expressed in bold.


Reviews:
[{"review_id": "1d9xgC1OOj", "reviewer": "Reviewer_d7bR", "summary": "", "questions": "1. Since the focus of the proposed method is on ConvQA,  why evaluate on general non-conversational retrieval, unlike previous dialog inpainting work (Dai et al 2022)?\n2. For the motivation of the paper, in the intro and in 3.1 it is stated that \"from observation\" the original inpainting method produces less relevant questions. In the intro it is then stated that the baseline has low RQUGE. However this is compared to the proposed method. Is it possible to quantify this without comparing to the proposed method (as this is the motivation)? This way the motivation would be more convincing.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank you for the thorough feedback on our study.\nWe will offer explanations, individually addressing the queries and questions you hold.\n\n### **Response for Reason to Reject 1 - Performance enhancement by re-ranking process**\nOur research aims to establish a robust framework for generating high-quality ConvQA datasets, wherein the re-ranking process during the inference stage stands as a component of our devised framework, alongside QAM and TDG, aimed at enhancing the quality of the generated questions. Furthermore, upon examining the outcomes of metrics (besides RQUGE [1]; directly in the re-ranking process), the performance of the results obtained by the simple re-ranking and the QAM + TDG demonstrates a similarity in their effectiveness compared to the baseline(see 2.  vs 3. in the table below), showcasing comparable levels of achievement. In addition, it is noteworthy that the combined application of re-ranking and QAM + TDG synergistically yielded additional performance enhancements(see 4. in the table below).\n\n|            | USR-DR (c) | USR-DR (f) | QRelScore(LRM) | QRelScore(GRG) | GPT2    |\n|------------|------------|------------|--------------|--------------|---------|\n| 1. baseline   | 0.9270     | 0.6455     | 0.4655       | 0.5077       | 0.6046  |\n| 2. QAM+TDG | 0.9472     | 0.7437     | **0.5003**       | **0.5859**       | 0.6232  |\n| 3. simple RR | **0.9585**     | **0.7592**     | 0.4895       | 0.5749       | **0.6257**  |\n| 4. QAM+TDG+RR | 0.9883     | 0.9416     | 0.5303       | 0.5893       | 0.6570  |\n\n(This table has been excerpted from Table 2 in the paper.)\n\n### **Response for Reason to Reject 2 - Needs for human/LLM evaluation for the proposed method without re-ranking**\nAs previously mentioned, re-ranking, being a component of the framework proposed in this paper, we conducted human and LLM evaluations with a dataset that integrates the entirety of the proposed framework. However, in alignment with your assertion, conducting a supplementary experiment by contrasting the dataset generated solely through the application of TDG+QAM with the Dialog Inpainter[2] could provide further insights through additional analysis. Therefore, we proceeded to conduct the following supplementary experiments.\n\nDue to time constraints, human evaluation could not be included in this rebuttal submission. The outcomes of the LLM evaluation comparing the TDG+QAM and the baseline can be observed below. The experimental settings remained consistent with those detailed in the paper. \n```\nDialogizer (with re-ranking) vs. baseline - Contextual Relevance\n0.61 (D) 0.22(E) 0.16(B)\nDialogizer (without re-ranking) vs. baseline - Contextual Relevance\n0.52(D) 0.24(E) 0.22(B)\n\nDialogizer (with re-ranking) vs. baseline - Well-formed\n0.37(D) 0.59(E) 0.04(B)\nDialogizer (without re-ranking) vs. baseline - Well-formed\n0.35(D) 0.60(E) 0.05(B)\n\nDialogizer (with re-ranking) vs. baseline - Overall Quality\n0.71(D) 0.15(E) 0.14(B)\nDialogizer (without re-ranking) vs. baseline - Overall Quality\n0.60(D) 0.19(E) 0.21(B)\n```\n(where D stands for Dialogizer, E stands for Equal, and B stands for Baseline[2])\n\nThe model trained with QAM+TDG, while not surpassing that of Dialogizer with re-ranking, still showcases a significant enhancement compared to the baseline. Comparing the two versions of Dialogizer (with and without re-ranking), it can be observed that the version without re-ranking exhibits a slightly reduced degree of performance improvement in terms of Contextual Relevance and Overall Quality while showing nearly similar results in Well-formedness. This can be attributed to the metric used for re-ranking, RQUGE, which considers contextual relevance.\n\nIt is deemed that the inclusion of the aforementioned experimental results and analysis in the final version would provide readers with additional insights.\n\n### **Response for Question 1 - Question about application to text retrieval task (Section 4.5)**\n\nIn contrast to the approach taken by [2](dialog inpainter), where the focus lay on proposing the passage to dialogue framework and demonstrating its utility by conducting experiments to establish the dataset's effectiveness in the context of ConvQA, our endeavor was driven by the objective of enhancing the quality of the generated ConvQA dataset. As such, we structured our experiments to validate this pursuit. Particularly, in order to assess the degree of answer relevance exhibited by the generated questions, we conducted experiments by applying them to a retrieval task[3]. The experimental setup of [2] involves generating an extensive amount of ConvQA data for pretraining to showcase performance enhancements, which certainly demonstrates the utility of the dataset. However, it falls short of focusing on confirming the quality of the generated questions.\n\n### **Response for Question 2 - Question about the motivation of this work**\n\nThe motivation behind this research was driven by the belief that the study conducted by [2] held great promise for advancing the field of ConvQA. However, upon personally examining the WikiDialog dataset released in their paper, the quality of the generated questions was not as high as initially anticipated. Upon manually examining a corpus of 100 sample question-answer pairs, we observed that more than 30 questions demonstrated a deficiency in maintaining a satisfactory level of context relevance. For instance, as demonstrated by the examples below, a substantial number of questions lacking contextual relevance were observed (drawn from various instances of WikiDialog). Also, overly general questions such as 'What is {document_title}?' or 'Are there any other interesting aspects about this article?' recur persistently.\n\n```\nQ: Do any other characters die?\nA: Mary Alice Young (Brenda Strong) is seen in this episode.\n```\n```\nQ: Who did she manage to beat?\nA: Jennifer Maguire is a 27-year-old marketing consultant, originally from Dublin, who said that when she first ran an office, she made £60,000 in six months with no training.\n```\n```\nQ: What are some of the songs on the album?\nA: While the album was praised for its top production quality, complaints were often made about the overuse of either vocoders or the auto-tune software.\n```\n```\nQ: what is the difference between common ground and joint ventures?\nA: \" Finding common ground\" is a technique people use to facilitate interpersonal relationships.\n```\nTherefore, we proceeded to quantitatively assess the dataset using the RQUGE, a reference-free metric known for its strong correlation with human judgment. WikiDialog achieved an approximate score of 2.7. Given that the range of RQUGE spans from 0 to 5 points, this result suggests considerable room for enhancement from the 2.7 score. Furthermore, building upon the evidence of identified subpar samples, we embarked upon a research endeavor focused on developing a higher-quality ConvQA dataset generation framework.\n\nFor the purpose of reference, we provide examples of questions generated by **Dialogizer** utilizing the same passages of the aforementioned examples. \n```\nQ: Who is seen in this episode?\nA: Mary Alice Young (Brenda Strong) is seen in this episode.\n```\n```\nQ: How did Jennifer Maguire start her career?\nA: Jennifer Maguire is a 27-year-old marketing consultant, originally from Dublin, who said that when she first ran an office, she made £60,000 in six months with no training.\n```\n```\nQ: Was there any criticism of the album?\nA: While the album was praised for its top production quality, complaints were often made about the overuse of either vocoders or the auto-tune software.\n```\n```\nQ: What is finding common grounds?\nA: \" Finding common ground\" is a technique people use to facilitate interpersonal relationships.\n```\n\n### **In conclusion,**\n\nonce again, we greatly appreciate your detailed comments. If your issues have been adequately resolved, we would be grateful if you could contemplate accepting our paper. Otherwise, please let us know about any remaining concerns you may have. We will revise the paper in the final draft according to your comments, including suggestions for improvement, missing references, and typos.\n\n[References]\n\n[1] RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question (https://aclanthology.org/2023.findings-acl.428) (Mohammadshahi et al., Findings 2023)\n\n[2] Dai, Zhuyun, et al. \"Dialog inpainting: Turning documents into dialogs.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Mao, Yuning, et al. \"Generation-augmented retrieval for open-domain question answering.\" arXiv preprint arXiv:2009.08553 (2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Proposed method improves over vanilla inpainting, and experiments are quite thorough.", "reasons_to_reject": "It seems from ablation study (table 2) that a lot of the gain (for automatic evaluation) can be attributed to re-ranking the final QA pair candidates with RQUGE, not from the contribution made by this paper (QAM and TDG). Human/LLM evaluation for the proposed method without re-ranking was not reported. \nMissing discussion and comparison against previous work on conversational question generation.", "questions_to_authors": "1. Since the focus of the proposed method is on ConvQA,  why evaluate on general non-conversational retrieval, unlike previous dialog inpainting work (Dai et al 2022)?\n2. For the motivation of the paper, in the intro and in 3.1 it is stated that \"from observation\" the original inpainting method produces less relevant questions. In the intro it is then stated that the baseline has low RQUGE. However this is compared to the proposed method. Is it possible to quantify this without comparing to the proposed method (as this is the motivation)? This way the motivation would be more convincing.", "paper_topic_and_main_contributions": "This paper aims to improve dialog inpainting, a conversational QA pair generation method, with the goal of generating questions that are more relevant to the conditioning answers. The proposed method generally follows dialog inpainting, training with dialog reconstruction, while introducing two mechanisms to guide generation: question-answer matching, to discern relevant QA pairs; and topic-aware dialog generation, conditioning on extracted keywords. Automatic metrics, human and LLM evaluations were performed, and application to text retrieval tasks were also shown. The main contributions of this paper are:\n1. Proposed an improved dialog inpainting method, using two additional mechanisms to guide conversational QA pair generation in order to obtain higher quality and more relevant questions.\n2. Showed the inpainted datasets can be utilized for application to text retrieval tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 3 caption dialgozier, Figure 4 caption trasnformer"}, "extra_text": {"Missing_References": "2.2 only discusses a single related work (Dai et al 2022). There's a line of previous work on conversational question generation or dialog generation for data augmentation, which should be discussed in this section, not limited to the papers below:\n\nKim et al, 2022, Generating Information-Seeking Conversations from Unlabeled Documents\nGao et al, 2019, Interconnected Question Generation with Coreference Alignment and Conversation Flow Modeling\nGu et al, 2021, ChainCQG: Flow-Aware Conversational Question Generation"}}, {"review_id": "N8E6FvrgMB", "reviewer": "Reviewer_7ih2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough examination of our paper. Your considerate evaluation, along with the constructive feedback and favorable appraisal of our efforts, is truly valued. We will provide responses to the inquiries you've requested and the questions you've posed.\n\n### **Response for Reason to Reject 1 - More detailed analysis of the dataset quality**\n\nAs you have pointed out, we have endeavored to assess the Dialogizer through both quantitative and qualitative evaluations. In addition, we conducted an analysis of the generated datasets through the **Section 5 Analysis**. In **5.1 Ablation Study(Table 2)**, we conducted an analysis of performance enhancements based on various methodologies we proposed. And in **5.2 Question Types(Table 3)**, we examined the distribution of question types in the generated dataset, WikiDialog2. Also, through **5.3 Case Study** and **Appendix H Generated Dialog Examples**, we strived to effectively showcase the quality of the datasets created using the Dialogizer.\n\n### **Response for Reason to Reject 2 - More examples that compare two methods**\n\nWe will provide a response to your inquiry regarding your request for further examples comparing dialogs generated by the Dialogizer and the Dialog Inpainter. The original paper's **Appendix H Generated Dialog Examples** contains instances of data created with the Dialogizer, and in addition, here we present the results of transforming the passages used in these examples into dialogs using the Dialog Inpainter. The first examples from each of the four source datasets are subjected to comparison. (Tables 7, 9, 11, and 13)\n\n**Configuration of the provided dialog examples**\n\nFor a direct comparison with the Dialog Inpainter, in the case of the Wikipedia source dialog dataset, we examine the dialogues generated by WikiDialog as provided by [1], the dialogues generated through our implementation of the Dialog Inpainter, and the WikiDialog2 produced using the Dialogizer. (Please note that the initial Dialog Inpainter implementation from [1] employs T5-XXL, whereas our implementations for both the Dialog Inpainter and Dialogizer utilize T5-base as the underlying backbone model.) For the remaining source datasets (Pubmed, CC-news, and Elsevier), we provide a comparison between the datasets generated using our implemented Dialogizer and the Dialog Inpainter.\n\n**A brief interpretation of the provided dialog examples**\n\nFrom the examples below, it can be seen that the dialog data generated using the Dialogizer exhibits a higher degree of context relevance than those generated by Dialog Inpainter. Especially noteworthy is the case of WikiDialog; it is worth emphasizing that despite being generated by the Dialog Inpainter, which is implemented with T5-XXL as the backbone model, the Dialogizer utilizing T5-base as its backbone has demonstrated the ability to create ConvQA data of higher quality. Furthermore, as we have indicated in the paper, within the WikiDialog example (1-1), it can be observed that \"What is {Document_title}?\" (in this instance, the document title is \"Grevillea Rudis\") and the question like \"Are there any other interesting aspects about this article?\" are generated, illustrating the generation of overly general questions. \n\nAs you have suggested, incorporating such additional comparative examples into the appendix could offer readers further insights and potentially enhance the credibility of the study. Your valuable feedback is greatly appreciated. \n\n### **In Conclusion,**\n\nwe would like to express our sincere gratitude for your comprehensive review of our paper. We will revise the paper according to your comments, including the inclusion of additional examples in the camera-ready version.\n\n___\n\n**1-1. Framework: Dialog Inpainter implemented by [1] / Source Dataset: Wikipedia**\n```\nQ: What is Grevillea rudis?\nA: Grevillea rudis Grevillea rudis is a shrub of the genus \"Grevillea\" native to an area along the west coast in the Wheatbelt region of Western Australia.\nQ: Are there any other interesting aspects about this article?\nA: The loose, spreading to erect shrub typically grows to a height of and has non-glaucous branchlets.\nQ: What about the leaves?\nA: It has simple flat, spathulate, irregularly lobed leaves with a blade that is long and wide.\nQ: Does it have any flowers?\nA: It blooms sporadically throughout the year and produces a terminal raceme regular inflorescence with cream or yellow flowers and white or cream styles.\nQ: What about fruit?\nA: Later it forms obovoid or ellipsoidal glandular hairy fruit that is long.\nQ: Can you tell me how to propagate?\nA: It will regenerate from seed only.\n```\n**1-2. Framework: Dialog Inpainter implemented by this work / Source Dataset: Wikipedia**\n```\nQ: What's Grevillea rudis?\nA: Grevillea rudis Grevillea rudis is a shrub of the genus \"Grevillea\" native to an area along the west coast in the Wheatbelt region of Western Australia.\nQ: How big is Grevillea rudis?\nA: The loose, spreading to erect shrub typically grows to a height of and has non-glaucous branchlets.\nQ: What does Grevillea rudis have?\nA: It has simple flat, spathulate, irregularly lobed leaves with a blade that is long and wide.\nQ: Are there any other interesting aspects about this article?\nA: It blooms sporadically throughout the year and produces a terminal raceme regular inflorescence with cream or yellow flowers and white or cream styles.\nQ: Does Grevillea rudis produce fruit?\nA: Later it forms obovoid or ellipsoidal glandular hairy fruit that is long.\nQ: Are there any other interesting aspects about Grevillea rudis?\nA: It will regenerate from seed only.\n```\n**1-3. Framework: Dialogizer implemented by this work / Source Dataset: Wikipedia**\nPlease see Table 7 in the paper.\n\n**2-1. Framework: Dialog Inpainter implemented by this work / Source Dataset: Pubmed**\n```\nQ: What is Orthostatic_hypotension?\nA: Orthostatic_hypotension causes ≈ 80000 hospitalizations per year in the United States .\nQ: What is the treatment for Orthostatic_hypotension?\nA: Treatments for Orthostatic_hypotension include Fludrocortisone , a mineralocorticoid analog that promotes sodium reabsorption ; and midodrine , an α-1 adrenergic agonist that is a direct vasoconstrictor .\nQ: Are there any other interesting aspects about this article besides Fludrocortisone, a mineralocorticoid analog that promotes sodium reabsorption, and midodrine, an 1 adrenergic agonist that is a direct vasoconstrictor?\nA: Although both medications are used to treat Orthostatic_hypotension , few studies have compared their relative safety\nQ: Are there any other interesting aspects about this article?\nA: We compared incidence rates of hospitalizations for all causes , and for congestive heart failure between users of Fludrocortisone and users of midodrine in a retrospective cohort study of Tennessee Medicaid adult enrollees( 1995–2009 ) .\nQ: Are Fludrocortisone and midodrine safe?\nA: Adjusted incidence rate ratios were calculated using negative binomial regression models.\n```\n**2-2. Framework: Dialogizer implemented by this work / Source Dataset: Pubmed** \nPlease see Table 9 in the paper.\n\n**3-1. Framework: Dialog Inpainter implemented by this work / Source Dataset: CC-news**\n```\nQ: How can I help you?\nA: Writing bug-free software is practically impossible, due to the impracticality of predicting every way in which code might be executed.\nQ: Are there any programming languages with hidden flaws vulnerable to hackers?\nA: But even if developers go above and beyond to avoid flaws that can be exploited by hackers, attackers can often still take advantage of vulnerabilities in the design of the underlying programming language.\nQ: Are there any other interesting aspects about the five programming languages with hidden flaws vulnerable to hackers?\nA: At the recent Black Hat Europe conference, IOActive security services revealed it had identified flaws in five major, interpreted programming languages that could be used by hackers in crafting an attack.\nQ: What are the five programming languages with hidden flaws vulnerable to hackers?\nA: \"With regards to the interpreted programming languages vulnerabilities, software developers may unknowingly include code in an application that can be used in a way that the designer did not foresee,\" it writes.\n```\n**3-2. Framework: Dialogizer implemented by this work / Source Dataset: CC-news**\nPlease see Table 11 in the paper.\n\n**4-1. Framework: Dialog Inpainter implemented by this work / Source Dataset: Elsevier**\n```\nQ: What can you tell me about Anticoagulation?\nA: There is clinical need for a laboratory biomarker to identify patients who, following an unprovoked venous thrombosis (VTE), are at low VTE recurrence risk and can discontinue anticoagulation after a limited treatment duration (3–6 m).\nQ: What is the secondary analysis of the ExACT study?\nA: This secondary analysis of the ExACT study aimed to evaluate whether quantitation of peripheral blood endothelial progenitor cells (EPCs) could improve prediction of VTE recurrence risk.\nQ: What was the primary analysis of the ExACT study?\nA: The ExACT study was a non-blinded, multicentre RCT comparing extended vs discontinued anticoagulation following a first unprovoked VTE.\nQ: Are there any other interesting aspects about this article?\nA: Adult patients were eligible if they had completed ≥ 3 months anticoagulation and remained anticoagulated.\nQ: How was the primary outcome of the ExACT study?\nA: The primary outcome was time to first recurrent VTE from randomisation.\nQ: What else is about ExACT study?\nA: Blood samples were taken at baseline and results correlated with clinical outcome over 2 years follow up.\n```\n**4-2. Framework: Dialogizer  implemented by this work / Source Dataset: Elsevier**\nPlease see Table 13 in the paper.\n\n[References]\n\n[1] Dai, Zhuyun, et al. \"Dialog inpainting: Turning documents into dialogs.\" International Conference on Machine Learning. PMLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper's results demonstrate that Dialogizer-generated datasets exhibit higher quality and context relevance compared to baseline methods. This finding indicates the potential impact of the proposed framework in advancing ConvQA research and potentially improving the performance of dialog systems in practical applications.\n\n2. The paper acknowledges the limitation of generating contextually-irrelevant questions using traditional dialog inpainting methods. By proposing Dialogizer and incorporating specific training tasks to address this limitation, the paper demonstrates an effort to overcome challenges faced by existing approaches.\n\n3. The paper supports its claims with both quantitative evidence, such as reference-free dialog metrics, and qualitative evidence through human evaluation. The combination of these methods enhances the credibility of the research outcomes.", "reasons_to_reject": "1. While the paper proposes a novel framework called \"Dialogizer\" for generating contextually relevant ConvQA datasets, the contribution may be seen as incremental rather than transformative. The paper builds upon existing dialog inpainting methods and introduces additional training tasks to address the limitation of contextually-irrelevant questions.\n\n2. The paper focuses solely on the task of generating ConvQA datasets and does not explore the broader impact of the proposed framework on dialog systems or its potential applications beyond dataset generation.\n\n3. The paper claims that Dialogizer-generated datasets exhibit higher quality and context relevance. However, the paper lacks a detailed analysis of the dataset quality and potential biases introduced by the automatic generation process. Are there more examples that compare the proposed datasets with the prior existing ones?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper's topic is about addressing the data scarcity issue in Conversational Question Answering (ConvQA) by proposing a novel framework called \"Dialogizer\" that automatically generates ConvQA datasets with high contextual relevance from textual sources.\n\n1. The paper introduces the concept of dialog inpainting as a method to generate ConvQA datasets from pre-existing text datasets. It segments the text dataset into sentence-level units, which are used as answers, and utilizes a trained dialog inpainter to generate questions corresponding to these answers, completing the conversation.\n\n2. To overcome the limitation of contextually-irrelevant questions, the paper proposes a novel framework called \"Dialogizer.\" The framework incorporates two training tasks: question-answer matching (QAM) and topic-aware dialog generation (TDG).\n\n3. During the inference process of generating dialog from the passage, re-ranking is conducted using contextual relevance as a metric, ensuring the generation of high-quality questions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tQ5CMhgqrS", "reviewer": "Reviewer_D2VE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your thoughtful evaluation of our paper. We are thankful for the positive feedback and for considering our work favorably. \n\nAs you pointed out, our research aims to establish a methodology for the automated construction of high-quality ConvQA datasets. To achieve this, we developed a framework called \"Dialogizer\" that integrates various methodologies and experimentally demonstrated the performance of the Dialogizer. We hold a positive outlook in envisioning that this study could be conducive to the ConvQA research field. Your constructive insights and suggestions for visual improvement would have greatly contributed to the refinement of our research. We will carefully consider your comments as we move forward with the final revisions. \n\nThank you once again for your valuable input.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-motivated and there is clear discussion and demonstrations of the limitation of previous methods.\n\n2. The proposed method is presented in a clear way. \n\n3. There are comprehensive experiments to show the effectiveness of the proposed method.", "reasons_to_reject": "I don't find any reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes Dialogizer, an inpainting model for the question generation given a passage. Previous methods suffer from low contextual relevance. To overcome this, the authors propose to incorporate two training tasks and re-ranking during the inference phase. Using this framework, four ConVQA datasets have been produced, which are of higher quality compared to the baseline dialog inpainting model. The evaluation has been done through automatic evaluation using diverse metrics and human evaluation.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I suggest using \\log in formulas for better visual effect.\n\nL254. + should be -\n"}, "extra_text": {}}, {"review_id": "mholiZRkU2", "reviewer": "Reviewer_ja3s", "summary": "", "questions": "1. Do authors try instruction tuning to achieve the goal of the propoese objectives in the paper? In my opinion, insturction tuning may achieve comparable or better performances. The authors should compare their methods with instruction tuning.\n\n2. I am confused about the section of Topic-aware Dialog Generation. How to extract topic words? How to construct the answer? Or how to contruct the question?\n\n3. Do you conduct hyper-parameters analysis?\n\n4. using GPT-4 to evaluate the generated texts are more reasonable.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank you for the comprehensive input on our study.\nWe will provide responses to each of the questions.\n\n### **Response for Reason to Reject 1-1 - Have low impact on current LLM-based research community**\n\nThere has been significant advancement and attention directed towards LLMs, which excel at learning and engaging in conversations across general domains. However, there remains a relative dearth in research and models focused on domain expert LLMs[1,2,3]. To create a Domain Expert LLM, the essential components are high-quality conversations and questions. By applying our methodology to papers or documents authored by experts, we can generate a substantial ConvQA dataset composed of high-quality questions and answers. Consequently, this approach could prove beneficial in the current LLM community for the creation of domain-specific expert LLMs. While the application of our methodology to LLMs might enable the generation of datasets with higher quality, it is important to note that such an endeavor could prove challenging at the academic level. Nevertheless, we believe that a notable advantage lies in the ability to swiftly create datasets for training LLMs on smaller scales.\n\n### **Response for Reason to Reject 1-2 - Comparison with ChatGPT**\n\nAs mentioned earlier, we believe that if we are able to create a high-quality ConvQA dataset using a model of approximately the same caliber as T5, there would be a clear advantage in terms of cost-effectiveness when compared to LLM. However, in alignment with your assertion, conducting additional experiments by supplying proper prompts or examples to ChatGPT, performing passage-to-dialog transformations, and subsequently comparing the quality with Dialogizer has been undertaken with the belief that this will offer supplementary insights.\n\nFor the experiments, we employed ChatGPT Plus (GPT-4, August 3 version). Due to cost considerations, we were unable to generate as many samples as Dialogizer, and thus, after generating 100 samples, we evaluated with RQUGE and proceeded with a manual analysis. Having experimented with various prompts, we have attached the best-performing prompt below in the rebuttal.\n\nThe sample dataset generated using GPT-4 achieved a RQUGE score of **2.6467**, which stands lower in comparison to the Dialogizer's performance. Upon conducting a direct analysis of the samples, it can be seen that  GPT-4 indeed excels in generating fine-grained and well-formed sentences; however, there is a noticeable diminishment in answer relevance that can be observed. Extracted from the dialog generated using a single passage, we present below examples of good QA pairs and bad QA pairs. The three versions of the complete dialog for this example, generated by Dialog Inpainter, Dialogizer, and ChatGPT, can be observed below in the rebuttal(above the prompt used for this experiment). Incorporating such content into the appendix could potentially offer readers supplementary insights. Also, in the future, exploring methods such as refining prompts or instruction tuning to leverage LLM for passage-to-dialog studies could potentially yield valuable insights. \n\n**Good Example (Generated by GPT-4)**\n```\nQ: Can you tell me about the history of Czech immigration in Argentina?\nA: Czechs in Argentina The Czech immigration in Argentina began during the World War 1 and it was split in four periods.\n```\n```\nQ: That's really interesting. How did the motivations or characteristics of the immigrants change in the later periods?\nA: The first one is slightly before to the World War 1, the second one from 1920 to 1930, the third one during the World War 2 and the fourth one, the smallest in migration proportion, during 1990. During the first two periods, the immigration group was mainly integrated by workers and farmers motivated by economic reasons.\n```\n**Bad Example (Generated by GPT-4)**\n```\nQ: That's fascinating. How have the Czech immigrants contributed to Argentinian society?\nA: Argentina has the largest Czech community in Latin America.\n```\n```\nQ: Do they maintain strong ties with their Czech heritage, or have they mostly assimilated into Argentinian culture?\nA: Czechs settled mainly in Buenos Aires, Gran La Plata, Rosario and Chaco.\n```\n### **Response for Reason to Reject 2 - Needs for evaluation on diverse metrics and quantitative evaluated analysis**\n\nIn the introduction section, we elucidated the motivation behind our research by highlighting the lower R**Q**UGE scores of datasets generated using the dialog inpainter. Subsequently, after introducing the Dialogizer, we indicated in lines 103-106 of the introduction that we conducted automatic evaluations using diverse dialog metrics. Furthermore, we underscored in the introduction our engagement in human and LLM evaluations, as well as an application to text retrieval tasks. While detailed experimental outcomes are not provided in the introduction, a concise summary of our final experimental results and their significance is encapsulated in lines 107-110 of the introduction.\n\nFor a more detailed explanation of our experiments: we have evaluated our methodology across diverse aspects by leveraging several metrics, including QRelScore(LRM & GRG), USR-DR(c), USR-DR(f), and the GPT-2 based dialog metric, in addition to R**Q**UGE. This can be seen in the Experimental Section (**Section 4.3**), **Table 1** and **Table 2**. Additionally, we verify our excellence through Human evaluation and LLM-based evaluation (ChatGPT evaluation) results (**Section 4.4**) and application to text retrieval task (**Section 4.5**). Through conducting diverse quantitative and qualitative assessments that compare the Dialogizer with the Dialogue Inpainter, we have provided evidence that the proposed dialogizer framework is able to generate high-quality ConvQA datasets.\n\n### **Response for Question 1 - Question about the instruction tuning**\n\nEngaging in passage-to-dialog using instruction tuning, as you suggested, could also serve as a valuable point of comparison. However, currently, the absence of a training set specifically tailored for LLM instruction tuning makes the implementation of such an experimental setting challenging[4,5]. Dialog Inpainter proposed a method for training passage-to-dialog without utilizing Supervised Fine-Tuning (SFT), and in this work, we further conducted research to enhance the quality by incorporating several methodologies. To achieve high-quality question generation through instruction tuning, the utilization of labeled datasets from relevant tasks is required. As of now, given the absence of such datasets, the methodology we employed seems to be the most suitable approach.\n\n### **Response for Question 2 - Question about the Topic-aware Dialog Generation**\n\nTDG constitutes a task wherein keywords are extracted from answers, employed to formulate prompts, and models are trained to generate questions accordingly. By utilizing the extracted keyword from s1, denoted as k(s1), a prompt is constructed as $(s_{p}, \"Keyword: k(s_1)\", \\bullet, s_1)$, following which $\\hat{u}_1$ is generated. For keyword extraction, we employed the T5-based keyword extraction model proposed by Pezik et al. (2022)[6]. A detailed explanation of TDG can be found in **sections 3.3** Topic-aware Dialog Generation and **3.4** Inference: Autoregressive Generation. The model utilized for keyword extraction is provided in **section 4.1** Model Implementation.\n\n### **Response for Question 3 - Question about hyper-parameters analysis**\n\nWe conducted multiple experiments for hyper-parameter tuning, varying hyper-parameters such as $\\lambda_{{QAM}}$ : (0.05, 0.1, 0.5) and $\\lambda{_{TDG}}$ : (0.05, 0.1, 0.5). Elaborated details on hyper-parameter tuning and the best-performing hyper-parameters can be found in Appendix C.2 Training configuration.\n\n### **Response for Question 4 - Question about using GPT-4 to evaluate the generated texts**\n\nIn the LLM evaluation, we utilized ChatGPT Plus (GPT-4). We appreciate the clarification, and in order to ensure clarity in the final version, we will specify the version in a footnote.\n\n### **Response for Typos Grammar Style and Presentation Improvements**\n\nR**Q**UGE[7] is a newly introduced reference-free metric that measures the answerability of questions with respect to the given context. Hence, in relation to the points, R**Q**UGE actually matches our intended purpose. A more explanation of R**Q**UGE can be found in Section 4.3 Automatic Evaluation - Evaluation Metrics.\n\n### **In conclusion,**\n\nThank you for your valuable feedback. We have provided responses to your concerns and questions. If your concerns are sufficiently addressed, we appreciate it if you would consider accepting our paper. Otherwise, please let us know about your remaining concerns.\n\n\n___\n**Examples of ConvQA data samples generated by Dialog Inpainter, Dialogizer, and ChatGPT**\n\nWhile it is unfeasible to present all examples, we present the outcomes of our experimentation with the foremost passage of WikiDialog as a means to provide a maximally objective comparison. \n\n**#1. Dialog Inpainter (baseline)**\n```\nQ: When did the Czechs settle in Argentina?\nA: Czechs in Argentina The Czech immigration in Argentina began during the World War 1 and it was split in four periods.\nQ: What led to the split?\nA: It is estimated that around 40,000 Czechs arrived to Argentina until 1970.\nQ: Are there any other interesting aspects about this article?\nA: Argentina has the largest Czech community in Latin America.\nQ: Where did the majority settle?\nA: Czechs settled mainly in Buenos Aires, Gran La Plata, Rosario and Chaco.\nQ: What period did they settle?\nA: There are four Czech immigration periods in Argentina.\nQ: What led to this?\nA: The first one is slightly before to the World War 1, the second one from 1920 to 1930, the third one during the World War 2 and the fourth one, the smallest in migration proportion, during 1990. During the first two periods, the immigration group was mainly integrated by workers and farmers motivated by economic reasons.\n```\n\n**#2. Dialogizer (ours)**\n```\nQ: What is the history of Czechs in Argentina?\nA: Czechs in Argentina The Czech immigration in Argentina began during the World War 1 and it was split in four periods.\nQ: How many Czechs arrived in Argentina?\nA: It is estimated that around 40,000 Czechs arrived to Argentina until 1970.\nQ: Is there a large Czech community in Argentina?\nA: Argentina has the largest Czech community in Latin America.\nQ: Where did the Czechs settle in Argentina?\nA: Czechs settled mainly in Buenos Aires, Gran La Plata, Rosario and Chaco.\nQ: How many immigration periods did the Czechs in Argentina have?\nA: There are four Czech immigration periods in Argentina.\nQ: What are the four Czech immigration periods?\nA: The first one is slightly before to the World War 1, the second one from 1920 to 1930, the third one during the World War 2 and the fourth one, the smallest in migration proportion, during 1990. During the first two periods, the immigration group was mainly integrated by workers and farmers motivated by economic reasons.\n```\n**#3. ChatGPT (GPT-4)**\n```\nQ: Can you tell me about the history of Czech immigration in Argentina?\nA: Czechs in Argentina The Czech immigration in Argentina began during the World War 1 and it was split in four periods.\nQ: What are the four periods you mentioned?\nA: It is estimated that around 40,000 Czechs arrived to Argentina until 1970.\nQ: That's fascinating. How have the Czech immigrants contributed to Argentinian society?\nA: Argentina has the largest Czech community in Latin America.\nQ: Do they maintain strong ties with their Czech heritage, or have they mostly assimilated into Argentinian culture?\nA: Czechs settled mainly in Buenos Aires, Gran La Plata, Rosario and Chaco.\nQ: Could you elaborate on what each of these four periods entails?\nA: There are four Czech immigration periods in Argentina.\nQ: That's really interesting. How did the motivations or characteristics of the immigrants change in the later periods?\nA: The first one is slightly before to the World War 1, the second one from 1920 to 1930, the third one during the World War 2 and the fourth one, the smallest in migration proportion, during 1990. During the first two periods, the immigration group was mainly integrated by workers and farmers motivated by economic reasons.\n```\n**ChatGPT prompt and dialog generation process**\n\n**STEP 1**\n```\nPlease fill in only [BLANK] in the next dialogue.\n\n[START]\nA: [BLANK]\nB: Czechs in Argentina The Czech immigration in Argentina began during the World War 1 and it was split in four periods.\n[END]\n```\n**STEP 2**\n```\nPlease fill in only [BLANK] in the next dialogue.\n\n[START]\nA: Can you tell me about the history of Czech immigration in Argentina?\nB: Czechs in Argentina The Czech immigration in Argentina began during the World War 1 and it was split in four periods.\nA: [BLANK]\nB: It is estimated that around 40,000 Czechs arrived to Argentina until 1970.\n[END]\n```\n... \n\n**[References]**\n\n[1] Peskoff, Denis, and Brandon M. Stewart. \"Credible without Credit: Domain Experts Assess Generative Language Models.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2023.\n\n[2] Ge, Yingqiang, et al. \"Openagi: When llm meets domain experts.\" arXiv preprint arXiv:2304.04370 (2023).\n\n[3] Shen, Yongliang, et al. \"Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.\" arXiv preprint arXiv:2303.17580 (2023).\n\n[4] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in Neural Information Processing Systems 35 (2022): 27730-27744.\n\n[5] Zhou, Wangchunshu, et al. \"Controlled text generation with natural language instructions.\" arXiv preprint arXiv:2304.14293 (2023).\n\n[6] Pęzik, Piotr, et al. \"Keyword extraction from short texts with a text-to-text transfer transformer.\" Asian Conference on Intelligent Information and Database Systems. Singapore: Springer Nature Singapore, 2022.\n\n[7] RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question (https://aclanthology.org/2023.findings-acl.428) (Mohammadshahi et al., Findings 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental results and analysis are extensive.\n\nThe proposed framework for generating CQA dataset seems reasonable.", "reasons_to_reject": "The proposed methods have low impact on current LLM-based research community.  May chatgpt can construct such datasets by providing proper prompts and examplars?\n\nI think ROUGH score can not full represent the quality of generated texts in introduction. The authors should present: 1. more evaluated metrics to validate the generated texts from Dialogizer and Dialogue Inpainter in different aspects. 2. quantitive evaluated analysis of texts from Dialogizer and Dialogue Inpainter rather than some specific examples. Selecting only a few examples lacks persuasiveness.", "questions_to_authors": "1. Do authors try instruction tuning to achieve the goal of the propoese objectives in the paper? In my opinion, insturction tuning may achieve comparable or better performances. The authors should compare their methods with instruction tuning.\n\n2. I am confused about the section of Topic-aware Dialog Generation. How to extract topic words? How to construct the answer? Or how to contruct the question?\n\n3. Do you conduct hyper-parameters analysis?\n\n4. using GPT-4 to evaluate the generated texts are more reasonable.", "paper_topic_and_main_contributions": "This study introduces Dialogizer as a novel framework for generating contextually relevant ConvQA datasets from textual datasets. The framework incorporates two training methodologies, in addition to dialog reconstruction, to address the limitation of generating contextually-irrelevant questions. These methodologies include a question-answer matching (QAM) task and a topic-aware dialog generation (TDG) task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "ROUGH score rather than RQUGH score in introduction?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: A dialog inpainting pipeline to convert dialogs into question-answering. Sound fidelity evaluation.
Decision: Accept-Main