Book2Dial: Generating Teacher-Student Interactions from Textbooks for Cost-Effective Development of Educational Chatbots (2024)

Abstract: Educational chatbots are a promising tool for assisting student learning. However, developing effective chatbots in education has been challenging, as high-quality data is seldom available in this domain. In this paper, we propose a framework for generating synthetic teacher-student interactions grounded in a set of textbooks. Our approaches capture one aspect of learning interactions where curious students with partial knowledge interactively ask a teacher questions about the material in the textbook. We highlight various quality criteria such dialogues should fulfill and compare several approaches relying on either prompting or fine-tuning large language models. We use synthetic dialogues to train educational chatbots and show the benefits of further fine-tuning in different educational domains. However, human evaluation shows that our best data synthesis method still suffers from hallucinations and tends to reiterate information from previous conversations. Our findings offer insights for future efforts in synthesizing conversational data that strikes a balance between size and quality. https://github.com/eth-lre/

text: Findings of the Association for Computational Linguistics ACL 2024, pages 9707–9731 August 11-16, 2024 ©2024 Association for Computational Linguistics
https://github.com/eth-lre/ book2dial

heading: 1 Introduction
text: Educational chatbots offer a scalable way to improve learning outcomes among students (Kuhail et al., 2023). However, building educational chatbots has been challenging as high-quality data involving teachers and students is difficult to obtain due to various practical reasons such as privacy concerns (Macina et al., 2023a). In response to this, we study the task of generating synthetic teacherstudent interactions from textbooks. We create a novel dataset of textbooks drawn from an open publisher of student textbooks and present a framework ( Book2Dial) to generate synthetic teacherstudent interactions from these textbooks.
Our teacher-student interactions take the form of conversational question-answering (QA) interactions (Choi et al., 2018; Reddy et al., 2019) where curious students ask teachers questions about the textbook and teachers answer these questions based on the textbook. Our approach primarily extends the direct instruction phase of teaching factual knowledge by facilitating more interactive information exchanges with student (Chi and Wylie, 2014). This differs from techniques focusing on teaching using scaffolding (Macina et al., 2023a) or Socratic questioning (Shridhar et al., 2022) which leads to deeper and wider human learning. However, the task of generating high-quality synthetic dialogue data is difficult (Dai et al., 2022). This is amplified in education, where the generated interactions should cover the teaching material in an informative and coherent way (Agrawal et al., 2012; Crosby, 2000). Therefore, it is important to have quality controls on such data, because students might otherwise receive wrong feedback, which could be detrimental to learning.
Thus, in this work, we also formulate some criteria that measure the quality of educational dialogues. For example, it is crucial that the chatbot does not provide students with incorrect information and stays grounded in the textbook, ensuring factual consistency with the knowledge taught. This is particularly important given that large language models (LLMs) are prone to ‘hallucinations’ or generating plausible but incorrect or unverified information (Ji et al., 2023). While a simple teacher strategy would be to answer with extracted passages from the textbook, this might hurt the coherence of the dialogue which is present in interactive educational situations (Baker et al., 2021). The teacher’s response should both be relevant to the student’s question (Ginzburg, 2010), as well as, informative as this ensures that key information from the textbook is covered in the dialogue (Tan et al., 2023). We formalize these requirements into
9707
criteria shown in Figure 1. Book2Dial comprises of three approaches taken from prior work: multi-turn QG-QA (Kim et al., 2022), Dialogue Inpainting (Dai et al., 2022) and prompting LLMs to role-play teacher and student. We use the formatting information in the textbook, such as titles or key concepts, to initialize student models with imperfect information. In contrast, the teacher models have perfect information and are expected to generate grounded responses based on the textbook. We fine-tune and prompt various open-source language models to generate teacher-student interactions.
We evaluate Book2Dial on the proposed quality criteria and use human evaluations to support our findings. Results reveal that data generated by role-playing LLMs scores highest in most criteria, as shown in Section 5.1.1 and 5.1.2, demonstrating reasonable efficacy in creating educational dialogues. The generated dialogues effectively incorporate textbook content but fall short in mimicking the natural scaffolding of educational conversations and exhibit issues like hallucinations and repetition, as discussed in Section 5.3. Despite these limitations, we show that the generated synthetic data may be used to fine-tune educational chatbots with benefits in some educational domains (Section 5.4), especially when the domain of the evaluation dataset matches with the fine-tuning data.

heading: 2 Related Work

heading: 2.1 Synthetic Data for Conversational QA
text: Prior work in educational research has focused on generating individual questions (Kurdi et al., 2020)
under two common settings: answer-aware and answer-unaware generation. The former approach starts by identifying an answer and then generates a question accordingly, whereas the latter generates a question without pre-determining the answer. These approaches have also been extended to generating multiple questions (Rathod et al., 2022), causal question generation (Stasaski et al., 2021), prediction of question types to ask (Do et al., 2023), or decomposing problems into Socratic subquestions (Shridhar et al., 2022). However, most works do not address conversational settings.
Datasets like QuAC (Choi et al., 2018), CoQA (Reddy et al., 2019) and UltraChat (Ding et al., 2023) focus on conversational question answering in non-educational settings. Previous work has also explored strategies for creating such data with humans or automatically by using models. For example, Qi et al. (2020) withholds the context required for answers from the questioner, leading to information-seeking questions. SimSeek (Kim et al., 2022) synthesizes datasets for conversational question answering from unlabeled documents but does not demonstrate significantly improved performance in downstream tasks. A recent work, Dialogizer (Hwang et al., 2023), proposes a framework for generating context-aware conversational QA dialogues. However, these methods do not take into account the specifics of the educational domain.
Other works outside of conversational question answering also explore data synthesis. For example, Kim et al. (2023) source dialogue data from a commonsense knowledge graph, Chen et al. (2023) prompt LLMs, and Bao et al. (2023) create
grounded dialogue datasets.

heading: 2.2 Educational Dialogue Datasets
text: The development of educational chatbots is highly reliant on high-quality data. Yet such data is hard to obtain. Therefore, previous works such as MathDial (Macina et al., 2023a) collect conversational data by pairing real teachers with an LLM that simulates students. Other datasets are commonly created by roleplaying both teacher and student, such as CIMA (Stasaski et al., 2020) or by transcribing classroom interactions (Suresh et al., 2022; Demszky and Hill, 2023) or logging online conversations (Caines et al., 2020). However, all of these methods are challenging to scale, and using nonexperts often leads to data quality issues (Macina et al., 2023a).
Thus, in this work, we explore data synthesis as a scalable way of creating such data. Data augmentation and synthetic data generation gain attention as effective techniques to overcome the challenges associated with manual data annotation. Synthetic data generation is shown to be a promising approach. For instance, Kim et al. (2023) demonstrates the potential of sourcing dialogue data from common sense knowledge. However, ensuring the objectivity of generated data remains a concern. Similarly, Zhang et al. (2018) introduces methods for task-oriented dialogue synthesis. However, its dependency on predefined schemas limits its scalability.

heading: 3 Educational Conversation Generation
text: We first introduce a framework for dialogue synthesis from textbooks in Section 3.1, and then discuss the quality criteria that the generated dialogues should fulfill in Section 3.2.
3.1 Book2Dial Framework
We set out to create meaningful teacher-student interactions from educational textbooks in the form of conversational QA pairs between the teacher and the student. To generate these interactions, we assume that the “teacher” is knowledgeable about the textbook content, and the “student” only knows limited information from the textbook. Thus, we provide the teacher model all the textbook information but withhold some information from a student model. For this, we can use the structuring and formatting elements found in textbooks, including 1) Titles: headings of sections and subsections; 2) Summary: summaries of chapters; 3) Other Metadata: key concepts, learning objectives, bold terms, and the introductory paragraph of each section; and assume that the student model only has access to this information.
During the conversation, the “student” asks inquisitive questions about the textbook while the “teacher” guides them by answering these questions and including additional information in their response. Formally, a dialogue d comprises of a sequence of T question-answer interactions: d = {(q1, a1), . . . , (qT , aT )}. The formalization of the
task is depicted in Figure 2. The student model pstu(qt|C, h<t) generates a question qt given the dialogue history h<t = {(qi, ai)}t−1i=1 and the partial context (formatting information) C. The teacher model ptea(at|S, h<t, qt) generates the answer response at given the question, the dialogue history, and the full textbook source S.

heading: 3.2 Evaluation of Educational Conversations
text: To build a high-quality educational conversation, we focus on the student asking questions that are specific enough to drive the conversation forward, and also answerable given the context. The teacher must then respond with an answer that is relevant to the question, factually consistent with the context, and informative to the student. Finally, the overall conversation should be coherent and grounded to the entire context, not just parts of it. We use this as our guiding principle and define seven criteria to evaluate the quality of a good educational interaction. Although these metrics may not necessarily be mutually exclusive or extensively address all possible features of high-quality educational dialog, each of them serves as an important aspect to be considered in the education domain. We selected those that could be instantiated by measures and are potentially useful to supplement the traditional factual classroom listening to a lecture with interactive chatbot interactions (Chi and Wylie, 2014; Ruan et al., 2019). We detail these criteria in the rest of this subsection.

heading: 3.2.1 Answer Relevance
text: Answer Relevance measures how directly related the answer is to the question in each QA pair in the dialogue. This criterion is crucial in education, as teachers should have subject-matter knowledge and can adapt responses to the student’s various knowledge levels (Lepper and Woolverton, 2002).
We assess the Answer Relevance of individual QA pairs and then combine these assessments to determine the dialogue’s overall Answer Relevance. We use BF1(qt, at), QuestEval and Uptake as metrics for Answer Relevance. The BF1 is an abbreviation for the BERTScore F1 (Zhang* et al., 2020) for semantic alignment between questions and answers using BERT embeddings. QuestEval (Scialom et al., 2021) generates questions from both the question and answer, then generates answers for these questions, comparing them to measure relevance. The Uptake metric (Demszky et al., 2021), specific to the education domain, analyzes
teachers’ responses to student utterances, focusing on their dependence and relevance. More details are in Table 5.

heading: 3.2.2 Coherence of the Dialogue
text: Coherence measures whether QA pairs in the dialogue form a logical and smooth whole, rather than independent QA pairs. Coherence is an important aspect of good dialogue (Dziri et al., 2019) and it is important in education to sequentially instruct students with the new textbook knowledge in a coherent way (Agrawal et al., 2012).
We adapt two metrics, BF1(qt, a<t) and BF1(qt, a(t−1)), to measure coherence, similar to the approach in (Kim et al., 2022). The first metric uses BERTScore F1 to evaluate the current question against each previous answer as references, whereas the second metric compares the current question solely with the immediately preceding answer using BERTScore F1. See Table 5 for more details.

heading: 3.2.3 Informativeness
text: Informativeness evaluates the amount of new information introduced by each student-teacher interaction in the dialogue. This criterion is important in education because the role of providing information is a key aspect of teachers’ responsibilities (Crosby, 2000). Informativeness is important to analyze the extent at which the textbook content is progressively introduced to the student.
To assess Informativeness, we use 1 - Overlap(at, a<t), calculating one minus the ratio of token intersection over union in the current and all previous answers for each QA pair. This metric is correlated with human evaluation, as detailed in Appendix E and Table 5.

heading: 3.2.4 Groundedness to the Textbook
text: This criterion assesses the amount of information from the textbook incorporated into the dialogue. Two metrics are used for assessment: Density, evaluating the average length of text spans extracted from textbook content S and included in the dialogues; Coverage, measuring the proportion of dialogue words originating from the textbook. Both metrics are adopted from (Grusky et al., 2018), and their formulas are shown in Table 5.

heading: 3.2.5 Answerability of the Questions
text: Answerability measures whether the student’s question is answerable given the textbook content.
While non-answerable questions could also represent the curiosity of a student (Scialom and Staiano, 2020), the answerability of questions given the context is generally important for a more useful dialog. We use the QA model1 to judge whether each question is answerable given the textbook content. We refer to this metric as Answerability. This approach is akin to the method employed in (Kim et al., 2022). More details are in Table 5.

heading: 3.2.6 Factual Consistency of the Answer
text: Factual Consistency measures whether the answer correctly responds to the student’s question. This criterion is crucial in education because it is important for students to learn accurate information (Metzger et al., 2003). Existing metrics like Q2 (Honovich et al., 2021) use a QA model to assess answer correctness, while RQUGE (Mohammadshahi et al., 2023) uses a QA model to evaluate the quality of the candidate question. In our scenario, we need to measure whether the answer contains correct information and accurately answers the question. Therefore, we build on the idea of Q2 and introduce a new metric referred as QFactScore:
α · sim(QA(qt, S), at) + β · sim(qt, at) (1)
It calculates the cosine similarity of embeddings between the predicted and original answers for each QA pair and also evaluates the similarity between the question and the original answer. This metric has been validated for its alignment with human evaluation, as detailed in Appendix E. The final score is the weighted sum of two similarity scores. More details are in Table 5 and Appendix C.4.

heading: 3.2.7 Specificity of the Question
text: Specificity assesses whether the question is specific, rather than general. An example of a generic question is ‘What is interesting about this passage?’. The Specificity criterion is crucial in education, as teachers should ask specific questions (Yang, 2017). We assess specificity through human evaluation, as there is no existing metric that captures specificity.

heading: 4 From Textbooks to Dialogues
text: In this section, we describe different methods used for generating dialogues from educational textbooks in Book2Dial, namely:
1https://huggingface.co/distilbert-base-cased-distilledsquad
1. Multi-turn QG-QA models. In this setting, we use fine-tuned QG and QA models interacting with each other.
2. Dialogue Inpainting. Dai et al. (2021) uses a span extraction model over the textbook as a teacher model, where the response is copied from the textbook and the question is generated by a QG model acting as the student.
3. Persona-based Generation. This approach uses LLMs like GPT-3.5, and leverages prompting to interactively simulate the student and the teacher and generate dialogues.
We describe these methods next. More implementation details are in Appendix C.

heading: 4.1 Multi-turn QG-QA models
text: This scenario utilizes separate QG and QA models to interact in a multi-turn scenario. As a representation of this approach from related work, we consider the SimSeek-asym model (Kim et al., 2022). The approach consists of two components:
1. A Question Generation (QG) model for generating conversational questions relying solely on prior information (i.e., formatting information relevant to the topic). This student model generates a question based on the dialogue history and filtered Information C: p(qt|C, h<t).
2. A Conversational Answer Finder (CAF) to comprehend the generated question and provide an acceptable teacher answer to the question from the evidence passage: p(at|S, h<t, qt).

heading: 4.2 Dialogue Inpainting
text: Dialogue Inpainting (Dai et al., 2022) is an approach for dialogue generation characterized by its information-symmetric setting. In this framework, both the student and teacher model are provided with the complete textbook text S. The teacher model iterates over each sentence in S and copies it as an answer. The student model is a QG model. We use data from the OR-QuAC (Qu et al., 2020), QReCC (Anantha et al., 2021), and Taskmaster-2 (Byrne et al., 2020) datasets to train the student model. For the student model, a dialogue reconstruction task is employed. At training time rather than distinguishing questions and answers, the dialogue reconstruction task treats a conversation as a sequence of utterances {ui}2Ti=1, To train it, a randomly chosen utterance ui is masked to create a
partial dialogue dm(i) = u1, · · · , ui−1, < mask > , ui+1, · · · , u2T . The model then predicts ui and is trained by minimizing the loss:
L(θ) = − ∑
d∈D Eui∼d[log pθ(ui | dm(i))] (2)
During inference, the model uses each sentence in the textbook as a teacher’s utterance and only predicts student utterances accordingly, {u2k−1}Tk=1 corresponding to {qi}Ti=1 in our notation. We basing our model (eq 2) on FLAN-T5-XL (Chung et al., 2024). More details are in Appendix C.2.

heading: 4.3 Persona-based Generation
text: Inspired by (Markel et al., 2023)’s idea of using LLMs to simulate student personas, we propose a method to simulate student and teacher personas by prompting LLMs for dialogue generation. We use one instance of the GPT-3.5 model to play the student and another to play the teacher.2 The teacher model is provided with all the information from the textbook, including content and all the formatting information. The information provided to the student model is varied. We consider four variants for generating dialogue in each subsection listed with an increasing amount of information provided to the student model: 1) Persona (Low Info) provides the student model with only the Title information, 2) Persona (Medium Info) provides both the Title and Summary information, 3) Persona (High Info) offers all formatting information, and 4) Persona (Single Instance) generates the entire dialogue using a single LLM instance, equipping one model with formatting and textbook content information. More details are in Table 7.
Considering that GPT-3.5 is not open-source, we use prompting to steer the models in dialogue generation. The prompt for Persona (High Info) and Persona (Single Instance) is detailed in Appendix C.3.

heading: 5 Results and Analyses
text: In this section, we aim to address the following research questions:
1. How does the choice of generation framework influence the quality of the generated data?
2We used the GPT-3.5-turbo API between 25th September and 4th October, 2023.
2. What is the optimal amount of information that should be incorporated into the student model to produce natural dialogues?
3. How does fine-tuning on the synthesized data in various domains improve the performance of models that are fine-tuned and evaluated on existing datasets?
To address these questions, we generate dialogues from textbooks across various domains and analyze the generated dataset. Textbook data: We collect 35 textbooks available on OpenStax3, spanning domains of math, business, science, and social science. From these, we select four textbooks to create our dialogue datasets. Table 6 provides statistics of the four textbooks. The first and second research questions are addressed in Sections 5.1 and 5.2, respectively, while the third question is answered in Section 5.4.

heading: 5.1 Automatic Evaluation
text: In this section, we discuss statistics and metrics for the generated datasets. We present aggregated statistics of the datasets across four textbook domains in Tables 1 and 2, also noting comparisons with existing datasets: MathDial (Macina et al., 2023a) – math tutoring conversations between human teachers and student LLMs, fact-based humanhuman conversations of QuAC (Choi et al., 2018), and NCTE transcripts of math classrooms (Demszky and Hill, 2023). Domain-specific results in the dataset are detailed in Tables 10 and 11. To adjust for varying dialogue lengths, we limit the number of turns to T = 12 for each model, as in (Kim et al., 2022).

heading: 5.1.1 Statistical Analysis
text: In dialogue, different types of questions emphasize various aspects. We hypothesize that “what” and “which” questions focus on factual information. In contrast, other question types, such as “why” and “how,” tend to reflect more complex inquiries, which are also important in educational contexts. In Table 1, we present the percentages of student questions including words what, which, why, and how4. Furthermore, the average token count for questions and answers across each dataset is also shown. The key findings are as follows:
3https://openstax.org/ 4This ratio excludes ’how much’ and ’how many’ questions
because they pertain to factual information.
Less factual questions in the Persona (Single Instance) dataset The Persona (Single Instance) model generates the fewest “what” or “which” questions compared to other synthesized datasets, suggesting more diverse questioning. It also has a similar question type distribution to MathDial.
More “how” questions in Persona datasets The four Persona datasets contain the highest ratio of ‘how’ questions, which suggests a higher ratio of questions asking for explanations.
High token counts in Persona datasets Datasets from Persona models feature the high average token counts in questions and answers, suggesting these dialogues are more verbose and informative. NCTE features high token counts in questions, typical in classroom transcripts with lengthy teacher inquiries and brief student responses.

heading: 5.1.2 Data Quality Metrics
text: We report the various data quality metrics in Table 2. Our key findings are as follows:
Persona datasets are solid in most of the criteria Persona models generated datasets outperform others in most metrics, indicating their good ability to create dialogues from textbooks. Note the MathDial, QuAC, and NCTE datasets each focus on different domains with different data collection designs and grounding texts, therefore they are not directly comparable in these metrics and should be interpreted with their specific context in mind. For example, a tutoring dataset like MathDial is expected to have lower answerability as tutors tend to ask challenging questions while the specific data collection of QuAC to include supporting arguments increases the density.
High Informativeness and Groundedness of Dialogue Inpainting dataset: Dialogue Inpainting models achieve the highest score across all models in Informativeness and Groundedness. This is expected as this model uses exact sentences from the textbooks as teachers’ answers.
Students with more information access perform better in automatic metrics. Datasets from Persona (High Info) and Persona (Medium Info) typically outperform or match with datasets from Persona (Low Info). This suggests that more information to the student may enhance the key criteria. However, the impact differences among formatting levels are not markedly significant, indicating a need for further research on this question.

heading: 5.2 Human Evaluation
text: To compensate for the limitations of automatic metrics, we conduct human evaluations of SimSeek, Dialog Inpainting, and Persona (High Info) dialogues based on seven criteria: Answer Relevance (AnsRel), Informativeness (Info), Groundedness (Gro), Coherence (Coh), Factual Consistency (Fact), Answerability (Ans), and Specificity (Spe). Questions for judging each criterion are in Table 12. We recruited 4 annotators to evaluate 12 dialogues each, yielding a Fleiss’ kappa of 0.74, indicating reasonable agreement. Evaluation details are in Appendix F, and results in Table 3.
Persona (High Info) excels among the three models, leading in Answer Relevance, Coherence, Factual Consistency, Answerability, and Specificity, rendering it the most suitable choice based on our dialogue generation objectives. This result aligns with the results of automatic metrics presented in Table 2. However, the dialogues generated by the Persona-based method exhibit only an average score in Informativeness, with a score of 0.74 indicating that approximately 26% of QA pairs do not contribute new information to the conversation. The Persona-based model, while leading in Factual Consistency among the three models, scores only 0.79, which indicates that approximately 21% of the QA pairs lack Factual Consistency. For educational dialogues, it’s imperative to aim for high Factual Consistency to ensure the reliability of the knowledge imparted. The primary reason for this issue is the hallucination in LLMs, where LLMs respond to questions using fabricated or false information not grounded in the textbook. This poses a significant challenge and calls for further research
into ways to better ground LLMs to text documents in the future.

heading: 5.3 Qualitative Human Analysis
text: We inspect dialogues generated by each model and find several limitations compared to natural educational conversations. We report qualitative descriptions of these findings:
Insufficient follow-up ability of Persona models Even though Persona models outperformed other methods in automatic and human evaluations, the model has several issues. Dialogues generated by Persona models are unlike natural conversations and resemble a series of artificial QA pairs about textbooks. Moreover, the dialogue does not have enough follow-up questions by students and rather broadly touches on the textbook content rather than going into depth about certain topics. We present an example of a dialogue demonstrating the insufficient follow-up ability of the Persona-based model in table 17.

heading: Repeating answers in SimSeek and Persona In
text: the SimSeek and Persona datasets, we find that teacher answers often reiterate information from previous interactions. SimSeek often generates questions related to the previous teacher’s answer,
while Persona often provides summaries of textbook content in each answer.

heading: Insufficient Specificity of Dialogue Inpainting
text: In alignment with the results of human evaluation, we find that the Dialogue Inpainting model tends to generate “general” questions, such as “What is interesting about this passage?” These types of questions, which are not specific to the textbook content, are less desirable in educational dialogue.

heading: 5.4 Fine-tuning for Educational Chatbots
text: In this section, we show the effectiveness of these imperfect but automatically created synthetic data based on any textbook domain for the downstream task of fine-tuning educational chatbots. We finetune simple chatbot models with our synthesized data and assess their performance on educational conversation task of next utterance generation.
Specifically, we use text generation models based on language models to generate teacher responses at given the dialogue history h<t, textbook grounding information S, and the question qt. We compare two scenarios: (1) a model fine-tuned on synthetic textbook data, then further fine-tuned and tested on various educational or informationseeking dialogue datasets; and (2) a model finetuned and tested solely on these dialogue datasets without fine-tuning on synthetic textbook data. We use FLAN-T5-LARGE (Chung et al., 2024) as our base language model. For our test sets, we use the MCTest and CNN splits of the CoQA dataset (Reddy et al., 2019), as well as the NCTE dataset (Demszky and Hill, 2023). The MCTest split contains dialogues about children’s stories; the CNN split contains conversations about the news; the NCTE dataset contains transcripts of elementary math classrooms.
We fine-tune the base model on four textbookbased synthetic datasets, each from a different subject: math, business, science, and social science. The datasets and training details are shown in Appendix G. The results are shown in Table 4. We report the BLEU score for the scenario where we fine-tune the base model on our textbook-generated dialogue dataset, with the difference between this fine-tuned version and the version without finetuning shown in brackets. We use two baselines: Zero-shot (0-Shot), which directly uses the FLANT5-LARGE model without any fine-tuning, and UltraChat, which is fine-tuned on the synthetic but non-educational UltraChat dataset (Ding et al., 2023) and tested on different target datasets.
We find that the model that is first fine-tuned on the social science textbook data achieves the highest score when tested on the MCTest and CNN splits of the CoQA dataset, with improvements of 4.16 and 1.99. Meanwhile, the model fine-tuned on the business textbook data achieves the highest score when tested on the NCTE dataset. The model fine-tuned on the math textbook data also shows improvements. As the social science textbook dataset contains the fewest math expressions, it improves the most in non-math domains but performs the worst in the math domain. We conclude that synthetic datasets created from textbooks, as well as Ultrachat synthetic data, may be effective for fine-tuning chatbots if they align with the target domain.
Upon a more qualitative human examination of the generated results, we find that the fine-tuned models on synthetic textbook data have a better understanding of the input context and generate more correct answers than the corresponding nonfine-tuned models. Some example generations are shown in Appendix G.1.

heading: 6 Conclusion
text: This work introduces a new task of generating conversational question-answering dialogues from textbooks to help fine-tune educational chatbots in various domains where high-quality dialogue data are scarce. We detail and compare various approaches and settings to simulate student-teacher interactions and create such data. We evaluate the generated dialogues, focusing on some measures of their quality, such as Answer Relevance, Informativeness, Coherence, and Factual Consistency. Our results indicate that the approach with LLMs role-
Book2Dial synthetic data. Two additional baselines are: Zero-shot (0-Shot), which directly uses the FLAN-T5-LARGE model without any training, and UltraChat, which is fine-tuned on UltraChat and tested on different target datasets.
playing as teachers and students for data synthesis performs well in these metrics. However, upon closer human inspection, we also observe several issues with the synthesized data, such as the problem of hallucinations and repeating information. Despite these issues, we show that the generated synthetic dialogues using Book2Dial can be used to fine-tune educational chatbots and achieve performance improvements in various educational settings.

heading: 7 Limitations
text: Focus on a specific teaching scenario and limitations in educational contexts In this work, we focus on a specific educational scenario where a curious student asks questions to a knowledgeable teacher. It is shown that the quality of the student’s questions (with deep reasoning ones) is correlated with their learning (Graesser and Person, 1994; Person et al., 1994). We do not model any of these aspects in our approach. Furthermore, recent approaches of teachers asking Socratic questions or providing indirect scaffolds and hints instead of providing students directly with answers are also shown to lead to better learning outcomes (Freeman et al., 2014). In our formulation, teachers directly provide students with answers. Our approach focuses on facilitating informational exchanges and is more suitable for helping students access the entire content of the textbook in a new interactive way and through their interests. This serves as a starting point for developing more sophisticated educational chatbots in the future. Future work could
focus on other interaction scenarios and combine our approach with Socratic questioning (Shridhar et al., 2022) and scaffolding (Macina et al., 2023b; Sonkar et al., 2023) to achieve significantly improved applicability to educational use cases.
Achieving the highest scores in all metrics is not the overall goal leading to the most effective human learning. Considering Informativeness, while a dialogue rich in information suggests a potential for a greater extent of learning by a student, there exists a trade-off, as excessive information can increase the student’s cognitive load and become overwhelming (Mayer and Moreno, 2003). Therefore, finding the optimal amount of information that the dialogue should contain needs careful consideration in future work. Similarly, for other metrics, educational practitioners could ideally set the target metrics and their combination for achieving better fine-tuning performance steered towards educational use cases.
Aspects of evaluation framework: Although we try to include various aspects of the evaluation in this work, it is not feasible to focus on all important educational aspects. We specifically focus on one setting, where students ask curious questions and the teacher provides answers. Therefore, comparing our datasets with the MathDial, QuAC, and NCTE datasets does not fully explain our datasets’ quality, as MathDial, QuAC, and NCTE datasets are focused on different interaction situations. In particular, none of the MathDial, QuAC, or NCTE datasets include textbook content; MathDial contains only short math problems and focuses on scaffolding, while QuAC is oriented towards factbased queries rather than student-teacher interactions; NCTE consists of classroom transcripts in which there are more than just two interlocutors.

heading: 8 Acknowledgements
text: This project was made possible by ETH AI Center Doctoral Fellowships to Jakub Macina with partial support from the Asuera Stiftung and the ETH Zurich Foundation. Nico Daheim acknowledges the funding by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE. We thank Xiaoyu Zhang for the insightful discussion about the design of the human evaluation experiments and Kumar Shridhar for help in collect-
ing the Openstax textbooks. Additionally, the authors wish to thank reviewers, members of the LRE group at ETH Zurich, and the participants in human evaluation experiments.

heading: A Metrics Formulas
text: The metrics mentioned in Section 3.2 are detailed and explained in Table 5, including formulas and explanations.

heading: B Textbook Statistics
text: The four textbooks we use to generate dialogue for experiments are collected from the OpenStax website. The math textbook is titled ’Introductory Statistics,’ the business textbook ’Business Ethics,’ the science textbook ’Physics,’ and the social science textbook ’Psychology 2e.’ The statistics of the four textbooks are shown in Table 6.
C Implementation Details
C.1 Information-seeking scenario In the SimSeek-ASYM setup, the CQG model ingests the title and summary information, each separated by special tokens. We use T5-Large as the student’s model and Longformer-Large as the teacher’s model.
The SimSeek-ASYM code5 can be executed with minor modifications. We use the same CQG and CAF models as in (Kim et al., 2022), which utilize T5 as the student’s model and Longformer as the teacher’s model.

heading: C.2 Dialogue Inpainting
text: We adopt a training regimen that integrates data from the OR-QuAC (Qu et al., 2020), QReCC (Anantha et al., 2021), and the movie and restaurant datasets from Taskmaster-2 (Byrne et al., 2020), employing the technique as described in (Dai et al., 2022). We randomly select 80% of the data as the training set, while the remaining 20% as the test set. We implement Dialogue Inpainting using the code framework of (Daheim et al., 2023), basing our model (eq 2) on FLAN-T5-XL (Chung et al., 2024), and train it with LoRA (Hu et al., 2022) to reduce computational load. We set an initial learning rate of 6.25e-5 and employ linear learning rate decay without warmup. For model optimization, we utilize checkpoints from the transformers library (Wolf et al., 2020). The negative log-likelihood of the ground-truth response is minimized using the AdamW optimizer, as detailed in (Loshchilov and Hutter, 2019). We assess model performance using the sacrebleu implementation of the BLEU metric, following (Post, 2018). We use one V100 GPU to
5https://github.com/naver-ai/simseek
train the model. The FLAN-T5-XL model has 3 billion parameters and takes 12 hours to train.
The model, while fundamentally designed to predict single utterances, is used autoregressively. It begins with the input sprompt, < mask >, s1 and sequentially generates questions using top-p sampling. This autoregressive process continues until the dialogue is wholly formed.

heading: C.3 Persona-based Generation
text: Prompt for Persona (High Info) The design of our prompts is chiefly driven by the requisites of context-awareness, speaker identification, and specificity. We incorporate guidelines and annotations to ensure GPT yields concise responses and minimizes redundant information. To distinguish between speakers, we prefix dialogues with labels: “Teacher:” or “Student:”. The prompt is shown below.
Prompt for simulating student Task: You are a student preparing to
ask questions about a textbook subsection to a teacher. Your goal is to uncover the key information from this subsection. Based on the teacher’s responses, you’ll further inquire to get a comprehensive understanding. Make sure to ask specific questions about the subsection’s content and avoid repeating queries from prior discussions. Information Provided:
1. Section Title: ...
2. Subsection Title: ...
3. Section Summary: ...
4. Bold Terms in Section: ...
5. Learning Objectives: ...
6. Concepts in Section: ...
7. Section Introduction: ...

heading: Previous Conversation:
text: Student:...
Teacher:...
*Note:* Frame your questions considering the information above and ensure they’re relevant to the content. Do not ask
questions about information you already
have. Only ask one question at a time.
Expected Output: Please phrase your question as a string.
Prompt for simulating teacher Task: You are a teacher preparing to answer a student’s question about a subsection of a textbook. The student’s question is: {question}. Provide a concise, specific response, ensuring it’s not a summary and distinct from any previous answers you’ve given.

heading: Information Provided:
text: 1. Section Title: ...
2. Subsection Title: ...
3. Subsection Content: ...
4. Section Summary: ...
5. Bold Terms in Section: ...
6. Learning Objectives: ...
7. Concepts in Section: ...
8. Section Introduction: ...

heading: Previous Conversation:
text: Student:...
Teacher:...
*Note:* When crafting your response, consider all the information above. Be sure your answer directly addresses the student’s question and is not a repetition of prior information. Expected Output: Please phrase your answer as a string.
Prompt for Persona (Single Instance) The prompt for the Persona (Single Instance) method is shown below. It uses one prompt to generate one dialogue. Task: generate a conversation between a student and a teacher using the given section. Introduction:
1. The conversation should contain 6 question-answer pairs.
2. The output conversation should be in this format: student: ... teacher: ... student: ...
3. The given section: ...

heading: C.4 QFactScore Implementation
text: For computing the embeddings of questions and answers, we use the “msmarco-distilbert-cos-v5” model from (Reimers and Gurevych, 2019). This model is suitable for computing cosine similarity and performs well in our task.
It is important to ensure that the QA model used in QFactScore is different from the QA model used
for generating dialogue datasets. This is because if the same QA model is used, the predicted answer is likely to be similar to the original answer in the dialogue. In QFactScore, we use the ’distilbertbase-cased-distilled-squad’ model, which differs from the GPT-3.5, T5, and Flan-T5 models that we used for generating the dataset.
QFactScore computes as the below equation. For each QA pair, it computes the cosine similarity between the embeddings of the QA model’s predicted answer and the original answer. Then, it assesses the similarity between the embeddings of the question and answer. The final score is the weighted sum of two similarity scores. The weight can be adjusted according to different applications, in our study we use α = 1 and β = 1.
α · sim(QA(qt, S), at) + β · sim(qt, at) (3)
We further evaluate the correlation between QFactScore and human evaluation of Factual Consistency in Appendix E. We also provide correlation between 1 - Overlap(at,a<t) and human evaluation of Informativeness in Appendix E.

heading: D Methods Comparison
text: The details of different types of Student Models and Teacher’s Models of each generation method and the detail of corresponding input are listed in Table 7. The term “Formatting” refers to formatting information, which contains a title, summary, introduction, learning objectives, bold terms, and key concepts from textbooks, which is introduced in Section-3.1. The “COPY” in the teacher’s model of Dialogue Inpainting indicates this method just copying a sentence from the textbook as the answer.

heading: E Metric Evaluation
text: To validate the effectiveness of the metrics introduced in this study, we calculate both Pearson and Spearman correlations between the metrics’ outcomes and the corresponding results from human evaluations. The results are shown in Table 8 and Table 9. The “1 - Overlap(at,a<t)” exhibits a Pearson correlation of 0.81 and a Spearman correlation of 0.77 with the Informativeness score in human evaluation, both with p-values below 0.005, suggesting that this F1 score could effectively represent Informativeness in evaluations.
On the other hand, QFactScore exhibits a Pearson correlation of 0.35 and a Spearman correlation
of 0.38 with Factual Consistency in human evaluation, both with p-values below 0.005. We interpret this as indicative of a moderate correlation, suggesting that this metric can approximate factual consistency to a certain extent. When comparing the correlation results with existing methods, including the use of GPT-3.5 scores derived from prompts, QuestEval, and QrelScore, the findings indicate that QFactScore’s correlation score surpasses others. However, Factual Consistency is a nuanced criterion that necessitates an assessment of whether the answer accurately addresses the question within the given context. Existing metrics struggle with this task, highlighting the need for more comprehensive evaluations in the future.

heading: E.1 Metrics Results Details
text: We provide the complete results of different metrics for datasets in four domains in this section. The results are shown in Table 10 and Table 11.

heading: F Human Evaluation Details

heading: F.1 Experiment Details
text: We adopt a human evaluation approach to assess the performance of dialogues generated by various methods. We recruit four annotators who have master’s degrees in Math, Science, Social Science, and Business. The annotators have educational backgrounds in Europe and Asia and are aged between 20 and 25. We recruit them by advertising on social media and reward them with non-monetary gifts for each annotator. As all annotators are satisfied with this payment, we consider this as adequate. To alleviate the burden on participants, we select 3 models from each method category for evaluation. To ensure the consistency of results across different domains, we choose datasets from four textbooks, each covering a different subject area: mathematics, business, science, and social sciences. From each textbook, we randomly select a subsection. For each subsection, we generate one dialogue using a different method, preparing each dialogue separately for evaluation. We use only the first 12 turns (6 QA pairs) of each dialogue for evaluation, similar to what is described in Section 5.1.2. During the
evaluation, each of the three participants receive 12 dialogues, with every dialogue corresponding to a related textbook subsection. Evaluators rate each question-answer (QA) pair within a dialogue based on eight criteria. The overall evaluation score for a dialogue is determined by averaging the scores of all its QA pairs. The specific evaluation criterion and corresponding questions are detailed in Table 12. Participants respond to each question with “yes” or “no”. The “yes” is coded as a score of 1, while the “no” is coded as a score of 0.
We provide the specific question the participants will be asked during human evaluation as shown in Table 12. The task is straightforward, we provide QA pairs for evaluation in an Excel file and the annotators just read the QA pair and give a score based on their judgement of each question.
The overall Fleiss’ kappa score for the annotations of the four annotators is 0.744. We further show the Cohen’s Kappa score between each participant in Table 13, which proves that each pair of participants has a substantial agreement.

heading: F.2 Disclaimer for Annotators
text: Thank you for participating in our evaluation process. Please read the following important points before you begin:
• Voluntary Participation: Your participation
is completely voluntary. You have the freedom to withdraw from the task at any time without any consequences.
• Confidentiality: All data you will be working with is anonymized and does not contain any personal information. Your responses and scores will also be kept confidential.
• Risk Disclaimer: This task does not involve any significant risks. It primarily consists of reading and scoring QA pairs.
• Queries: If you have any questions or concerns during the task, please feel free to reach out to us.
F.3 Instructions for Experiments Thank you for participating in our evaluation experiment. The data collected through this process will be used to assess the quality of our methods.
Follow these steps to score each QA pair:
1. Accessing the Data: Open the provided Excel file, which contains the QA pairs for evaluation.
2. Scoring Each QA Pair: For each pair, read the question and the corresponding answer carefully.
3. Scoring Scale: Answer each question with “yes” or “no”.
4. Entering Scores: Enter your score for each QA pair in the designated column in the Excel sheet. Please stick to the scoring scale provided.
5. Consistency: Try to maintain consistency in your scoring. Refer to the example evaluations provided if you’re unsure.
6. Completion: Once you have scored all the QA pairs, save the file and return it to us as instructed.
We appreciate your time and effort in this task.

heading: G Fine-tuning for Educational Chatbots Details
text: We source four textbooks from the OpenStax website for our study. These include ’Introductory Statistics’ for math, ’Business Ethics’ for business studies, ’Physics’ for science, and ’Psychology 2e’ for social science. We use the entire textbook dialogue dataset for fine-tuning.
In line with the methodology described in (Macina et al., 2023a), the models with fine-tuning are trained for 10 epochs during fine-tuning and 10 epochs for further training. The models without fine-tuning are trained for 10 epochs during training. For CoQA CNN and MCTest dialogue datasets, during either fine-tuning or training, we use 60% of data for training, 20% for validation, and 20% for testing. For the NCTE dataset, we randomly select 10,000 dialogues for training, 2,000 dialogues for validation, and 2,000 dialogues for testing. We set an initial learning rate of 6.25e5 and employ linear learning rate decay without warmup. For model optimization, we utilize checkpoints from the transformers library (Wolf et al., 2020). The negative log-likelihood of the groundtruth response is minimized using the AdamW optimizer, as detailed in (Loshchilov and Hutter, 2019). We assess model performance using the sacrebleu implementation of the BLEU metric, following (Post, 2018). We use one V100 GPU to train the model; the FLAN-T5-LARGE model has 0.8 Billion parameters and takes 7 hours to train. The result in Table 2 is the average of 3 runs.

heading: G.1 Examples of Results
text: We demonstrate an example of a comparison between the ground truth, the prediction from the model without fine-tuning, and the prediction from the model with fine-tuning in Table 15. These dialogues are based on the MCTest split of the CoQA dataset and the predictions are generated based on the same context information as shown in Table 14. From Table 15, we find that in the third answer, the prediction example without fine-tuning is overly verbose, suggesting a poor grasp of the question, ’What was the first thing she did that morning?’ which focuses on the person’s initial action. In contrast, the model without fine-tuning simply replicates the entire context sentence. On the other hand, the prediction with fine-tuning accurately mirrors the ground truth dialogue, effectively comprehending the concept of the “first thing” and suitably
condensing the sentence from the context.

heading: G.2 Generated Synthetic Datasets Overview
text: We provide the overview of our generated dataset in Table 16.
G.3 Implementation Details For each educational chatbot, we use the code framework of Daheim et al. (2023) with an initial learning rate of 6.25e-5 and linear learning rate decay without warmup. For model optimization, we utilize checkpoints from the transformers library (Wolf et al., 2020). The negative log-likelihood of the ground-truth response is minimized using the AdamW optimizer, as detailed in (Loshchilov and Hutter, 2019). We assess model performance using the sacrebleu6 implementation of the BLEU metric, following (Post, 2018). We use one V100 GPU to train the model. The FLAN-T5-LARGE model has 0.8 billion parameters and takes 7 hours to train.

heading: G.4 Details of Insufficient Follow-up Ability of the System
text: To evaluate the follow-up ability of the system, we conduct a random examination of 12 dialogues of each model. Our analysis focuses on the continuity and depth of the QA pairs within each dialogue. Although the pairs consistently discuss the same
6https://pypi.org/project/sacrebleu/
topic, we observe a lack of in-depth follow-up on previously mentioned aspects. For instance, instead of a pattern where an initial dialogue introduces a piece of knowledge and subsequent dialogues delve deeper into that same knowledge, the conversations often shift to other related aspects. We present an example of a dialogue demonstrating the insufficient follow-up ability of the Persona-based model in table 17.

heading: G.5 Example of Generated Dialogues
text: We show examples of generated dialogues from each model in this section. Example dialogues generated are shown in Table 19, all these dialogues are generated based on the same context and formatting information as shown in table 18.

heading: Future Work
text: Existing metrics struggle with this task, highlighting the need for more comprehensive evaluations in the future