Chain-of-Exemplar: Enhancing Distractor Generation for Multimodal Educational Question Generation (2024)

Abstract: Multiple-choice questions (MCQs) are important in enhancing concept learning and student engagement for educational purposes. Despite the multimodal nature of educational content, current methods focus mainly on text-based inputs and often neglect the integration of visual information. In this work, we study the problem of multimodal educational question generation, which aims at generating subjectspecific educational questions with plausible yet incorrect distractors based on multimodal educational content. To tackle this problem, we introduce a novel framework, named Chainof-Exemplar (CoE), which utilizes multimodal large language models (MLLMs) with Chainof-Thought reasoning to improve the generation of challenging distractors. Furthermore, CoE leverages three-stage contextualized exemplar retrieval to retrieve exemplary questions as guides for generating more subject-specific educational questions. Experimental results on the ScienceQA benchmark demonstrate the superiority of CoE in both question generation and distractor generation over existing methods across various subjects and educational levels.

heading: 1 Introduction
text: Multiple-choice questions (MCQs) are important in education for promoting deep and extensive knowledge acquisition. Research (Davis, 2009) indicates that well-crafted questions with educational intent are closely linked to heightened student engagement and achievement. A key aspect in question generation is the quality of the distractors (Gierl et al., 2017). Questions with inadequate or lowquality distractors are less challenging and easier to solve. Generating plausible, yet incorrect distractors is crucial in educational contexts, as effective and challenging distractors can significantly enhance students’ reading comprehension and contribute to their overall academic success. However,
∗ Corresponding author.
it is costly and time-consuming to manually produce MCQs, since even professional test developers do not manage to write more than three or four good MCQs per day (Kim et al., 2012).
To alleviate the human labour, automatic MCQ generation has received extensive attention. Previous research (Berre et al., 2022; Liang et al., 2018; Ren and Zhu, 2021; Qiu et al., 2020) primarily focuses on text-based inputs for MCQ generation, while generating MCQs from multimodal contexts is still relatively underexplored. Such emphasis on text often leads to underutilization of visual information, which is prevalent within educational content, such as textbooks (Lu et al., 2022) or examinations (Zhang et al., 2023a). Additionally, some latest studies (Wang and Baraniuk, 2023) neglect the creation of challenging and thought-provoking distractors, which is essential for high-quality educational question generation. Moreover, questions generated by current methods tend to be too general and not tailored for specific subjects or educational levels. As illustrated in Figure 1, the question generation model is tasked with creating educational questions from a biology textbook that includes both textual descriptions and visual illustrations. The example question focuses on subject-specific knowledge, and the accompanying distractors are carefully crafted to be plausible yet incorrect, enhancing the educational value of the questions.
In the light of these challenges, we propose a novel framework named Chain-of-Exemplar (CoE), which combines retrieved exemplars and Chain-ofThought (CoT) reasoning to generate educational
7978
questions and distractors from multimodal inputs of texts and images. Specifically, we employ multimodal large language models (MLLMs) to encode multimodal contexts and incorporate them into a three-stage multimodal-CoT framework that separates question generation, rationale generation, and distractor generation. The CoT helps trigger the reasoning capability of MLLMs leading to the generation of plausible and confusing distractors. Meanwhile, to generate more specialized questions for educational purposes in specific subjects, we leverage retrieved exemplary educational questions as demonstrations to guide the generation. Finally, we adopt an easy-to-apply multi-task training strategy to finetune our generative models.
The main contributions of this work can be summarized as follows:
• We propose a three-stage framework, namely CoE, to generate customized questions and plausible distractors, via multi-task finetuning MLLMs to perform multimodal-CoT reasoning.
• To enhance the generation of specialized educational questions in specific subjects, we utilize retrieved exemplary educational question as demonstration to guide the generation.
• Experimental results on ScienceQA benchmark show that CoE outperforms existing methods and effectively takes advantage of MLLMs. Our code will be released via https://github.com/ Luohh5/Chain-of-Exemplar.

heading: 2 Related Works
text: Question Generation Question generation (QG) (Pan et al., 2019) plays a crucial role in applications like conversational systems (Gao et al., 2019; Do et al., 2023; Zeng et al., 2023; Deng et al., 2022) and intelligent tutoring systems (Xu et al., 2022; Yao et al., 2022; Zhang et al., 2022; Zhao et al., 2022; Dugan et al., 2022; Deng et al., 2023b). Evolving from prior research based on syntactic trees or knowledge bases (Heilman and Smith, 2010; Kumar et al., 2015), most existing studies typically adopt deep neural networks (Du et al., 2017; Li et al., 2019; Dong et al., 2024) for question generation. With the advent of pre-trained and large language models (PLMs/LLMs), recent works (Bulathwela et al., 2023; Wang et al., 2022; Wang and Baraniuk, 2023) design various finetuning strategies to enhance the QG capabilities
of language models. Regarding educational purposes, multiple-choice question generation (Berre et al., 2022) holds great importance, where distractor generation (Ren and Zhu, 2021; Qiu et al., 2020) plays a crucial role. Apart from text-based inputs, there is also a growing body of research focusing on QG from images (Mostafazadeh et al., 2016; Li et al., 2018). However, it is worth noting that most existing work focuses on uni-modal QG, leaving the potential of multimodal QG largely unexplored.
Multimodal Question Answering Since question generation serves as the inverse task of question answering (QA), addressing the challenges in this field effectively necessitates drawing insights from QA studies. Due to the multimodal nature of information flow in real-world applications, researchers (Hannan et al., 2020; Talmor et al., 2021; Luo et al., 2023) emphasize the importance of answering questions that require information across multiple modalities, which is typically referred as multimodal question answering. Notably, several studies have focused on multimodal question answering in educational contexts, such as textbookbased (Lu et al., 2022) and exam-based (Zhang et al., 2023a) questions. Generating questions based on these contexts holds great potential for constructing intelligent tutoring systems and facilitating personalized learning experiences for students.
Chain-of-Thought Reasoning Recently, to solve complex reasoning tasks, CoT prompting (Wei et al., 2022) is proposed to decompose complex problems into a series of intermediate steps by prompting LLMs. Subsequently, the CoT reasoning has been effectively applied in various contexts, including multi-modal reasoning (Zhang et al., 2023b; Wu et al., 2023), multi-lingual scenarios (Shi et al., 2023; Qin et al., 2023), dialogue systems (Wang et al., 2023a; Deng et al., 2023a), and knowledge-driven applications (Trivedi et al., 2023; Wang et al., 2023b). Apart from these, there has been a surge in the development of other Chainof-X methods and most of them primarily focus on augmenting the LLMs with guidance to improve reasoning capabilities. For instance, Chainof-Knowledge (Li et al., 2023) augments LLMs by dynamically incorporating grounding information from heterogeneous sources for more factual rationales. Chain-of-Note (Yu et al., 2023) augments LLMs with a series of reading notes to retrieve
documents for more precise and contextualized reasoning. Inspired by these works, we equip LLMs with retrieved exemplars for contextual knowledge supplementation and guidance for generation.

heading: 3 Method
text: Given a correct answer A and multimodal context C = {I, T}, where I represents the image and T represents the texual paragraph, the task of multimodal educational question generation aims to generate a relevant question Q and several distracting answers A′. The overview of the proposed CoE framework is illustrated in Figure 2.

heading: 3.1 Model Architecture
text: As illustrated in Figure 2, CoE is composed of four distinct modules: a question generator module GQG, a rationale generator module GRG, a distractor generator module GDG, and a Contextualized Exemplar Retrieval (CER) module R. The question generator, rationale generator, and distractor generator utilize the same pre-trained multimodal language models (e.g., Qwen-VL) as the backbone with sharing weights. By employing these three generators, we introduce a CoT reasoning strategy to decompose multi-step problems into intermediate reasoning steps (rationale) and then generate the distractors. To guide the generation, we introduce a similar Contextualized Exemplar Retrieval module (Section 3.2) to retrieve the most relevant example from training data and use it as demonstration for a given test instance.

heading: 3.2 Contextualized Exemplar Retrieval
text: In order to retrieve a similar sample as exemplar E for more subject-specific generation, we introduce a Contextualized Exemplar Retrieval (CER) module to discern the analogy between each sample in training data D and associate them, as shown in
Figure 2. Specifically, we first encode the attribute information (i.e., textual context T , answer A, and question Q) of each example into a vector using the AnglE (Li and Li, 2023).
Vt = M(T ), Va = M(A), Vq = M(Q), (1)
where M(·) and V denote the encoder and vector representations, respectively. All the vectors lie in a latent sample space that contains rich semantics. If two vectors are close in the latent space, they are more likely to share similar information in analogous field. Subsequently, we calculate the cosine similarity of each attribute vector between the given testing instance S and each other samples Si ∈ D, then retrieve the nearest neighbor in the latent space as the most relevant example:
I = argmax i∈{1,2...,N} max(Simit, Sim i a, Sim i q),
Simit = (V it)TVt∥∥V it ∥∥ 2 ∥Vt∥2 , Simia = (V ia)TVa
∥V ia∥2 ∥Va∥2 ,
Simiq = (V iq)TVq∥∥V iq ∥∥ 2 ∥Vq∥2 ,
(2)
where I denotes an index of the most similar sample among all N samples and E = SI . We concatenate the test instance with the retrieved exemplar into a prompt for formatted input and feed it into the generator modules:
XQG = {A, T, I, EQG}, (3) XRG = {Q,A, T, I, ERG}, (4) XDG = {Q,R,A, T, I, EDG}, (5)
where EQG contains exemplar image, context, answer and question while ERG and EDG are further
expanded with rationale and distractors. In the subsequent generation process, the retrieved exemplar provides supplementary contextual knowledge that may not be present in test instance’s context and exerts flexible control of the output to make its style similar to the exemplar, which is especially effective for example with limited context. In this manner, the CER module retrieves relevant information as supplementary for the original sample to ground the generation on the subject at hand.

heading: 3.3 Chain-of-Exemplar Reasoning
text: To construct the framework of Chain-of-Exemplar, we combine the CER module and Chain-ofThought (CoT) reasoning to generate educational questions and distractors. Specifically, The CoE reasoning framework consists of three generation stages: (i) question generation, (ii) rationale generation, and (iii) distractor generation. All three stages share the same model architecture but differ in the input and output formats.
Question Generation In the question generation stage, we feed the question generator with retrieved exemplar EQG , answer input A, and context input C including textual paragraph T and associated image I . The primary objective is to train a question generation model GQG:
Q = GQG(A, T, I, EQG). (6)
Rationale Generation In the rationale generation stage, the generated question Q is appended to the original input XQG = {A, T, I, EQG} and the exemplar EQG is supplemented with corresponding rationale as ERG to construct the further input in the second stage, XRG = {Q,A, T, I, ERG}. Then, we feed the updated input to the rational generation model to generate intermediate reasoning as the rationale.
R = GRG(Q,A, T, I, ERG), (7)
Distractor Generation Similarly, the input in the final distractor generation stage is constructed by expanding the exemplar ERG with corresponding distractors and concatenating the generated rationale R with the previous input XRG as XDG = {Q,R,A, T, I, EDG}. Subsequently, we feed the modified input to the distractor generator by
A′ = GDG(Q,R,A, T, I, EDG), (8)
where A′ denotes the plausible yet incorrect answers for the question Q.

heading: 3.4 Multi-task Training Procedure
text: After formatting all prompt inputs, we perform instruction fine-tuning on a multimodal large language model in a multi-task way. Specifically, we assemble the formatted data by combining and shuffling all examples from the three tasks: question generation, rationale generation, and distractor generation. Following the teacher forcing method, we utilize the groundtruth question and rationale as input in distractor generation. Then we minimize the sum of negative log-likelihood loss LNLL averaged over tokens in three generation tasks as our training objective:
LNLL = − 1
L
L∑
l=1
yl log ( exp (yl)∑l i exp (yi) ) , (9)
Ltotal = LQGNLL + LRGNLL + LDGNLL, (10)
where L is the max length of output sequence, yl and yl denote the l-th token in the groundtruth sequence and prediction sequence respectively. By training the MLLMs on these tasks simultaneously, our goal is to prevent intermediate errors during CoT training that may disrupt reasoning, as well as induce the model being more robust to the wording choices of the prompts.

heading: 3.5 Inference
text: The inference phase also consists of question generation, rationale generation, and distractors generation stages. Given the image I , context T , answer A, and retrieved exemplar EQG , the question generator generates corresponding questions Q for next stage. Subsequently, the rationale generator utilizes all the aforementioned inputs along with the generated question and expanded exemplar ERG to generate rationales R for intermediate reasoning. Finally, the distractor generator use all the previous inputs, including the generated rationale and augmented exemplar EDG , to predict plausible distractors A′. It’s worth noting that we only calculate the maximum between answer and context similarity for exemplar retrieval since the question for the test instance is not given during inference.

heading: 4 Experiment

heading: 4.1 Experimental Setups
text: Datasets We conduct the experiments on the reversed ScienceQA dataset (Lu et al., 2022). ScienceQA is the first large-scale multimodal educational dataset that annotates detailed lectures and
explanations for the answers. It contains 21,208 multimodal science questions with rich domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills, showing outstanding generalizability across different domains. The benchmark dataset is split into train, validation, and test sets with 12,726, 4,241, and 4,241 examples, respectively. Note that in order to transform the format from QA to QG setup, we reverse the ScienceQA data by utilizing the context and correct answer as input, and generating the corresponding multichoice question as output. The details of dataset statistics are presented in Table 1.
Evaluation Metrics We adopt both automatic and human evaluation to measure the performance of our method. Specifically, we choose 2 automatic evaluation metrics including BLEU-4 (Papineni et al., 2002) and ROUGE-L (Lin, 2004) for question generation, both of which have been widely used in existing QG works. Additionally, we choose ROUGE-L and Accuracy (Chung et al., 2020) as automatic evaluation metrics for distractor generation. Specifically in Accuracy, we replace the origin options in ScienceQA with a combination of correct answer and the generated distractors, and leave the rest of the data unchanged. We adopt a multimodal question answering model (Zhang et al., 2023b) (trained by ScienceQA dataset) to evaluate the accuracy of the "modified" multiplechoice questions. Therefore, a higher accuracy score indicates poorer generation quality.
Baselines We compare CoE with the state-of-theart (SOTA) methods in ScienceQA, including VLT5 (Yeh et al., 2022), MultiQG-Ti (Wang and Baraniuk, 2023), and Multimodal-CoT (Zhang et al., 2023b). Note that we train the Multimodal-CoT by utilizing reversed task format of ScienceQA for question and distractor generation task. Due to limited prior work on automatic multimodal question and distractor generation, we use off-the-shelf model APIs as the baselines. Specifically, we use ChatGPT API (Ouyang et al., 2022) with zero-shot
and in-context learning (Kaplan et al., 2020) with up to three examples, each of which is formatted exactly the same as our preprocessed data points in the ScienceQA dataset. More details about ChatGPT baselines and other experimental setups are presented in Appendix A

heading: 4.2 Evaluation on Question Generation
text: We first conduct evaluation on the question generation, including both automatic evaluation and human evaluation.

heading: 4.2.1 Automatic Evaluation
text: Table 2 presents the comparison of the automatic evaluation results of CoE with previous state-ofthe-art models, which demonstrates that all the strong baselines fail to compete with CoE on both BLEU4 and ROUGE-L. Among the baselines, MultiQG-TI and Multimodal-CoT, which instantiate the question generator with multimodal large language models, largely outperform VL-T5 which simply extends pre-trained language models with visual understanding ability. Meanwhile, compared with MultiQG-TI that leverages image captions in the context to provide vision semantics, Multimodal-COT achieves much better performance by utilizing image features. Furthermore, the results clearly show that ChatGPT fails at the multimodal QG task in our setting. Although its performance steadily improves with more examples in the in-context learning setting, ChatGPT trails CoE by a significant margin.
Additionally, among the 3 subject classes, all the question generation baselines consistently demonstrate superior performance in social science (SOC) while exhibiting the lowest performance in language science (LAN). They also achieve performance gain for the questions with paired images (IMG), but perform poorly in the absence of any textual or image hints (NO). Moreover, the performance of CoE exhibits high consistency across different subjects and grades, which justifies the generalizability of the framework in education field.
However, the BLEU-4 and ROUGE-L metrics solely focus on evaluating the exact match between the generated question and the groundtruth, neglecting the aspect of question diversity. To address this concern, we incorporate an additional metric to evaluate the question diversity automatically and report it in Appendix B.

heading: 4.2.2 Human Evaluation
text: We further conduct human evaluation to evaluate the quality of the generated questions and investigate if the generated distractors can confuse the examinees in the real human test. For question generation, we randomly select 50 question samples generated by different methods and employ three annotators with good English background to rate them from 1 (worst) to 5 (best) based on 4 metrics: (1) Readability measures whether the generated questions are easy to read and understand for students of corresponding grade; (2) Appropriateness examines whether the generated questions are aligned with the corresponding subjects; (3) Complexity estimates the level of reasoning or cognitive effort required to answer the generated question for students of corresponding grade; (4) Engagement measures whether the students find the questions engaging and have interest in answering the questions. The annotator guideline is presented in Appendix E.
Table 3 illustrates the human evaluation results of question generation. Although the groundtruth questions win the highest score on all the metrics, our proposed CoE outperforms all the rest baselines and achieves a very close performance to the groundtruth. Furthermore, an average score of 4.48 indicates that our model can reliably generate challenging and thought-provoking educational questions that exhibit impressive readability, appropriateness, complexity, and engagement. Notably, the usage of MLLMs brings huge results difference between Multimodal-CoT and VL-T5. Interestingly, although ChatGPT performs poorly on most metrics, it demonstrates superior performance in readability and engagement in comparison to VLT5, which indicates its strong few-shot learning capability of generating readable paragraphs.

heading: 4.3 Evaluation on Distractor Generation
text: We then evaluate the performance in terms of distractor generation for constructing multiple-choice questions, also including both automatic evaluation and human evaluation.

heading: 4.3.1 Automatic Evaluation
text: Table 4 summarizes the automatic evaluation results for distractor generation. Similar to QG, MultiQG-TI and Multimodal-CoT demonstrate superior performance in comparison to VL-T5 due to the strong cognition and generation capabilities of MLLMs. Notably, giving the credit to the leveraging of CoT reasoning, the distractors generated by Multimodal-CoT are more confusing and challenging than MultiQG-TI. Similarly, there remains a significant disparity between the performance of ChatGPT and CoE.
In further analysis of the results across 3 subjects, we observe that the performance of all evaluated methods does not exhibit consistent superiority within a specific subject, which is different from QG. In fact, our CoE demonstrates its highest performance in the domain of natural science (NAT), while yielding the poorest performance in social science (SOC), showcasing a notable deviation from QG outcomes. Besides, CoE achieves impressive results in terms of the ROUGE-L score for questions accompanied by paired text (TXT), albeit with compromised accuracy. Conversely,
questions lacking both paired text and image (NO) exhibit a reduction in ROUGE-L score, without impacting accuracy, which highlights the strong adaptability in no contexts distractor generation of CoE. Similarly, the high performance consistency across different subjects and grades of our CoE framework provides further evidence of its generalizability.

heading: 4.3.2 Human Evaluation
text: Moreover, we invited another three annotators to answer the selected 50 MCQs with the generated distractors from different methods as well. The Accuracy of their answering would serve as a human evaluation metrics for assessing the quality of generated distractors. Besides, we also adopt a 5- point scale for other 3 metrics to evaluate the quality of generated distractors including: (1) Overlap examines whether the generated distractors are completely overlapping with the correct answer; (2) Plausibility estimates whether the generated distractors are semantically relevant to the given context and question; (3) Distinctiveness measures the originality of the generated distractors in comparison to the groundtruth. Elaborate evaluation guideline is depicited in Appendix E.
Table 5 summarizes the human evaluation results of distractor generation. In general, our CoE outperforms all other baselines and even surpasses the groundtruth in accuracy, which indicates that the distractors generated by CoE are distinctive and thought-provoking enough to distract humans from the correct answer. However, although we have adopted a series of strategies such as CoT and CER module to enhance the plausibility of generated distractors, the distinctiveness remains limited to improve significantly. Contrarily, due to the unknown pretrained knowledge inside the blackbox, ChatGPT often performs well in zero-shot and fewshot text generation, especially in terms of read-
ability and diversity. Therefore, it demonstrates impressive performance in both overlap and distinctiveness but falls short in plausibility, resulting in higher accuracy in annotators’ MCQ answering.

heading: 4.4 Ablation Study
text: We perform ablation studies to investigate the effects of the proposed approaches in terms of chainof-thought reasoning, Contextualized Exemplar Retrieval module, and multi-task learning, as presented in Table 6. There are several notable observations as follows:
• When dropping the Chain-of-Thought (CoT) reasoning, the distractors are directly generated based on context and question with associated answer, whose performance drops by 15.7% Acc and 0.9 R-L respectively. The results show that the intermediate rationale could indeed enhance the distractor generation.
• When we drop the Contextualized Exemplar Retrieval (CER) module, the overall performance declines a lot, especially in terms of a significant -14.3% drop in Acc, which demonstrates that adding the CER module indeed helps retrieve supplemental information for the original sample and benefit generation. Besides, More details about the ablation study of CER module are presented in Appendix C.
• When we use single-task learning in place of multi-task learning (MTL) as our finetune strategy, we can see declines in both generation tasks, specifically with a decrease of -6.4 in B-4 score and an increase of +13.1% in Acc score, which further verifies the effectiveness of utilizing multi-task learning to mitigate intermediate errors during CoT finetuning.
• As for the input context, dropping either the image or the textual paragraph results in a substantial decline in performance for both generation tasks, particularly with a significant -14.8 and - 21.0 decrease in B-4. It highlights the advantages of incorporating both visual and textual information when generating questions and distractors.

heading: 4.5 Case Study
text: To qualitatively evaluate the four modules in CoE, we visualize an example from ScienceQA in Figure 2. Based on the context, image, and answer input in this example, the question generator is obviously powerless to generate appropriate question without referring to the exemplar question "Which property do these three objects have in common?". Moreover, the exemplar rationale "An object has different properties ... property that all three objects have in common is salty" provides valuable information supplementary for the rationale generator and serves as a demonstration to guide the generation. Furthermore, generating such diverse and plausible distractors is benefit from the informative intermediate reasoning which explains the definition of "property" and describes the commonalities of the objects in image.
To further estimate how the CER module and CoT reasoning affect the question and distractor generation, we present generated examples in Figure 3 and Figure 4. The example in Figure 3 illustrates that when we add the CER module, the generated question "Which property do these four
objects have in common?" which asks about the "object property" exhibits more impressive complexity and appropriateness for education. Conversely, when dropping the exemplar, the generated question "What do the tastes of these four foods in images like?" appears simplistic and trivial for answering. Moreover, Figure 4 demonstrates that when we drop out the intermediate reasoning, the distractor generator fails to reason from the explanation regarding "what kind of animal’s feet have grabbing prey adaptation", which results in generating distinct distractor "bighorn sheep" and irrelevant distractor "octopus".

heading: 5 Conclusions
text: In this paper, we present a novel framework called Chain-of-Exemplar (CoE), which combines retrieved exemplars and Chain-of-Thought (CoT) reasoning to generate educational questions and distractors from multimodal inputs of texts and images. Specifically, we utilize MLLMs to encode multimodal contexts and incorporate them into a three-stage multimodal-CoT framework, namely question generation, rationale generation, and distractor generation. Meanwhile, we introduce a Contextualized Exemplar Retrieval (CER) module to retrieve exemplary educational questions as demonstrations to guide the generation. We finally adopt an easy-to-apply multi-task training strategy to finetune our generative models. Our experiments on ScienceQA benchmark demonstrate that CoE outperforms existing methods and achieves new state-of-the-art performance.

heading: Acknowledgements
text: This research/project is supported by the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP) Funding Initiative, and NExT Research Center. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
Limitations
Distinctiveness of Generation As mentioned in Appendix B, our proposed CoE can achieve a much better performance than most baselines in question generation. Meanwhile, we acknowledge the limitation of the question diversity compared to ChatGPT. Similar in Section 4.3.2, while the distractor generation of CoE exhibits a superior distinctiveness compared to the majority of baselines, it still falls short of meeting annotators’ expectations. In contrast to our CoE, which employ supervised learning and the diversity of generation heavily relies on the finetuning data, ChatGPT is able to generate highly distinctive question and distractors by utilizing in-context learning without any supervision. Consequently, there remains room for future research to explore effective finetuning strategies and investigate how to incorporate external knowledge or distill knowledge from ChatGPT into the open-source LLMs.
Hallucination Issues Another limitation of CoE is that our rationale generation module may suffer from the typical flaw of hallucination issues, i.e., making fabricated intermediate reasoning that is irrelevant to the context and answer. Hallucinated rationale would mislead the generation process, resulting in irrelevant and trivial distractors generation. One potential solution involves replacing the greedy decoding strategy used in Chainof-Thought reasoning with self-consistency (Wang et al., 2023c) by sampling diverse reasoning paths and selecting the most consistent rationale for distractor generation, which mimic multiple different ways of thinking. The greater the diversity of reasoning paths, the more authentic and reasonable rationale can be generated by CoT. We believe that this future research direction will prove valuable and promising in effectively tackling hallucination issues.
Exemplar Resource During training, our CER module retrieves domain-specific exemplars from training split of ScienceQA to guide the generation. However, the generators trained under the guidance of these exemplars greatly restricts the quality and diversity of the generated questions and distractors, primarily due to the strong dependence on training data. Actually, this limitation is not exclusive to our work. The educational MCQ generation heavily relies on training data in specific domain to generate questions and distractors for educational purpose. Therefore, we acknowledge the need for future research to explore methods for incorporating the retrieved exemplar with external knowledge to reduce the reliance on training data and enhance the generation quality.

heading: B Question Diversity
text: To measure the diversity of generated questions, we utilize Distinct-n scores (Li et al., 2016) as an automatic evaluation metric. Specifically, it calculates the number of distinct n-grams in corpus-level, with a higher count indicating a greater diversity of questions. We consider values of n ranging from 1 to 4. As presented in Table 7, the performance of all methods improves as the value of n increases from 1 to 4. Additionally, ChatGPT demonstrates the capability to generate highly distinctive questions by utilizing zero-shot or few-shot in-context learning without any supervision, which is similar to the performance in distractor generation. Except to ChatGPT, our CoE outperforms all other baselines, which indicates that the questions generated by CoE exhibit impressive appropriateness while maintaining high diversity.

heading: C Analysis of Exemplar Retrieval
text: Analysis of Exemplar Retrieval Strategies We construct an experiment by training the generators with different exemplar retrieval strategies to investigate whether the exemplar retrieval strategy influence the performance. Specifically, we utilize 3 retrieval strategies: Random, Maximum, and Summation. In "Random" strategy, exemplars are randomly selected from the training data. The "Maximum" strategy denotes using the argmax for the maximum between answer, context, and question similarities, while the "Summation" strategy denotes combining the three signals by summing up them as follows:
I = argmax i∈{1,2...,N} (Simit + Sim i a + Sim i q),
Simit = (V it)TVt∥∥V it ∥∥ 2 ∥Vt∥2 , Simia = (V ia)TVa
∥V ia∥2 ∥Va∥2 ,
Simiq = (V iq)TVq∥∥V iq ∥∥ 2 ∥Vq∥2 ,
(11)
As depicted in Table 8, the performance of randomly selected exemplars fails to compete with that of contextually retrieved exemplars in both question and distractor generation, which further justify that contextually retrieved exemplar indeed provides valuable and useful information to the generators and validate the effective of contextually retrieval. Moreover, retrieving exemplar using "Summation" strategy demonstrates superior performance in comparison to "Maximum" strategy, indicating that combining the three signals in exemplar retrieval yields more appropriate and relevant exemplars that benefit both question and distractor generation.
Effect of the Number of Exemplars To further analyse the potential impact of the number of exemplars on generation performance, we vary the number of exemplars, denoted as N , and retrieve the top N exemplars with the highest similarity. As shown in Figure 6, we observe an improvement in both question and distractor generation performance as N increases from 1 to 3, verifying the effectiveness and utility of information the exemplar provides to the generators. However, we note that when N = 2, the limitation of the maximum input length results in the truncation of certain content within the exemplar, preventing it from performing at its optimum capacity. Therefore, both the question and distractor generation performance improve slowly when N ≥ 2.

heading: D Analysis of Different Base Models
text: To analyse the generality of our CoE framework, we conduct an experiment to utilize other base models in place of Qwen-VL (Bai et al., 2023) as the backbone for question and distractor generation, including LLaVA (Liu et al., 2023), InstructBLIP (Dai et al., 2023), mPLUG-Owl (Ye et al., 2023), and VisualGLM-6B (Ding et al., 2021; Du et al., 2022). Note that we employ the same prompt for all base models to ensure fairness in the comparison. As summarized in Table 9, Qwen-VL outperforms all the rest base models, showcasing its high applicability and suitability in our framework. Generally, while there are slight difference in performance among the 5 base models, they consistently demonstrate superior performance in both question and distractor generation, which further confirms the effectiveness and versatility of our
CoE framework.

heading: E Guideline of Generation Quality Evaluation
text: We present the guideline of human evaluation for question and distractor generation quality in Figure 7.

heading: Future Work
text: Consequently, there remains room for future research to explore effective finetuning strategies and investigate how to incorporate external knowledge or distill knowledge from ChatGPT into the open-source LLMs. Hallucination Issues Another limitation of CoE is that our rationale generation module may suffer from the typical flaw of hallucination issues, i. e. , making fabricated intermediate reasoning that is irrelevant to the context and answer. Hallucinated rationale would mislead the generation process, resulting in irrelevant and trivial distractors generation. One potential solution involves replacing the greedy decoding strategy used in Chainof-Thought reasoning with self-consistency (Wang et al. , 2023c) by sampling diverse reasoning paths and selecting the most consistent rationale for distractor generation, which mimic multiple different ways of thinking. The greater the diversity of reasoning paths, the more authentic and reasonable rationale can be generated by CoT. We believe that this future research direction will prove valuable and promising in effectively tackling hallucination issues. Exemplar Resource During training, our CER module retrieves domain-specific exemplars from training split of ScienceQA to guide the generation. However, the generators trained under the guidance of these exemplars greatly restricts the quality and diversity of the generated questions and distractors, primarily due to the strong dependence on training data. Actually, this limitation is not exclusive to our work. The educational MCQ generation heavily relies on training data in specific domain to generate questions and distractors for educational purpose. Therefore, we acknowledge the need for future research to explore methods for incorporating the retrieved exemplar with external knowledge to reduce the reliance on training data and enhance the generation quality