Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models (2024)

Abstract: Teachers are important to imparting knowledge and guiding learners, and the role of large language models (LLMs) as potential educators is emerging as an important area of study. Recognizing LLMs’ capability to generate educational content can lead to advances in automated and personalized learning. While LLMs have been tested for their comprehension and problem-solving skills, their capability in teaching remains largely unexplored. In teaching, questioning is a key skill that guides students to analyze, evaluate, and synthesize core concepts and principles. Therefore, our research introduces a benchmark to evaluate the questioning capability in education as a teacher of LLMs through evaluating their generated educational questions, utilizing Anderson and Krathwohl’s taxonomy across general, monodisciplinary, and interdisciplinary domains. We shift the focus from LLMs as learners to LLMs as educators, assessing their teaching capability through guiding them to generate questions. We apply four metrics, including relevance, coverage, representativeness, and consistency, to evaluate the educational quality of LLMs’ outputs. Our results indicate that GPT-4 demonstrates significant potential in teaching general, humanities, and science courses; Claude2 appears more apt as an interdisciplinary teacher. Furthermore, the automatic scores align with human perspectives.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3138–3167 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction Large language models (LLMs) have demonstrated great performance in various natural language processing (NLP) tasks, including question answering (Saad-Falcon et al., 2023), information retrieval (Liu et al., 2023b), reasoning (Kojima et al., 2022), and generation (Chung et al., 2023), etc. Beyond these general NLP applications, LLMs are
also widely used in other domains, such as education. In the educational field, LLMs can now be used as substitutes for teachers. They can help automated teaching or assisted learning applications, thereby alleviating the pressure on human teachers. Additionally, LLMs can recommend appropriate elective courses based on a student’s knowledge state, learning style, and interests, automatically generating practice problems of corresponding difficulty levels, and identifying areas where a student is struggling to provide targeted improvement.
However, the capability of questioning is a crucial aspect in the educational field. As LLMs take on the role of teachers, can they pose high-quality questions like human educators? Therefore, evaluating what constitutes a high-quality question in education becomes necessary. According to Anderson and Krathwohl’s educational taxonomy (Anderson and Krathwohl, 2001; Elkins et al., 2023), we consider that high-quality questioning in the educational field must meet the following characteristics: i) achieve a higher level across the six domains including memory, understanding, application, analysis, evaluation, and creation; ii) be
3138
relevant to the given context; iii) comprehensively cover the content of the context, and iv) also reflect the important knowledge of this context. We consider that questions meeting these characteristics can effectively assess students’ knowledge levels, and LLMs capable of posing such questions can assume the role of competent human educators. The first characteristic is the most basic requirement for LLMs to act as human teachers, while the following three characteristics measure the excellence of LLMs in their role as a teacher.
Evaluating and enhancing the capability of LLMs to generate questions of high quality standards in the educational domain requires a benchmark. However, previous studies have mainly viewed LLMs from a student’s perspective, focusing on tasks like reading comprehension (Bai et al., 2023; Tran and Kretchmar, 2023; Chen et al., 2023; Cheng et al., 2023; Izacard and Grave, 2020; Kawabata and Sugawara, 2023; Zhou et al., 2023b,a) and exam evaluations (Zhang et al., 2023b; Huang et al., 2023; Zhong et al., 2023; Li et al., 2023; Wang et al., 2023; Zeng, 2023; Wei et al., 2023). However, these tasks focus on adopting contexts to passively answer questions or make reasoning, and these tests treat LLMs as students, assessing their abilities by how they answer questions, while the LLM’s questioning capability through generating educational questions is under-studied. Current education-related research is far from adequate to determine LLMs’ question raising capability as a teacher, and there isn’t a benchmark that studies the overall teaching abilities of LLMs, seeing them as teachers. Although some role-playing tasks (Shao et al., 2023) mimic professional dialogues but don’t truly assess the LLMs’ teaching capabilities. Therefore, if we want LLMs to assist in teaching effectively, we need to evaluate and enhance their teaching abilities, as possessing knowledge and guiding others to learn are distinct skills.
Therefore, in this paper, we have developed a benchmark for assessing whether LLMs generate high-quality questions in the field of education, guided by professional educational theories. Unlike general questioning, as shown in Fig. 1 (a), our benchmark requires that the generated questions not only be fluent and readable but also meet the fundamental characteristics proposed earlier (i.e. the first characteristic), as shown in Fig. 1(b). Specifically, we draw on Anderson and Krathwohl’s educational taxonomy (Anderson and Krathwohl, 2001) to prompt LLMs to generate questions
at six levels for each context. We select tasks from three domains, including general, single-discipline, and interdisciplinary domains, to more comprehensively assess the strengths of LLMs as teachers in various fields. Based on the four characteristics proposed earlier, we have also designed four evaluation metrics: consistency, relevance, coverage, and representativeness, to assess the value of questions posed by LLMs in the educational domain, thereby comprehensively evaluating the questioning capability of LLMs as teachers in education through evaluating their generated educational questions. Our experiments reveal that LLMs like GPT-4, Claude2, and GPT-3.5 demonstrate good questioning capability across domains as teachers in education through evaluating their generated educational questions. In summary, our contributions are threefolds:
• We introduce the problem of evaluating questioning capability in education as a teacher for LLMs through evaluating their generated educational questions, building a framework based on educational theory that includes six cognitive levels and tasks from three different domains.
• We establish four evaluation metrics to assess the questioning capability in education as a teacher of LLMs through evaluating their generated educational questions.
• We conduct experimental evaluations of 11 LLMs, providing quantitative standards and subject orientations for each LLM’s questioning capability as a teacher.
2 Datasets and Task Setups We propose a benchmark named Dr.Academy, which has tasks from three domains. The first two request LLMs to generate questions in the general and monodisciplinary domain, respectively, based on the six levels of Anderson and Krathwohl’s educational taxonomy (Anderson and Krathwohl, 2001), including memory, understanding, application, analysis, evaluation and creation. The third one requests LLMs to generate questions that intersect multiple subjects. The overview of Dr.Academy is shown in Fig. 2.
2.1 Context Construction Initially, we collect 10,000 contexts from the general domain and produce an additional 10,000 contexts specifically for the monodisciplinary domain.
In the general domain, the contexts are sourced from the SQuAD dataset (Rajpurkar et al., 2016), an extractive reading comprehension dataset derived from Wikipedia articles, and are utilized as the foundation for the LLMs to generate questions. In the monodisciplinary domain, we generate corresponding contexts for each of the multiple-choice questions in the MMLU dataset (Hendrycks et al., 2020), which covers a broad spectrum of subjects, with GPT4 1. These contexts include essential information related to the question and all candidate choices. The prompt for generating contexts is shown in Table 1. We also conduct manual evaluations on the generated contexts for the MMLU questions. In this process, we engage three graduate students from different disciplines to perform the evaluations. For each discipline, we randomly select 1% of the questions to undergo manual assessment. If these entries do not achieve a manual evaluation score of 4 (on a scale of 1-5), we will regenerate the contexts.
2.2 Task Setup
We have designed three tasks and each task requires LLMs to generate questions catering for the corresponding domain. Finally, these generated questions will be used to evaluate the questioning capability in education as a teacher of LLMs. The prompt for generating questions is shown in Table 8 (row “Generation”).
General domain tasks request an LLM to generate questions with the collected contexts from SQuAD based on the six levels of Anderson and
1https://chat.openai.com/
Krathwohl’s educational taxonomy (Anderson and Krathwohl, 2001), including memory, understanding, application, analysis, evaluation and creation. For instance, in Fig. 2 (a), at the memory level, a question might ask specific details like “What are the religious features of a school building?”; at the understanding level, it could be about reasons such as “Why is a school considered to have Catholic characteristics?”; at the creating level, questions could be more open-ended, involving imagination and design, etc. This task is designed to evaluate “which LLM is more suitable to be a general course teacher”.
Monodisciplinary domain tasks request an LLM to generate questions with the generated contexts from MMLU, focusing on either humanities (like history, geography) or sciences (like physics, chemistry), based on the same six educational levels. In science, for instance, a memory-level question might ask about element symbols and formulas, such as “What is the chemical formula for ammonium sulfate?”; an application-level question related to real-world phenomena, like “choose a substance that reacts with hydrochloric acid to produce carbon dioxide”. This task is designed to evaluate “which of the two LLMs is more suitable to act as a humanities teacher and a science teacher.”
Interdisciplinary domain tasks request an LLM to generate questions that cross multiple subject areas, reflecting each subject’s characteristics. For example, in Fig. 2(c), when merging literature and geography, a question might seek an explanation of the geographical phenomenon described in a poem’s line. In combining art and geography, a
Prompt
Craft a comprehensive and integrative textbook text that seamlessly weaves together the various knowledge points encompassed by the multiple-choice options. Begin by presenting a thorough exploration of each topic in individual paragraphs, ensuring the narrative flows logically from one subject to the next. The text should serve as a foundation that could logically precede the multiple-choice question. The goal is to offer a well-rounded and complete context that encapsulates the knowledge points mentioned in the options, setting the stage for the question to arise naturally from the presented material.
Table 1: The instruction prompt in generating contexts based on a given multiple-choice questions.
question might ask about the geographical features represented in a song. A less successful example of an interdisciplinary question is one where the involved disciplines are unrelated, such as asking about Einstein’s theory of relativity and then about the Cauchy inequality in mathematics. This question touches on physics and mathematics but lacks a meaningful connection between the two, making it not truly interdisciplinary. This task is designed to evaluate “which LLM is more suitable to act as a interdisciplinary teacher.”, qualifying if LLMs solving the problem requires understanding knowledge from both subjects.
2.3 Evaluation metrics
We adopt consistency, relevance, coverage, and representativeness to evaluate LLMs’ generated questions in the general and monodisciplinary domains, respectively, while using relevance and representativeness to evaluate questions in the interdisciplinary domain. The difference of metrics selection is because questions in interdisciplinary domain lack a comprehensive contextual framework. For instance, in reality, there is no distinct academic discipline like “historical geography.” This absence of a well-defined, unified context means that metrics such as coverage and consistency do not apply.
To validate the effectiveness of these metrics, we consult ten experts in education to rate the effectiveness of these metrics within the field of education on a scale of 1 to 5. They consistently award these metrics scores of 4 and above, which leads us to believe that these metrics are meaningful for evaluating questions in education. We also align these metrics with manual evaluations in Figure 6, indicating that our metrics are indeed significant within the field of education. Additionally, experiments are conducted to compare these metrics with human scoring in order to corroborate the validity and reasonableness of them (see Fig. 6). The prompt for evaluating questions is shown in Table 8 (see row “Evaluation”). Specifically, consistency is to assess whether the question accurately corresponds to a pre-defined level of the educational taxonomy, relevance is to assess whether the question is related to the provided text content or theme, coverage is
assessed by determining if all generated questions based on a context encompasses a major portion (over 50%) of this given context, representativeness evaluates whether a question captures the main content or core ideas of the text. Metrics are rated on a binary scale, with 1 for criteria met and 0 for not met, as shown in Fig. 3 and Table 2. We adopt GPT4 to score each question three times. A question that scores 1 in two out of three instances meets the metric’s requirement.
3 Experiments In this section, we conduct extensive experiments to evaluate different LLMs’ questioning capability through evaluating their generated educational questions in the proposed Dr.Academy.
3.1 Experimental Setups Our experiments are conducted on 8 Nvidia A100 GPUs, each with 80GB of memory, and we use PyTorch 2 in Python 3. We set the maximum sequence length for both input and output sequences to maximum 1000 tokens.
3.2 Datasets, Baselines and Metrics The baseline LLMs for this evaluation are BLOOM7B (Workshop et al., 2023) BLOOM-176B (Workshop et al., 2023), Claude2 (Bai et al., 2022), Falcon-7B (Almazrouei et al., 2023), Falcon180B (Almazrouei et al., 2023), GPT3.5 (Brown
2https://pytorch.org/ 3https://www.python.org/
et al., 2020), GPT4 (OpenAI, 2023), LLaMA27B (Touvron et al., 2023), LLaMA2-70B (Touvron et al., 2023), Vicuna-7B (Chiang et al., 2023), and Vicuna-33B (Zheng et al., 2023). For the task of manual evaluation, we enlist the help of three graduate students specializing in education. We begin by informing the annotators about the purpose of each task and the scoring guidelines. Following this, we request them to score the responses. During the evaluation phase, we randomly select 1000 questions generated by each LLM for each task and ask three volunteers to manually evaluate the generated responses using the same criteria applied to GPT-4. To ensure the reliability and validity of the human ratings, we calculate the Inter-rater Agreement using Krippendorff’s Alpha (IRA). For ratings that exhibit low agreement (< 0.7), we remove the particular statement from consideration and replace it with a new one. This method guarantees the precision and consistency of our human assessment.
3.3 Main results
Question 1: Which LLM is more suitable to be a general course teacher? Answer 1: GPT4! The term “suitable” is used to assess the effectiveness of each LLM in different academic subjects. This evaluation helps in identifying which LLMs perform best in specific educational disciplines. In the general domain tasks, various LLMs demonstrate diverse performances as shown in Table 3 and Table 6. Specifically, GPT4 achieves a perfect score of 100% in both consistency and relevance, indicating its strong capability in understanding task requirements and generating relevant questions. However, its coverage score of 54.5% suggests there is room for improvement in generating questions that encompass more content. In representativeness, with a score of 80.1%, GPT-4 shows a good grasp of the context’s core content and viewpoints, crafting questions with depth and breadth. BLOOM-176B and Claude2 also score perfectly
in relevance, reflecting their excellent performance in linking questions to the context’s themes and content. However, their lower scores in coverage and representativeness indicate potential for improvement in capturing the full extent and core insights of the texts. Moreover, “Aver” in Table 3, Table 4 and Table 5 represent the average result of the corresponding dimensions under each domain, which are obtained using in-context learning (i.e. ICL). ICL is to introduce a human-written sample into the prompt which typically improves LLMs’ performance across all metrics, while most LLMs show a decline in the 0-shot setting, demonstrating the critical role of ICL in enhancing the quality of question generation.
Question 2: Which of the two LLMs is more suitable to act as a humanities teacher and a science teacher? Answer 2: Both are GPT4!
In the monodisciplinary domain tasks, LLMs are compared based on their performance in humanities and sciences as illustrated in Table 4 and Table 6. The results reveal that the majority of the LLMs perform marginally better in the scientific disciplines compared to the general domain. Specifically, GPT4 excels across all metrics, particularly in the science disciplines, where it scores higher than in the humanities, indicating great capability in handling science content. Following closely is Claude2, which nearly matches or equals GPT4 in Relevance and representativeness in the humanities,
demonstrating a deep understanding and effective processing of humanities content. Claude2 also maintains a high performance in the science disciplines. GPT3.5 shows competitive strength across the four metrics, especially in relevance and representativeness within the science subjects, approaching the leading performance of GPT4. BLOOM176B scores significantly higher in consistency within science compared to the humanities, and also demonstrates good capability in coverage and representativeness, suggesting its strengths in processing logical and scientific data.
Question 3: Which LLM is more suitable to be a interdisciplinary teacher? Answer 3: Claude2!
Results of LLMs in the interdisciplinary domain tasks are shown in Table 5 and Table 6. It shows that Claude2 outperforms other LLMs with scores of 89.1% in relevance and 93.3% in representativeness. Following closely is GPT4, with scores of 87.8% in relevance and 91.2% in representativeness, also indicating strong performance. GPT3.5 and LLaMA2-70B also show high scores, particularly in representativeness, suggesting their capability in understanding key textual content and generating in-depth questions. On the other hand, BLOOM-7B, Falcon-7B, and Vicuna-7B perform relatively poorly on both metrics. Specifically, BLOOM-7B scores below 40% in both relevance and representativeness, which may suggest a need for further enhancement in understanding interdisciplinary content and generating high-quality questions.
Question 4: Which LLM is more suitable to be a all-around teacher? Answer 4: GPT4!
We also comprehensively compare the performance of LLMs in three tasks as shown in Table 6 and Fig. 4. Specifically, in the general domain tasks, GPT4 scores the highest and ranks first. In the monodisciplinary domain tasks, including humanities and science, GPT4 also has the best performance. In the interdisciplinary domain tasks, Claude2 occupies the first rank. Finally, for
the comprehensive rating, GPT4 ranks first again with the highest score. Overall, GPT4 shows the best performance in most tasks, while Claude2 also demonstrates strong capability in certain tasks. Overall, GPT4, Claude2, and GPT3.5 perform well in these assessments, demonstrating their versatility and adaptability as high-performance models. On the other hand, BLOOM-7B and Falcon-7B tend to perform weaker in most fields, which may make them more suitable for specific application scenarios.
Question 5: What’s the relationship among metrics for various LLMs? Answer 5: Pairwise positive correlations!
We also analyze the relationship among four metrics in three tasks as shown in Fig. 5. Fig. 5 (a) and Fig. 5 (b) represent the relationship between four metrics of question quality generated by different LLMs in the general and monodisciplinary domains, respectively. The size of the circle indicates coverage, with larger circles covering more content of the text. The darker the color of the circle, the higher the relevance of the questions to the text. The “Average1” in the first graph represents a group of LLMs, which are zoomed in on the second
graph, and the “Average2” in the second graph represents a subset of these LLMs, which are further examined in the third graph. In Fig. 5 (a), analyzing the general domain tasks, we see a positive correlation between relevance and consistency across all three graphs. Representativeness also shows a positive correlation with relevance and consistency, but the correlation is weaker. As relevance and consistency increase, the LLMs have darker colors and larger circles, indicating higher relevance and coverage. Although larger LLMs like BLOOM176B show good coverage, not all models with large coverage have high relevance. For example, Falcon-180B does not perform as well as BLOOM176B in relevance, suggesting a need for balance between the breadth of text coverage and the accuracy of question generation. In the third graph, LLMs like GPT4 maintain high relevance while also achieving good coverage. In Fig. 5 (b), for the monodisciplinary domain tasks, the correlations between the four metrics are not as pronounced as in the general domain. The third graph shows little color variation, indicating that representativeness does not change much with increased relevance and consistency. However, there are LLMs like GPT4 that stand out in all metrics, shown in the top right corner with a dark color and large size. But GPT3.5, while showing good representativeness and relevance, has only average consistency. In Fig. 5 (c), analyzing the interdisciplinary domain tasks, generally shows a positive correlation between relevance and representativeness, although it’s not as clear in the second graph. Overall, LLMs like GPT4, Claude2, BLOOM-176B, and GPT3.5 perform well across all four metrics, while the 7B series models tend to perform less well. The metrics also tend to show positive correlations with each other.
Question 6: Is the automatic scores generated by GPT4 agree with human perspectives? Answer 6: Yes, the Pearson correlation coefficient reaches 0.947 and Spearman rank correlation reaches 0.87!
We adopt Pearson correlation coefficient that normalized to a 1-100 scale to investigate the difference between automatic and human scores for different LLMs as shown in Fig. 6. The human scores for each metric and the corresponding agreement of human annotators on these metrics are listed in Table 7. We find that automatic and human evaluations for LLMs generally agree, showing a high positive correlation with the Pearson correlation coefficient reaching 0.947 and Spearman rank correlation reaching 0.870. Specifically, GPT4 performs excellently in both automatic and human scoring, with minimal difference, indicating widespread recognition of its capability. Similarly, Claude2 has close scores in both evaluations, indi-
cating balanced performance in assessment tasks. It’s important to clarify that the process of generating questions and scoring them is separate. During the scoring phase, there is no knowledge of which LLM generates which question. We believe that even if other LLMs are used to evaluate GPT-4’s performance against a comparatively weaker 7b model, the results would still favor GPT-4, a conclusion also supported by human evaluations. The findings suggests that automatic scoring has the potential to partially replace human scoring for evaluating questioning capability in education as a teacher of LLMs through evaluating their generated educational questions.
3.4 Case study
We present a set of general domain questions generated by GPT4, identified as the leading LLM in this area, in Fig.7. The top-performing LLMs in monodisciplinary (Humanities), monodisciplinary (Science), and interdisciplinary domains have their
questions displayed in Fig.9, Fig.10, and Fig.8, respectively. Additionally, further examples from baseline LLMs are shown in figures ranging from Fig.11 to Fig.26. In Fig.7, the first question tests memory by asking about the true ruler of the Twilight Realm. The second requires understanding the reasons behind the Mirror of Twilight’s durability. The third applies Link’s transformation ability to his quest. The fourth analyzes the contrasting rules of Midna and Zant. The fifth evaluates the use of the Mirror as a punishment tool. Finally, the sixth encourages creating an alternative ending for the saga. These questions showcase GPT-4’s range from simple recall to creative thinking, and the generated questions can be adopted to make students quickly grasp the key content, indicating its strong questioning capability in education as a teacher through evaluating their generated educational questions.
We also show a good case in the interdisciplinary tasks generated by Claude2, which shows outstanding comprehensive capability that can effectively combining different subject knowledge to pose accurate and challenging questions. In chemistry and physics combination questions, it understands the basic principles of chemical reactions and the physical properties of bubble motion in liquids, demonstrating analysis and synthesis ability in interdisciplinary questions. Claude2 shows high adaptability and understanding in complex tasks involving multiple disciplines.
4 Related Work 4.1 Question Generation
The potential of LLMs in generating questions for educational purposes garners significant academic attention. Chen et al. (2019) have developed a reinforcement learning approach specifically for generating natural questions. Elkins et al. (2023) evaluate the educational utility of questions produced by LLMs, organizing them according to their levels
of difficulty. Tavares et al. (2023) investigate the methods LLMs use to create questions, focusing on the tracking of dialogue states. Kai et al. (2021) introduce a method involving double hints for the generation of questions about visuals. Uehara et al. (2022) emphasize the importance of sub-questions in improving the effectiveness of primary visual questions. Arora et al. (2022) examine various prompting techniques for LLMs and analyze the differences in model responses. Abdelghani et al. (2022) utilize the capabilities of GPT-3 to foster curiosity-driven questioning among children. Collectively, these studies underline the evolving role of LLMs in reshaping question generation instead of searching valuable questioning points.
4.2 Test-based Benchmark
There has been an increasing focus on evaluating the capability of LLMs in the context of standardized exams and academic benchmarks. For example, Zhang et al. (2023b) introduce GAOKAO Benchmark to evaluate the intuitive benchmark of Chinese college entrance examination questions; Huang et al. (2023) propose the first comprehensive Chinese evaluation package C-EVAL; Zhong et al. (2023) present a human-centric benchmark AGIEval designed for evaluating foundation models; Li et al. (2023) introduce CMMLU, a comprehensive Chinese benchmark covering multiple disciplines; Wang et al. (2023) introduce SciBench to systematically investigate the reasoning ability required to solve complex scientific problems;; Liu et al. (2023a) propose M3KE, a large-scale multilayer and multi-disciplinary knowledge assessment benchmark; Zeng (2023) propose a method to evaluate the multi-task accuracy of large Chinese language models across various domains; Zhang et al. (2023a) introduce FinEval, a benchmark designed for financial knowledge evaluation in LLMs; Wei et al. (2023) focus on the Chinese Elementary School Math Word Problems dataset to evaluate reasoning capability; Dao et al. (2023) explore ChatGPT’s potential to complete the Vietnam National High School Graduation Exam; Raina and Gales (2022) propose performance criteria to assess the generated multiple-choice questions; Chen et al. (2018) present LearningQ, an educational question generation dataset containing over 230K document-question pairs. However, these tests treat LLMs as students, assessing their abilities by how they answer questions instead of seeing them as teachers.
5 Conclusions and Future Work
In conclusion, our study presents a pioneering investigation into the questioning capability in education as a teacher of large language models (LLMs) through evaluating their generated educational questions, shifting the traditional role of LLMs from learners to educators. We have developed a comprehensive benchmark, named Dr.Academy, based on educational taxonomies that assesses LLMs’ abilities to generate questions across various domains with four evaluation metrics. Our findings indicate that models like GPT4, Claude2, and GPT3.5 demonstrate promising teaching potential. Looking ahead, the future directions of this research include refining the evaluation metrics for even more nuanced assessments of teaching effectiveness and expanding the range of subjects and domains covered.
Limitations
One limitation of our study is that it primarily focuses on the ability of large language models (LLMs) to generate questions, which is just one aspect of teaching. Actual teaching involves more complex interactions, including providing feedback, adapting to students’ needs, and fostering critical thinking, areas not fully captured by our current benchmark. Additionally, our approach relies heavily on the textual content, which may not comprehensively represent the nuances of human teaching methods that include non-verbal cues and personalized interactions. Therefore, while our findings offer valuable insights into the potential of LLMs as teaching aids, they should be viewed as a starting point for more in-depth research into the multifaceted nature of teaching and learning processes.
Acknowledgements
This work is supported by Science and Technology Commission of Shanghai Municipality Grant (No. 22511105902), Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103), the National Natural Science Foundation of China (No.62072323), Shanghai Science and Technology Innovation Action Plan (No. 22511104700), and the Zhejiang Lab Open Research Project (NO. K2022NB0AB04).
References Rania Abdelghani, Yen-Hsiang Wang, Xingdi Yuan,
Tong Wang, Hélène Sauzéon, and Pierre-Yves Oudeyer. 2022. Gpt-3-driven pedagogical agents for training children’s curious question-asking skills. arXiv preprint arXiv:2211.14228.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. The falcon series of open language models.
Lorin W Anderson and David R Krathwohl. 2001. A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives: complete edition. Addison Wesley Longman, Inc.
Simran Arora, Avanika Narayan, Mayee F Chen, Laurel J Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. 2022. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02441.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022. Constitutional ai: Harmlessness from ai feedback.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Guanliang Chen, Jie Yang, Claudia Hauff, and GeertJan Houben. 2018. Learningq: a large-scale dataset for educational question generation. In Proceedings of the international AAAI conference on web and social media, volume 12.
Jialin Chen, Zhuosheng Zhang, and Hai Zhao. 2023. Modeling hierarchical reasoning chains by linking discourse units and key phrases for reading comprehension. arXiv preprint arXiv:2306.12069.
Yu Chen, Lingfei Wu, and Mohammed J Zaki. 2019. Reinforcement learning based graph-to-sequence model for natural question generation. arXiv preprint arXiv:1908.04942.
Daixuan Cheng, Shaohan Huang, and Furu Wei. 2023. Adapting large language models via reading comprehension. arXiv preprint arXiv:2309.09530.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023).
John Joon Young Chung, Ece Kamar, and Saleema Amershi. 2023. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. arXiv preprint arXiv:2306.04140.
Xuan-Quy Dao, Ngoc-Bich Le, Xuan-Dung Phan, and Bac-Bien Ngo. 2023. Can chatgpt pass the vietnamese national high school graduation examination? arXiv preprint arXiv:2306.09170.
Sabina Elkins, Ekaterina Kochmar, Iulian Serban, and Jackie CK Cheung. 2023. How useful are educational questions generated by large language models? In International Conference on Artificial Intelligence in Education, pages 536–542. Springer.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322.
Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
Shen Kai, Lingfei Wu, Siliang Tang, Yueting Zhuang, Zhuoye Ding, Yun Xiao, Bo Long, et al. 2021. Learning to generate visual questions with noisy supervision. Advances in Neural Information Processing Systems, 34:11604–11617.
Akira Kawabata and Saku Sugawara. 2023. Evaluating the rationale understanding of critical reasoning in logical reading comprehension. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 116–143.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212.
Chuang Liu, Renren Jin, Yuqi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng, Shuting Zhang, Jianxiang Peng, Peiyi Zhang, Qingqing Lyu, et al. 2023a. M3ke: A massive multi-level multi-subject knowledge evaluation benchmark for chinese large language models. arXiv preprint arXiv:2305.10263.
Jiongnan Liu, Jiajie Jin, Zihan Wang, Jiehan Cheng, Zhicheng Dou, and Ji-Rong Wen. 2023b. Reta-llm: A retrieval-augmented large language model toolkit. arXiv preprint arXiv:2306.05212.
OpenAI. 2023. Gpt-4 technical report.
Vatsal Raina and Mark Gales. 2022. Multiple-choice question generation: Towards an automated assessment framework. arXiv preprint arXiv:2209.11830.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Jon Saad-Falcon, Joe Barrow, Alexa Siu, Ani Nenkova, Ryan A Rossi, and Franck Dernoncourt. 2023. Pdftriage: Question answering over long, structured documents. arXiv preprint arXiv:2309.08872.
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-llm: A trainable agent for roleplaying. arXiv preprint arXiv:2310.10158.
Diogo Tavares, David Semedo, Alexander Rudnicky, and Joao Magalhaes. 2023. Learning to ask questions for zero-shot dialogue state tracking. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2118–2122.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa,
Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and finetuned chat models.
Son Quoc Tran and Matt Kretchmar. 2023. Singlesentence reader: A novel approach for addressing answer position bias. arXiv preprint arXiv:2308.04566.
Kohei Uehara, Nan Duan, and Tatsuya Harada. 2022. Learning to ask informative sub-questions for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4681–4690.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2023. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10635.
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. 2023. Cmath: Can your language model pass chinese elementary school math test? arXiv preprint arXiv:2306.16636.
BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz,
Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, ZhengXin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, JanChristoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhat-
tacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2023. Bloom: A 176b-parameter open-access multilingual language model.
Hui Zeng. 2023. Measuring massive multitask chinese understanding. arXiv preprint arXiv:2304.12986.
Liwen Zhang, Weige Cai, Zhaowei Liu, Zhi Yang, Wei Dai, Yujie Liao, Qianru Qin, Yifei Li, Xingyu Liu, Zhiqiang Liu, et al. 2023a. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models. arXiv preprint arXiv:2308.09975.
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023b. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric
benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364.
Tongquan Zhou, Yao Zhang, Siyi Cao, Yulu Li, and Tao Wang. 2023a. Complementary advantages of chatgpts and human readers in reasoning: Evidence from english text reading comprehension. arXiv preprint arXiv:2311.10344.
Yucheng Zhou, Xiubo Geng, Tao Shen, Chongyang Tao, Guodong Long, Jian-Guang Lou, and Jianbing Shen. 2023b. Thread of thought unraveling chaotic contexts. arXiv preprint arXiv:2311.08734.

heading: Future Work