Title: Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models
Benchmark: Dr.Academy
Summary: This paper introduces Dr.Academy, a benchmark for evaluating the question generation capabilities of LLMs in educational contexts. It evaluates questions generated by LLMs across general, monodisciplinary, and interdisciplinary domains using a cognitive framework based on Anderson and Krathwohl’s taxonomy. The quality of LLM's output is evaluated by automatic metrics which correlate with human scores.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Question generation for education
Phenomenon defined by authors: Yes
Phenomenon definition:  "According to Anderson and Krathwohl’s educational taxonomy, we consider that high-quality questioning in the educational field must meet the following characteristics: i) achieve a higher level across the six domains including memory, understanding, application, analysis, evaluation, and creation; ii) be relevant to the given context; iii) comprehensively cover the content of the context, and iv) also reflect the important knowledge of this context."
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The LLMs are prompted to generate educational questions based on textual contexts, accross 3 domains (general, monodisciplinary and multidisciplinary) and mapped to the 6 levels from Anderson & Krathwohl’s taxonomy.
Task item definition: The context the LLM has to use to generate the educational questions.
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: The contexts are generated based on pre-existing question-answering datasets (general domain: SQuAD, monodisciplinary: MMLU)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 30,000 contexts (10,000 per domain)
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: Simple mean to aggregate automatic scores, Pearson and Spearman correlation between human and automatic ratings​, and Krippendorff’s Alpha inter-rater agreement for human ratings.
Metric subscores: Yes
Metric subscores detail: Yes, subscores are provided by domain (general, mono-humanities, mono-sciences, interdisciplinary) and by evaluation metric (consistency, relevance, coverage, and representativeness).
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: No, no link is provided
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors directly assess the validity of their benchmark through theoretical alignment with Anderson and Krathwohl’s taxonomy, expert evaluation of the evaluation metrics, and empirical correlation with human judgments.
Taxonomy: Education / 
New taxonomy: Specific Application / Profesisonal Domains