Title: Language Models as Science Tutors
Benchmark: TUTOREVAL
Summary: TUTOREVAL is a benchmark for evaluating language models as science tutors, consisting of questions about long chapters from STEM textbooks written by experts. The benchmark combines long contexts, free-form generation, and scientific knowledge across multiple disciplines. They use GPT-4 as an evaluator to mark the answers against key points provided by experts.
Contribution: The authors also create TUTORCHAT, a dataset of 80,000 long synthetic dialogues about textbooks that they use to fine-tune Llemma models with 7B and 34B parameters. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Real-life usability of LMs as scientific assistants/tutors
Phenomenon defined by authors: No
Phenomenon definition: It is a bit confusing if “problem-solving” or “scientific knowledge.” There is no definition of “scientific knowledge” but sub topics are environmental science, math, physics, computer science, life sciences
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a textbook chapter and a student question about it (which may be misleading), generate a tutoring response that explains concepts, corrects misunderstandings, and addresses the specific query while demonstrating scientific knowledge and teaching ability.
Task item definition: A single item consists of a textbook chapter excerpt (e.g., "One of the key numbers we can extract from mass and momentum is the invariant mass..."), a student question that may contain misconceptions (e.g., "How is the first equation in the chapter consistent with Einstein's E = mc² formula? It looks wrong."), expert-annotated key points that a good answer should cover (e.g., "The student is mistaken in saying the first equation is wrong..."), and the LM Tutor's generated response (e.g., "The first equation you mentioned, W²c⁴ = (∑ᵢEᵢ)² - (∑ᵢp̄ᵢ)²c², is indeed related to the famous E = mc² formula...").
Task source: Expert-crafted task examples (e.g. hand-written examples)
Task source detail: Questions were written by 17 annotators who are all STEM researchers with teaching experience. They wrote questions based on open-source textbook chapters that they would be qualified to teach in a class.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Simulates humans seeking to understand textbook materials, where a student asks questions about concepts from a chapter, which is a realistic educational setting.
Dataset size: 834
Dataset size detail: The dataset spans 5 domains with uneven distribution: Math (362), Computer Science (205), Physics (174), Environmental Science (53), and Life Sciences (40). It is skewed towards Math and Computer Science due to the expertise of the experts who write the questions
Dataset metadata available: Yes
Dataset metadata detail: Closed-book: whether the question can be understood without reading the chapter first.
Answer in chapter: whether the answer is contained in the chapter or requires substantial external knowledge.
Misleading: whether the question is phrased in a misleading way, such as including false assumptions or having an overly-confident tone.
Difficulty: how hard the question is relative to the chapter's intended audience (categorized as easy or hard).
Domain: the scientific field the question belongs to (Math, Computer Science, Physics, Environmental Science, Life Sciences)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Response format detail: Models generate free-form explanations in response to the questions, with outputs up to 800 tokens long. These are evaluated on correctness and presentation by GPT-4.
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: The GPT-4 evaluator is prompted to grade the output's presentation and correctness. Presentation measures clarity of explanation and general form. Correctness tracks factual mistakes and incorrect references to the chapter. The LM evaluator is instructed to follow the key points (these are bullets provided by humans experts characterising a good answer) as a guide for judging the answer. There is a human evaluation to validate GPT-4's reliability as an evaluator.
Metric aggregation: Simple Mean
Metric statistics: Simple mean across questions for each category. They also compute correlation metrics between human and GPT-4 evaluations: Pearson, Spearman, and Kendall-τ coefficients to validate GPT-4 as a reliable evaluator.
Metric subscores: Yes
Metric subscores detail: Scores are provided by domain (Math, Physics, Computer Science, Life Sciences, Environmental Science), by difficulty (Easy, Hard), for misleading questions, and for closed-book evaluation. They also separate presentation scores from correctness scores, though correctness is considered the more meaningful metric.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Maybe - it relies on the GPT-4 evaluator being able to assess the correctness of the answer relative to human key points. If the GPT-4-as-a-judge lacked nuanced scientific understanding, it may fail to evaluate another LLM's response against the key points (e.g., requires capabilities for classic entailment,contradiction task). Howver, they do show high correlation empirically between GPT-4-as-a-judge and human evaluators. 
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They conduct ablation studies on their evaluation method to show the importance of key points, presentation scores, and using GPT-4 rather than GPT-3.5; They also compare model performance on TUTOREVAL with performance on other math-oriented datasets like GSM8K and MATH. They validate GPT-4 as an evaluator by comparing its rankings to human evaluations through correlation metrics (Pearson, Spearman, and Kendall-τ), showing high agreement between GPT-4 and human rankings.
Taxonomy: User Interaction / 
Alternate taxonomy: General Science
New taxonomy: General Capability / User Interaction