Title: Have LLMs Advanced Enough?
A Challenging Problem Solving Benchmark For Large Language Models
Benchmark: JEEBench
Summary: Curate a set of 515 challenging engineering, mathematics, physics and chemistry questions from the entrance exams to top Indian universities.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Logical and mathematical reasoning on top of deep in-domain knowledge of pre-engineering level Physics, Chemistry and Mathematics.
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Authors label this as a "reasoning" benchmark, but are explicit about this benchmark being specifically for the subset of problems that require "Logical and mathematical reasoning on top of deep in-domain knowledge of pre-engineering level Physics, Chemistry and Mathematics."
Task definition: Standard text based question and answer format.
Task item definition: In this problem, only one option will be correct.
Give a detailed solution and end the solution
with the final answer.
Problem: A uniform wooden stick of mass
1.6 kg and length l rests in an inclined manner on
a smooth, vertical wall of height h(< l) such that a
small portion of the stick extends beyond the wall.
The reaction force of the wall on the stick is perpendicular to the stick. The stick makes an angle of
30◦ with the wall and the bottom of the stick is on
a rough floor. The reaction of the wall on the stick
is equal in magnitude to the reaction of the floor on
the stick. The ratio h/l and the frictional force f at
the bottom of the stick are (g = 10 ms s2)
(A) h/l = √3/16 , f = 16√3/3 N
(B) h/l = 3/16 , f = 16√3/3 N
(C) h/l = 3√3/16 , f = 8√3/3 N
(D) h/l = 3√3/16 , f = 16√3/3 N
Task definition detail: Variety of single-correct multiple choice, multiple-correct multiple-choice, integer and number solutions. 
Task source: Human exam questions (e.g. GRE questions)
Task source detail: IIT Exams (Top Indian University Entrance Exams)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 515
Dataset metadata available: Yes
Dataset metadata detail: Domain (Physics, Chemistry, Maths) and sub-domain (Electromagnetism etc)
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), For multiple correct multiple choice questions, if there were 4 correct answers and the taker selects 3, they score 0.75. If they selected 4 correct and an additional 5th incorrect, they score 0. This is to mimic actual IEE exam scoring. 
Metric definition detail: counted as correct for numeric answers if within 0.01
Metric aggregation: Simple Mean
Metric statistics: simple mean, optionally the negative scoring as above
Metric subscores: Yes
Metric subscores detail: Domain (physics, chemsitry maths), Sub-domain (e.g electromagnetism)
Metric metascoring: The real IEE exam features negative scoring, to prevent guessing. They provide results with an without this being used.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning