Title: What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
Benchmark: ChemLLMBench
Summary:  The paper develops a benchmark to  assess the capabilities of five LLMs on chemistry, using eight chemistry tasks requiring understanding, reasoning, and explanation abilities.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: chemistry ability
Phenomenon defined by authors: Yes
Phenomenon definition: "We identify three key chemistry-related capabilities including understanding, reasoning and explaining to explore in LLMs and establish a benchmark containing eight chemistry tasks."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Capabiliities: understanding, reasoning and explaining
Tasks: name prediction, property prediction, yield prediction, reaction prediction, retrosynthesis, text-based molecule design, molecule captioning, and reagents selection
Task definition: 8 chemistry tasks: name prediction, property prediction, yield prediction, reaction prediction, retrosynthesis, text-based molecule design, molecule captioning, and reagents selection
Task item definition: A task item consists of a chemistry-specific prompt (e.g. reactants) and the expected output (e.g. chemical reaction product).
Task source: Real task examples (e.g. GitHub issues)
Task source detail: chemistry datasets: BBBP, Tox21, PubChem, USPTO, and ChEBI
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 100
Dataset metadata available: Yes
Dataset metadata detail: Task type: name prediction, property prediction, yield prediction, reaction prediction, retrosynthesis, text-based molecule design, molecule captioning, and reagents selection
Sampling method: Random sample (creators defined a task space and sampled from it), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Validation
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: mean and standard dev
Metric subscores: Yes
Metric subscores detail: Reported separately for each chemistry task.
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors consider validity but consulting experts, comparing performance with established baselines and  evaluating different experimental settings (e.g. prompt strategy)
Taxonomy: Chemistry / 
New taxonomy: Specific Application / Physical Sciences