Title: Can LLMs Solve Molecule Puzzles? A Multimodal Benchmark for Molecular Structure Elucidation
Benchmark: MolPuzzle
Summary: 217 examples of chemical structure elucidation. ie given a count of the number of each type of atom e.g C6 H22 O2, work out the molecular structure in SMILES notation. This is a long and challenging process that requires running and understanding many types of chemistry tests (H-NMR, C-NMR etc). They break down this task into sub-tasks with 20,000 examples of each sub-task. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Molecular structure elucidation (go from atom counts to geometric structure)
Phenomenon defined by authors: Yes
Phenomenon definition: They define molecular structure elucidation in 3 stages: 1) Molecular understanding. Given a mass spectrum (atom counts), list possible sub-molecular building blocks. 2) Spectrum understanding: Given building blocks and images of spectrums from various tests such as IR, MASS, H-NMR, refine list of which building blocks may be present. 3) Molecule construction. Put everything together into a SMILES string.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given molecular formula (atom counts), produce a SMILES string that fully defines the molecular structure.
Task item definition: There are 217 examples of molecular formula -> SMILES string, with attached spectrum images. There are 20,000 examples of sub-task examples.
Task definition detail: The task is somewhat conflated with a method.  
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 217
Dataset size detail: Provide 20,000 QA pairs for the sub-tasks.
Dataset metadata available: Yes
Dataset metadata detail: Split into stages (see phenomena description), and further into types of spectrum.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Response format detail: SMILES string format
Metric definition: Exact Match (accuracy, F1, precision, recall), Many chemistry specific metrics, such as molecule validity, Fingerprint Tanimoto Similarity etc.
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Accuracies provided for all sub-tasks. 
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Biology / 
New taxonomy: Specific Application / Physical Sciences
Procedural notes: Running their method is quite involved. The LLM is used as part of a wider method that involves checking suggested molecules are valid etc.