MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs
Authors: Christoph Bartmann
Johannes Schimunek
Mykyta Ielanskyi
Philipp Seidl
Günter Klambauer
Sohvi Luukkonen
Venue: ICLR 2026 Poster (2026)
Keywords: chemical language model
chemical reasoning model
chemistry
large language model
molecular graph
molecular structure
Primary area: datasets and benchmarks
TL;DR: We propose MolecularIQ, a molecular structure reasoning benchmark focused exclusively on symbolically verifiable tasks.

Abstract: A molecule’s properties are fundamentally determined by its composition and structure encoded in its molecular graph. Thus, reasoning about molecular properties requires the ability to parse and understand the molecular graph. Large Language Models (LLMs) are increasingly applied to chemistry, tackling tasks such as molecular name conversion, captioning, text-guided generation, and property or reaction prediction. Most existing benchmarks emphasize general chemical knowledge, rely on literature or surrogate labels that risk leakage or bias, or reduce evaluation to multiple-choice questions. We introduce MolecularIQ, a molecular structure reasoning benchmark focused exclusively on symbolically verifiable tasks. MolecularIQ enables fine-grained evaluation of reasoning over molecular graphs and reveals capability patterns that localize model failures to specific tasks and molecular structures. This provides actionable insights into the strengths and limitations of current chemistry LLMs and guides the development of models that reason faithfully over molecular structure.

MOLECULARIQ: CHARACTERIZING CHEMICAL REASONING CAPABILITIES THROUGH SYMBOLIC VERIFICATION ON MOLECULAR GRAPHS
Christoph Bartmann 1.* Johannes Schimunek 1.*
Mykyta Iclanskyi*
Philipp Scidl
Günter Klambauer1.2 Sohvi Luukkonen'
'ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria
"Clinical Research Institute for Medical Artificial Intelligence, Johannes Kepler University Linz, Austria *Equal contribution.
ABSTRACT
A molecule's properties are fundamentally determined by its composition and structure encoded in its molecular graph. Thus, reasoning about molecular prop- erties requires the ability to parse and understand the molecular graph. Large Language Models (LLMs) are increasingly applied to chemistry, tackling tasks such as molecular name conversion, captioning, text-guided generation, and prop- erty or reaction prediction. Most existing benchmarks emphasize general chemical knowledge, rely on literature or surrogate labels that risk leakage or bias, or reduce evaluation multiple-choice questions. We introduce MOLECULARIQ, molec- ular structure reasoning benchmark focused exclusively on symbolically verifiable tasks. MOLECULARIQ enables fine-grained evaluation of reasoning over molec- ular graphs and reveals capability patterns that localize model failures to specific tasks and molecular structures. This provides actionable insights into the strengths and limitations of current chemistry LLMs and guides the development of models that reason faithfully over molecular structure.
INTRODUCTION
From task-specific chemoinformatics models to unified I.LM-based chemistry assistants. Re- cent advances in autonomous robotic laboratories and agentic large language model (LLM)-based systems bear the promise of end-to-end automation of the scientific workflow, from hypothesis gen- cration to experiment execution (MacLeod et al., 2020; Tom et al., 2024; Lu et al., 2024a; Yamada al., 2025). In these emerging pipelines, LLMs not merely conversational interfaces: they act as decision- decision-making engines that propose experiments, design molecules, and interpret results. In parallel, general-purposs and chemistry-sppecialzze LLMs being adopted as accessible tools for molecular reasoning and property prediction. A key appeal of the language model approach to chemical problems is that classical chemoinformatics models are typically task-specific (e.g.. prop- erty prediction, reaction prediction, or de novo design), whereas a single LLM could, in principle, address all of these tasks within one unified interface. Furthermore, these models have the poten- tial to lower barriers for non-experts, including wet-lab chemists, to perform tasks that traditionally required dedicated chemoinformatics expertise. Consequently, the interest in both evaluating the chemistry capabilities of general LLMs (Guo et al., 2024; Mirza et al., 2025; Huang et al., 2024; Lu -oeaiiispeciaalied chemistry LLMs (Liu et al.,
et al., 2024b; Runcie et al., 2025) and developing 2023b;a; Zhang al., 2024; Yu al., 2024; Liu al., 2024; Fang al., 2024; Li et al., 2024b; Zeng et al., 2024; Narayanan et al., 2025; Zhao et al., 2025a;e; Ye et al., 2025; Wang et al., 2025a; Tang al., 2025) has surged.
Leaderboard: httts::/huggigggaeee ct Code: https: tttp://gtgttuttto// com/ml hotpa//omoo/mliiooeoe/eollllllll lewiilrillllrgroooooooo
http://wggggagaaacooooaaaaaaaeee http:///ww......ooo.ccoo/ooaaaaa Leadeeeerrrrrr_eeeeeeeeeeee Leaderboard

---

Symbolically verifiblle molecular featured
Graph Topology
comnistry--ypeed Topology
Composition
Perception
Functional Groups
Synthesls
frmils hilepen...
HSA 1E0, ordetie
M
B0C Tognants
Reasaning tasks
Complexity axds
Molecular
many branch poira??
HYp-DH
Load
Leod'3
Indesing
1037
Cansnkel
Kokualzed
HBA stos?
lupior ingie
Gencration
Desient suriitton
devorrtte An ec.4
Astiee CONCO
ID) count?
Figure 1: Overview of MolecularlQ benchmark for measuring chemical structure reasoning capa- bilities LLMs.
Structural understanding is the prerequisite for molecular reasoning, not just one capabil- ity among many. A fundamental principle in chemistry and chemoinformatics the structure- property relationship, which means that the molecular structure of an organic molecule determines its properties. Thus, a model that cannot reliably identify functional groups, ring systems, or atomic connectivity cannot be expected to reason about how these features influence reactivity, toxicity, or biological activity. Structural understanding is, consequently, a key prerequisite upon which all meaningful molecular modification depends.
Current benchmark design obscures whether LLMs truly reason over molecular structure. However, current chemistry benchmarks ill-suited to testing this capability in LLMs. Bench- marks emphasizing general chemistry knowledge mainly measure factual recall, while property and reaction benchmarks rely on public datasets, which likely appear LLM training corpora, mak- ing it impossible to disentangle structural reasoning from pattern recognition and memorization of molecule- property pairs (Carlini et al., 2021). Furthermore, when exact ground truths are un- available, some benchmarks rely on surrogate verifiers (predictive models or heuristics), which can introduce bias and be exploited, so higher scores do not necessarily reflect genuine reasoning Renz et al. (2019).
Symbolically verifiable tasks serve as a diagnostic for molecular reasoning competence. Sym- bolically verifiable graph-based tasks provide a testbed for evaluating structural reasoning. Since LLMs process molecules as ID token sequences (e.g., SMILES strings), these tasks implicitly assess the model's 's ability to infer 2D graph topology from linear representations. Because every answer can be computed exactly from the molecular graph, symbolically generated ground truths guarantee label correctness and eliminate surrogate verifier bias; moreover, while input molecules may appear in pretraining corpora, the specific task formulations and multi-property queries are algorithmically generated, reducing the likelihood of direct answer recall. Although the tasks we consider can be solved trivially by cheminformatics software, their value as evaluation tools lies precisely in this property: they establish floor below which no model should fall if it has genuinely internalized molecular structure. In turn, model that appears strong on property prediction benchmarks yet fails at basic substructure identification is likely exploiting specifc correlations rather than reasoning over structure.
MOLECULAAIQQ: a fully symbolically verifiable benchmark for assessing reasoning over molecular structures. Building on this perspective, we introduce MOLECULAARIQ, a chemical structure reasoning benchmark composed exclusively of symbolically verifiable tasks defined on molecular graphs. MOLECULARIO covers three task types feeture counting and indexing, and con- strained generation - and spans three complexity dimensions multitask load, molecule complexity, and molecule representation. This multi-dimensional design enables controlled, fine-graing diag- nosis of where and why models fail, complementing existing chemistry benchmarks with a targeted evaluation of internalized structural competence. Figure gives an overview of the MOLECULARIQ benchmark.
2

---

Contributions:
We introduce MOLECULARIO, the first fully symbolically verifiable benchmark for molec- ular understanding, where every answer can be programmatically validated against the molecular graph. It spans three task types and three complexity axes, enabling granular diagnosis of structural reasoning capabilities.
We conduct a comprehensive evaluation of 38 generalist, chemistry-specializdd and reasoning-optimizzed LLMs, revealing that structural understanding remains a critical bot- tleneck: models that cannot reliably parse molecular graphs lack the foundation required for trustworthy molecular reasoning.
We release open infrastructure for advancing chemical AI, including a public leaderboard, task generation process, symbolic verification tools, training datasets to both evaluate mod- els but also support reinforcement learning with verifiable rewards.
2 RELATED WORK
Research on Large Language Models (LLMs) for chemistry falls into two streams: (1) benchmark- ing efforts assessing generalist and specialist capabilities, and (2) the development of specialized chemistry LLMs via fine-tuning, reinforcement learning (RL), or multimodal architectures.
Generalist benchmarks and their limits. Early evaluations prioritized broad coverage using exam- style, multiple-choice questions spanning many fields in chemistry(Sun et al., 2024; Wang et al., 2024a; Huang et al., 2024; Zhang et al., 2024; Wang et al., 2024b; Mirza et al., 2025). While effec- tive for assessing factual recall, such formats risk rewarding pattern matching or memorization over genuine reasoning, and do not test the explicit structural comprehension that underpins chemical properties. To address this, FromtierScience (Wang et al., 2025b) recently introduced expert-level open-ended research tasks. However, its reliance on model-based grading (using advanced I. L.MMs with rubrics) introduces significant evaluation noise. Furthermore, manually curated expert ques- tions are challenging to scale for the systematic and granular diagnosis of model failure modes.
Benchmarking structural reasoning. Recent work has shifted toward directly evaluating molec- ular comprehension. Guo et al. (2024) introduced MolPuzzle to probe structure elucidation from spectra; however, its reliance on textbook-style molecules risks data contamination, while its multi- modal nature separates it from pure graph reasoning. Similarly, ChemiQ (Runcie et al., 2025) offers symbolic mostly tasks, but many serve primarily as SMILES parsing checks and have been saturated by base models. Other work has examined how molecular string representation format affects LLM performance on chemical tasks (Baker et al., 2025). A newer wave of benchmarks targets specific subdomains but suffers from data leakage or narrow scope. FGBench (Liu et al., 2025) evalu- ates functional-grow grrup reasoning but inherits labels directly from MoleculeNet (Wu et al., 2018), a common pretraining corpus. MolEnr2Fix (Wu et al., 2025) probes error detection and correction in molecular descriptions but is limited to a single task modality. ChemCoTBench (Li et al., 2025) as- sesses editing and reaction prediction using USPTO data (Lowe, 2017), again risking contamination, and relies on non- -on-deterministic LLM judges.
Finally, generative benchmarks like TOMG-Bench (Li et al., 2024a) and MEGA (Fernandez al., 2025) focus on constraint satisfaction and property optimization. While valuable for generation, they do not measure whether a model genuinely understands the underlying structure-property rela- tionships. Crucially, none of these frameworks systematically vary task type, molecular complexity, or representation format localize precisely where reasoning breaks down.
The model-evaluation gap. The landscape of chemistry models is diversifying rapidly, spanning instruction- models (Zhang et al., 2024; Yu et al., 2024) and RL-enhanced reasoners (Zhao et al., 2025b; Narayanan et al., 2025). Despite this progress, evaluation remains fragmented. Mod- els are typically assessed on disparate, self-curated datasets for captioning (ChEBI-20), reaction prediction (USPTO), or property prediction (MoleculeNet) (Edwards et al., 2021; Lowe, 2017; Wu et al., 2018). These ad hoc protocols are often tailored to specific models and utilize approximate oracles, undermining meaningful cross- model comparison. Without a unified benchmark that offers symbolic ground truths and controlled complexity axes, disentangling genuine structural reasoning from pattern matching remains open challenge.
3

---

Taak (Innar)
180
8%
9%
Feature Groups
Counting (1740)
Craph Topology
Generation (1573)
50
15%
17%
Topology
Cheriisty-lyped
Composition
Load 1 (2633)
Sharee (50)
60
22%
16%
16%
Chemica Perception
Functional Groups
n=1111
Load 3 (900)
Synthesis
Bertz (outer)
40
32%
250-10 000 (1180)
1000 (11.80)
20
20%
11%
Counting
Figure 2: MOLECULARIQ composition. (left) Number of questions by task type, multitask load, and Bertz complexity. (right) The percentage of feature types present for counting, indexing, and generation.
MOLECULARIQ
MOLECULARIQ is fully symbolically verifiable benchmark for assessing reasoning molec- ular structures. covers three task types feature counting, index-based attribution, constrained generation (Section 3.1) covering various symbolically verifiable molecular features categories (Section 3.2). Additionally, MOLECULARIIQ incorporates three orthogonal complexity axes (Section 3.3): SMILES representation format, molecular complexity, and multitask load. This creates a multi-dimensiona capability profile that allows localization of failure modes along be- spoke axes and the question types. released two forms (Section 3.4): a static variant, denoted as MOLECULARIEE containing total of 5111 questions (Figure 2), and a dynamic variant MOLEC- ULARIQD, designed to address future concerns such as overfitting and saturation.
six
3. REASONING TASKS
We define three types of tasks:
Feature counting establishes baseline comprehension of molecular graphs. High perfor- mance may arise from shortcuts that bypass true graph reasoning.
Index-based attribution requires models to ground answers in specific atoms/bonds/sub. structures, closing off spurious solution paths.
Constrained generation frames molecular design the generation of molecules that sat- isfy given specifications, a capability essential for practical applications.
For each feature-counting question, except for hydrogen counting and molecular formula, where indexing is not possible, MOLECULARIQ also provides an equivalent masex question on the same molecule. This allows assess whether model's 's correct count stems from pattern recog- nition or genuine reasoning over the molecular graph.
3.2 SYMBOLICALLY VERIFIABLE MOLECULAR FEATURES
Our chemical-reasoning tasks target molecular features across six categories: graph topology, chemistry-typed graph topology, composition, chemical perception, functional groups, and synthe- sis/fragmentation. Below we briefly describe each category: detailed definitions appear in Sec- tion A. For each feature, we have an RDKit-based symbolic solver (Landrum et al., 2006), en- tasks. For all features, except hydrogen counting
abling both counting and constrained-generee and molecular formula, we can also determine the indices of the atoms participating in the feature.
Graph topology captures purely structural properties of the molecular graph independent of chem- ical typing. Tasks probe recognition rings, fused rings, bridgehead atoms, ring size extremes, as well as linear termini and branch points. These test whether model can parse and reason about the bare connectivity of the molecular skeleton.
Chemistry-typer graph topology extends graph topology with chemical annotations. Features to probe include identifying aromatic vs. aliphatic rings, heterocycles, saturation, sp* carbon hy- bridization, longest carbon chains, and stereochemical descriptors (R/S centers, E/Z. double bonds,

---

specified unspecified stereochemistry). This family checks whether models integrate chemical typing and stereochemistry into their structural reasoning.
Composition focuses counting and enumerating atomic constituents of molccules. Tasks in- volve identifying numbers of carbon, hetero, halogen, heavy, or hydrogen atoms, and deriving the molecular formula. These probe basic but essential symbolic operations directly tied to chemical composition.
Chemical perception covers substructural and electronic properties commonly used in chemin- formatics. Tasks include identifying hydrogen bond donors (HBD) and acceptors (HBA), rotatable bonds, and assigning oxidation states. This family assesses whether models can recognize functional features that underpin molecular properties.
Functional groups targets detection of chemically meaningful substructures such as alcohols, amines, carboxylic acids, and other standard functional groups. This probes chemical reasoning the level of reactivity patterns and transformations.
Synthesis and fragmentation encompasses tasks relevant to retrosynthesis and scaffold analysis. It includes BRICS fragmentation. template-buse baseed reaction prediction, and Murcko scaffold extraction. These tasks evaluate whether models can reason about molecules in terms of synthetic building blocks and core scaffolds, bridging structural analysis with applications drug design.
3.3 COMPLEXITY AXES
SMILES representations. We use SMILES (Weininger, 1988), the most widely adopted molecular string representation, to represent the molecular graphs. Their non-uniqueness and the availability of aromatic vs. kekulized (dcaromatized) forms provide an additional, controllable complexity axis. If an LLM truly reasons over structure, canonicalization should be irrelevant; conversely, drop from canonical to non-canonical SMILES would indicate reliance on memorized canonical pat- terns. Yu al. (2024) reported that training the chemistry LLM LlaSMol with canonical SMILES improved performance, and Runcie et al. (2025) observed significant performance drops on several ChemIQ tasks when using randomized (non-canonical) SMILES. Likewise, perfect graph under- standing would render kekulization immaterial though kekulized inputs may be harder because inferring aromaticity adds a reasoning step. To probe both factors, we apply two independent ran- dom decisions per molecule, each with 50% probability. Additionally, for a subset of models, we also tested the effect of ring index relabeling on counting and indexing questions involving cyclic molecules.
Molecular complexity. Understanding how models perform across different regimes of molecular complexity is essential for identifying the types of molecules which they succeed and for diag- nosing failure modes. As molecules grow larger and more branched, with more rings and functional groups, models must keep track of longer-range structural dependencies and growing number of relevant fcatures, which can challenge both representation and reasoning. In our benchmark, we adopt coarse-grained categorization based the Bertz complexity index (Bertz, 1981). ), defining three bins: 0-250, 250-1000, and 1000+. Figure A1 illustrates example molecules per complexity bin, and Figure A2 shows that MOLECULARIQ spans broader complexity range than ChemiQ and ChemCOTBench. The dynamic dataset version allows adjusting these bins for customized evalua- tion, for example probe how model performance changes molecular complexity increases.
Multitask load dimension. Chemical reasoning often involves solving several sub-tasks simulta- neously, motivating an explicit evaluation along the multitask axis. Many realistic tasks, such as reaction prediction, mechanism recognition, or constraint-dri -rrivinnntdriennnnn molecule design, decompose into elementary operations like identifying functional groups or determining implicit hydrogens. Prior work has reported low performance on format conversion tasks, such as -IULEC (Runcie et al., 2025) or IUPAC- (Narayanan et al., 2025), which require accurate coordination of multiple atomic and structural counts. Poor results may stem either from inherent task diffi- culty or from a broader inability of models to manage multiple chemical sub-tasks. To disentangle these factors, MOLECULARIQ systematically varies the multitask load. We evaluate four regimes: a single-task setting (one count, index, or constraint) and multitask settings with 2, 3, or 5 simulta- neous requests.
5

---

3.4 BENCHMARK CREATION
This section outlines the procedure for constructing MOLECULARIQ. The benchmark consists of sampled molecules with exact, precomputed ground-truth values for the task each molecule was assigned to. We anticipate that future models will be fine-tuned on this task battery. To prevent data leakage, we also provide a pool of training molecules.
Molecule pool. We retrieved all single fragment molecules that contained at least one carbon atom from PubChem (Kim et al., 2023). To avoid contamination of the training pool and redundant evaluations, we removed all molecules in etherO, ChemlQ. LlaSMol, and ChemDFM evaluation sets. Training and test pools. To promote best practices for model development, we partition the initial molecule pool into three sets: training pool, an easy test pool, and hard test pool. The easy pool contains molecules from clusters present in training; the hard pool is sampled from held- out clusters. We cluster molecules using MinHashL.SH (Broder, 1997; Indyk & Motwani, 1998; Leskovec et al., 2020) over ECFP fingerprints (2, 512) (Rogers & Hahn, 2010) with a 0.7 Tanimoto similarity threshold. The resulting splits comprise 1.3M training molecules and 1.0M molecules each in the easy and hard test sets.
MOLECULARIQ our primary benchmark. It comprises 849 unique molecules drawn from the hard test pool and total of 5,111 questions distributed across the complexity axes, For all molecules in the hard test pool, we compute the ground-truth values for the molecular features and Bertz complexities using RDKit. Then, each question is formed by sampling a (task, question template, feature(s), molecule) tuple. The sampled molecule serves as part of the input for counting and indexing tasks, and as a feasibility reference for constrained generation to ensure the constraints satisfiable. Benchmark samples are produced via a weighted scheme that balances reasoning tier, multitask load, molecular complexity, and ground-trute truth value distributions (see Section A.2).
We provide a framework to dynamically adjust and scale the dataset to com- munity needs. It supports sampling new molecules from the defined pools and efficiently comput- ing ground truths for any RDKittreadable SMILES. Potential uses include: (i) creating validation sets from the easy test pool for model training: (ii) refreshing or expanding MOLECULARII when models saturate or overfit; and (iii) constructing focused evaluation sets for specific subfields (c.g., natural products).
MOLECULARIQD.
4 EVALUATION AND LEADERBOARD
Language Model Evaluation Harness integration. We integrate MOLECULARIQ into the in-evaluat wwwuuaaionhaaaneee framework (Gao et al., 2024), enabling standardized evaluation across both local and API-served models. Each of the evaluated models (Section B.1) uses tailored configurations with model-specific sampling parameters, preprocessing functions, and extraction methods ensure fair comparison while optimizing for each model's strengths (Section B.2).
Symbolic extraction. Fair benchmark comparison requires reliable answer extraction that focuses on semantic content rather than surface formatting. Recent work has highlighted how weak ex- traction procedures can artificially inflate or deflate model performance, with models potentially learning format compliance rather than genuine capabilities (Chandak et al., 2025; Shao al., 2025; Gudibande et al., 2024). We address this through: (a) hierarchical extraction procedures that grace- fully handle diverse output formats, and (b) key-specific matching with canonical normalization that decouples format compliance from chemical accuracy while preserving property-leve granularity for fine-grained performance analysis.
Scoring. Each task is scored with a binary symbolic verifier; per-instance accuracy is the mean over three independent rollouts to account for stochastic generation (Wang al., 2022; Brown al., 2020). Reported scores therefore lie in [0, 1] (higher is better). For subset of models, we also performed eight roll-outs, and for those models, the differences were negligible compared to three rollouts (Table C11).
Metrics. We report (i) overall accuracy averaged across all benchmark instances and (ii) granular breakdowns by reasoning task, multitask load, molecular complexity, molecular-feature category, and SMILES format, enabling systematic assessment of model strengths and weaknesses (Srivastava al., 2023). For some analyses, we use binary success metric: an instance counts as successful

---

Table I: Results the MOLECULARIQ benchmark for perp-frforming models. We compare the top 3 ineemistry-spccialieee LLMs and top 10 generalist LLMs, reporting overall accuracy and accuracy across task families (in %). Models sorted by overall accuracy. Error bars indicate standard errors across instances. Model size refers to parameter count; R and indicate reasoning and chemistry models.
Task family
Size
R
Overall
Counting
Indexing
Generation
Chemistry LLMs
TxGemma-27B
27B
5.0 0.2
7.0 + 0.5
1.8 0.3
6.2 + 0.5
EtherO
24B
V
V
6.5 + 0.3
3.2 + 0.3
0.1 + 0.0
17.5 + 0.8
ChemDFM-R-14B
14B
V
V
8.7 + 0.4
12.9 + 0.8
2.8 + 0.4
10.5 + 0.8
Generalist LLMs
GLM-4.6
355B (A32B)
/
16.2 + 0.4
15.9
11.3 + 0.6
22.0 + 0.9
Qwen-3 30B
30B (A3B)
V
22.7 0.5
2..0
16.5 + 0.8
29.4 + 1.0
SEED-OSS
36B
V
0.5
2.99
25.6 3.9
20.0 0.9
GPT-OSS 120B (Med)
120B (A5B)
0.5
0.
22.3 0.8
28.0 1.0
GPT-OSS 20B (Med)
20B (A4B)
V
0.5
0.9
22.2 0.8
32.1 1 1.0
Qwen-3 Next 80B
80B (A3B)
V
32.3
30.7 1.0
26,9 0.9
40.0 1 1.1
GPT-OSS 20B (High)
(A4B)
/
35.8 0.6
39.1
32.8 + 1.0
35.2 + 1.0
DeepSeek-R1-on R1-0528
67 (A37B)
V
37.1 0.6
3
30.2 0.9
45.5 + 1.0
Qwen-3 235B
235B (A22B)
V
39.2 0.6
1.0
34.5 1 1.0
46.7 + 1.1
GPT-OSS 120B (High)
120B (ASB)
47.5 0.6
46.8 1.1
42.5 1 1.1
53.7 1 1.1
if at least two of the three rollouts are correct, and the success rate is the percentage of success- ful instances. Additionally, to evaluate whether failures are due to semantic errors or malformatted outputs, measure type-validity rate, the percentage of outputs satisfying expected syntactic con- straints, for each task category (integers for counting, integer lists for indexing, valid SMILES for generation).
Living benchmark. The full leaderboard will be hosted online: https: hettps://buggineee htw.g/ggggggcc..eeeeeoooaaa CC/ spa ces //1 htlmmmljjkke mmllskmmmkkkamaa /oommolmll eeuuelllle ecularIQ eaderboare Leaderboard (Fourrier et al., 2024), we will accept submissions of new chemistry L.MSs (open weights free API access). Authors provide (i) an 1m-eva
Following the model of the Open LLM
configuration file, (ii) optional model-specific preprocessing and extraction functions, and (iii) brief description of training methodology and data sources.
5 RESULTS AND DISCUSSION
We evaluated 38 open-weight LLMMs on MOLECULARIQ using the protocol in Section 4: 27 general- purpose and 11 chemistry-speciilizee models. See Table B1 for model details. In this section, we focus on the analysis of the top-10 models and key findings, focusing on the overall results and counting tasks. Table reports the overall score and per-reasoning-task scores for the top models, and Figure 3 illustrates the performance profile of these models on the counting tasks. In Section C, we provide results for all evaluated models for all task types and further break down performance by multitask load, molecular complexity, colecular-featur category, and SMILES format, and extended failure mode analysis.
Large state-of-the-art LLMs with high reasoning budgets lead molecular-structure reasoning. GPT-OSS 1 120B (High) is the best-performin model overall and across most subcategories in our multi-dimensional complexity profiling. Nine out of ten top models belong to the same family of modern mixture-of-expprt (MoE) reasoning models, with SEED-OSS being the non-MoE model. Overall, this aligns with trends in general language modeling and reasoning benchmarks, where these models also oftte-of-the-art (e.g., LiveBench (White et al., 2025)). However, LiveBench, Qwen-3, DeepSeek-RR RI, and GLM-4.6 models significantly outperform GPT-OSS, whereas on our molecular-struct tasks, observe the inverse, with GL.-4.66 in particular performing notice-

---

a) Bertz
b) Functional Groups
c) Mu thtas Load
d) Feature Types
50
GPT-055 1200 (High)
64
6R
Owen 2358
35
DeepSeel-81
Success raño (56)
GPT-055 20n (Hihh)
19
29
28
76
21
Owen 3 Next 808
25
25
29
28
19
42
22
22
50
GPT-055 208 (Med)
42
28
26
13
20
27
24
24
23
20
23
18
52
45
36
10
GPT-055 1208 (Med)
24
21
SEED 055
Qwen:3 308
GLM-4.5-
24
12
16
14
34
24
<250
250-1k
RON
Con
6
SYN
e) Multitask Load Change
1) Task Change
9) SMILES Representation Change
47%
524
475
Ruprosentatooo (N)
NON
ON% OX
524
Acomatic
Canoric
13%
20
Acorati
42%
31%
37H
07%
12%
35%
20
fandom
Count
10
Suconss Pal
Fut -Succss
Sucoba -fod
Tut -hal
Canonical Aromatic Accuracy (%4
Figure 3: Performance profile of top 10 models for counting tasks. (a-d) Heatmaps of average success ratio per model (%) by (a) Bertz complexity of the molecule, (b) functional -roup families (ROH: hydroxyls, т-sys: r-systems, S: sulfurs, Hal: halides, Ar: aromatics, ROR: ethers, C=O/S: (hhio)carbonyls, Alk: alkyls, NR: amines, C=N/N=O:
nitriles/nitros), (c) multitask load, and (d) molecular feature groups (C: composition, CP: chemical perception, CT: chemistry-tyeet topology, FG: functional groups. SYN: synthesis). (e-f) Success/failure transitions averaged across top models for matched questions (e) additional sub-tasks added to the same prompt, and (f) when mov- ing from counting to indexing variants of the same underlying property, probing whether correct counts are grounded in explicit substructure identification. (g) Sensitivity to input representation, comparing accuracy between different SMILES representations.
ably worse than the other models. We also find that a higher reasoning budget generally improves performance (Figures C7 and C8); for GPT-OSS, the budget-induced performance gaps within given model are larger than the gaps between different model sizes.
Constrained generation exposes model limitations under compositional demands. Most top- 10 models score highest on constrained generation (Table 1), yet this strength does not generalize. Accuracy drops sharply as constraint-set prevalence (i.e., how frequently the requested features co-occur in PubChem molecules) decreases (Figure C1) and once three or more constraints are requested (Figure C12). These patterns indicate that models lack genuine compositional reasoning, as success on common constraints does not transfer to rarer combinations.
Multitask load makes questions harder overall, but can aid solving individual structural sub- tasks. Figures 3(a) and (c) show that counting task performance declines with both Bertz complexity and multitask load, but the effect of multitask load is substantially larger. Similar patterns persist across all models and indexing (Tables C2-C3). However, in the multitask case with subtasks, an answer counts as correct only if all n subtasks are answered correctly. Let Psingle denote the sin- gle task success rate. If multitask load had no additional effect, the expected success rate at load n would be Paingee- For all top models, the observed n-task success rates exceed this baseline, in- dicating that multitask prompting actually helps models solve the underlying structural reasoning subtasks, even though the full questions become harder. This is consistent with Figure 3(e). which shows that the solvability of most properties is largely stable across task loads, and that, for slightly more properties, adding subtasks increases their solvability rather than decreasing it.
Top models transfer from counting to indexing with only modest performance luss. For the top models, the performance gap between feature counting and indexing is relatively small, with
8

---

decrease of ~5-30% depending on the model (Table 1), suggesting that most correct counts arise from correctly locating the underlying substructures, ie.. genuine pattern recognition which show that the solvability of most molecular features stays the same for the counting and indexing questions (~80%), and that, for slightly more properties, changing the task from counting to indexing increases their solvability rather than decreasing it.
based reasoning. instead of memorized feature statisties. This is consistent with Figures 3(e) and C11,
SMILES perturbations reveal reliance on canonicalization and token patterns. We test whether models rely on canonical SMILES conventions by evaluating counting and indexing under random- ized, kekulized, and ring- -ngueerateteed representations. Performance drops consistently across all perturbations and all top-10 10 models (Figures 3g), C5, C6). These sensitivity patterns suggest a strong reliance on canonical token patterns, ring- alogcclouure conventions, and aromatic notations.
analyses expose uneven ability profiles. Figure 3(d) reveals uneven competence to answer questions about different feature types. Composition- related features are easiest (70-90% for top models), while syathesis/fragmentatio remains difficult (Tables C4- C7). Figure 3(b) shows disparities in handling molecules with different functional groups: aromatic and alkyl patterns are handled well, but organosulfur and oxidized nitrogen motifs (C=N/N=O) yield low success rates (Figure C10 (b)).
Feature- and Lunctional-group--ee
Failures reflect missing reasoning, not extraction artifacts or invalid outputs. One concern for symbolically evaluated benchmarks is that scores might be dominated by extraction errors rather than genuine reasoning failures. Two analyses argue against this. First, Table C13 and Figure C13 show that type validity is high, often 80-90% for top models, while accuracy is substantially lower, indicating that most failures are semantically wrong answers, not malformed outputs. Second, Fig- ure C8 reveals that incorrect answers are on average significantly longer than correct ones across all top-10 10 models, suggesting verbose tfain-offthoouhtt correlates with confusion. To understand why errors occur, we analyze 300 question - trace pairs on which no model succeeds, rating chains- of of-thought along chemistry and reasoning dimensions (Section C.2.6, Figure C14 & C15). The chemistry failure-mode heatmap shows that failing traces handle basic SMILES parsing and ring topology but break down on funetional-grouy recognition, property attribution, and stereochemistry. The reasoning heatmap indicates adequate task fidelity and method selection but weak constraint tracking, quantitative precision, and error awareness. We provide example failures from different models along with their judged scores in Appendix C.2.7. These illustrate common mistakes such as relying on SMILES lowercase conventions for aromaticity or using canonical @/2@ characters instead of applying CIP rules for stereochemistry assignment. Together with more than 1000 uni- versally failed questions (Figure C9), these findings confirm that MOLECULARII IQ probes genuine weaknesses in chemical reasoning, not output formatting.
Naïve chemistry fine-tuning systematically hurts. Recent open-weight general-purposs LLMs surpass chemistry-specialized models on MOLECULARIQ, reflecting the rapid gains in generic rea- soning. More strikingly, chemistry fine-tuning on task sets that differ from ours, apart from limited overlap (e.g..
for several models; constrained generation for etherO), does not translate into better structure reasoning and systematically degrades performance relative to the base models (Table C12). Only ChemDFM-R shows minor improvement over base model. This suggests that narrow, task-specific instruction tuning can miscalibrate reasoning or overfit to format artifacts, yielding weaker transfer to our symbolically evaluated tasks. On average, chemistry- models have an 18 percentage point lower type validity rate than their base models (median 11 points), indicating that the chemistry tuning has degraded their general language modeling capabil- ities and their ability to follow formatting instructions by overfitting them to narrow, task-specific output patterns (Table C13).
LIMITATIONS AND OUTLOOK
Narrow feature suite. By design, MOLECULARIQ focuses exclusively on symbolically verifiable tasks. This ensures exact ground truth but excludes important classes of problems, such as prop- erty prediction (e.g., solubility, bioactivity) or reaction outcomes, where exact symbolic solvers are unavailable. As result, our benchmark cannot yet cover the full spectrum of tasks relevant molec- ular design and drug discovery. Future work may combine symbolic tasks with trusted numerical approximations (e.g., QM-based solvers) to broaden task coverage while maintaining verifiability.

---

Restriction to 2D structure. Our benchmark currently evaluates models exclusively based on 2D molecular representations (SMILES). This choice reflects the present state of chemistry- LLMs: SMILES, compact string encodings of molecular graphs, remain the primary and most intuitive input format for text-based models, whereas 3D representations would require multimodal architectures that only beginning emerge. SMILES, therefore, remains the most practical and widely supported interface for assessing molecular reasoning capabilities in today's 's models. Nev- ertheless, we acknowledge that many chemically relevant phenomena depend on 3D information, including stereochemistry, conformational preferences, and spatial constraints.
Focus on single molecules represented with single modality. All tasks in MOLECULARIQ operate on individual molecules. However, many critical applications in chemistry involve reasoning across sets of molecules, such as reaction prediction, retrosynthesis planning, relative ranking of candidate designs. Extending the benchmark to multi-molecule tasks would enable evaluation of these higher-order reasoning capabilitiis. Our benchmark reflects current practice, where chemistry models rely mainly on SMILES and natural language. Future models will likely integrate multiple modalities to capture a broader range chemical information.
Outlook. Iterative evaluation of state-of-the-art chemical reasoning capabilities is cornerstone for sustainable model development. We envision MOLECULARIQ playing a central role in future model evaluation, evolving into a living benchmark. Moreover, anticipate that MOLECULARIQD will allow the community to adapt evaluation sets to emerging needs, while its symbolic solver can serve efficient reward model for reinforcement learning.
ACKNOWLEDGEMENT
The ELLIS Unit Linz, the LIT AI Lab, and the Institute for Machine Learning supported by the Federal State of Upper Austria. We thank the projects FWF AIRI FG 9-N (10.55776/FG9), Al4GreenHeettinGrris, (FFG- 899943), Stars4Waters and FWF Bilateral Artificial Intelligence 10.55776/COB12. We thank NXAI GmbH, Audi AG, Sil- icon Austria Labs (SAL), Merck Healthcare KGaA, GLS (Univ. Waterloo), TÜV Holding GmbH, Software Competence Center Hagenberg GmbH, dSPACE GmbH, TRUMPF SE + Co. KG. We thank Niklas Schmidinger and Thomas Schmied for encouraging us to share our chemical reasoning benchmark and for highlighting its relevance to the LLM community.
(I7ZON-CL62200-CCIIAATTTT
ETHICS STATEMENT
This work introduces a benchmark for evaluating chemical reasoning in large language models. The benchmark is based solely on publicly available molecular structures from PubChem and symbol- ically verifiable tasks, avoiding the use of sensitive proprietary data. Our tasks are designed to measure reasoning ability rather than to suggest new molecules with biological activity, and thus do not constitute drug discovery or generate compounds of concern. We caution that improvements on our benchmark should not be interpreted as evidence of safety or efficacy in real-world chemi- cal applications. The benchmark is intended to advance transparency, reproducibility, and rigorous evaluation in AI for chemistry while mitigating risks of data leakage and misuse.
REPRODUCIBILITY STATEMENT
The exact benchmark version used in this paper is publicly available on the Hugging Face Hub as https: htth/hhgggggggee co ct/ training pool is provided separately as (tttaasts//ll-tuummmmlllllaaaatt vO.0, 0, and the mo melecularig-trainee co
The public leaderboard is hosted at https: http//hhggggggaac///o//o// leaderboard and is evaluated on a non-public suc- cessor version of MolecularIQ to maintain benchmark integrity. A central overview repository linking all MolecularIQ components, including symbolic RDKit-based solvers, task generators, dataset construction, and evaluation code, is available https: //gi //ggthb.b../..//////////g/.... com/ml /juu/ mo moleculariq.
10

---

REFERENCES
AI@Meta. Llama 3 model card, 2024. URL
blob/ma http//maan//MM/LLL./ CARD md.
George Baker, Mario Sanz-Guerrero, and Katharina von der Wense. Molecular string representa- tion preferences in pretrained Ilms: A comparative study in zero-& few-shot molecular property prediction. Proceedings of the 2025 Conference on Empirical Methods Natural Language Processing, pp. 1071-1088 2025.
Guy W Bemis and Mark A Murcko. The properties of known drugs, 1. molecular frameworks. Journal of Medicinal Chemistry, 39(15):2877-289, 1996.
Steven H Bertz. The first general index of molecular complexity. Journal of the American Chemical Society, 103((2):3599-360) 1981.
Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB/00171), pp. 21-29. IEEE, 1997.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, et al. Language models are few-shot learners. Advances in Neural Information Processing Sys-
tems, 33:1877-1901, 2020.
ByteDance Seed Team. Seed-OSS open-source models. ByteDance-Seedd/ seeeeeddseed-osee 2025.
https://githi:t com/
Robert S Cahn, Christopher Ingold, and Vladimir Prelog. Specification of molecular chirality. Ange- wandte Chemie International Edition in English, 5(4):385-415 1966.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Vore Katherine Lec, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pp- 2633-2650, 2021.
Nikhil Chandak, Shashwat Goel, and Ameya Prabhu. Incorrect Baseline Evaluations Call into Ques- tion Recent LLM-RI. Claims. Notion Blog, 2025.
Jorg Degen, Christof Bressceid-Gerlacee Andrea Zaliani, and Matthias Rarey. On the art of com- piling and using drug-like chemical fragment spaces. ChemMedChem, 3(10):1503, 2008. 2008.
Carl Edwards, ChengXiang Zhai, and Heng Ji. Text2Mol: Cross-modal molecule retrieval with natural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 595-607, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. 10. 0.0. 18653/
10w53/81/620000 /v112221...... emnlp-main. 47.
Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models, March 2024. 2xiv:2306.08013 [q-bio].
Nelson Fernandez, Air Liquide, Maxime Illouz, Luis Pinto, Entao Yang, and Habiboulaye Amadou Boubacar. Mega: A large-scale molecular editing dataset for guided-action optimization. Work- shop: N.euris,ce-NeuriPS,, 2025.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open LLM Leaderboard v2, 2024. URL https:/ htw.gggggaaccgc.eecccooo co/ spaces/ loen-ilm-leadezte catpard/ppppeneiieeeee opeooppenn__ 1im leaderboard.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, et al. The Language Model Evaluation Harness, July 2024. URL /tR wwt.orrrrrrrr////oooooooo.o.ss.t 12608602.
Gemma Team. Gemma, 2024. URL www.tttps///m
Gemma Team. Gemma 3, 2025. URL :ttptp::://oo.. //tp:///o.oooo/o/ wew.ggggeeeeeggaaeee /eemaa Report.
11

---

Arnav Gudibande, Eric Wallace, Charlie Victor Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary language models. In The Twelfth International Conference on Learning Representations, 2024.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in Ilms through reinforce- ment learning. Nature, 645(808) (:6/808))6633--11 )::33-6334 2025. 10. 1038/ / 184416688000 0225-09222-2
Kehan Guo, Bozhao Nan, Yujun Zhou, Taicheng Guo, Zhichun Guo, Mihir Surve, Zhenwen Liang, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Can LLMs Solve Molecule Puzzles? A Multimodal Benchmark for Molecular Structure Elucidation. Advances in Neural Information Processing Systems, 37:144721-3 December 2024.
Huang, R Zhang, X He. X Zhi, H Wang, X Li, et al. ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models, September 2024. arXiv:2409. 13989 [cs].
Erich Hückel.. Quanstentheoretischee beiträge zum benzolproblem: Ii. quantentheorie der induzierten
polaritäten. Zeitschrift für Physik, 72(5):310-3) (003:310-3330 10-337 1931.
Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of Computing, pp. 604-613, 1998.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxiv.org// abs 2310. 06825.
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, et al. Pubchem 2023 update. Nucleic Acids Research, 51(D1))D1333-D11380 2023. wtw11//////gkaaaaea
Greg Landrum et al. Rdkit: Open-source cheminformatics, 2006.
Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. Mining of massive data sets. Cam- bridge University Press, 2020.
Hao Li, He Cao, Bin Feng, Yanjun Shao, Xiangru Tang, Zhiyuan Yan, Li Yuan, Yonghong Tian, and Yu Li. Beyond Chemical QA: Evaluating L.M'' Chemical Reasoning with Modular Chemical Operations. arXiv preprint arXiv:2505.2 2025.
Jiatong Li. Junxian Li, Yunqing Liu, Dongzhan Zhou, and Qing Li. Tomg-bench: Evaluating Ilms on text-based open molecule generation. arXiv preprint arXiv:2412.146422 2024a.
Jiatong Li. Yunqing Liu, Wenqi Fan, Xiao-You Yong Wei, Hui Liu, Jiliang Tang, et al. Empowering molecule discovery for molecule-caption translation with large language models: A ChatGPT perspective. IEEE Transactions on Knowledge and Data Engineerüng, 2024b.
Xuan Liu, Siru Ouyang, Xianrui Zhong, Jiawei Han, and Huimin Zhao. Fgbench: A dataset and benchmark for molecular property reasoning at functional group-level in large language models. arXiv preprint arXiv:2508.01055, 2025.
Yuyan Liu, Sirui Ding, Sheng Zhou, Wenqi Fan, and Qiaoyu Tan. MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction. arXiv preprint arXiv:2406. 12950. 2024.
Zequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, and Tie- Yan Liu. Molxpt: Wrapping molecules with text for generative pre-training. arXiv preprint arXiv:2305.1064 2023a.
Zhiyuan Liu. Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Molca: Molecular grapaelanguage modeling with cross cross-modal projector and uni- modal adapter. arXiv preprint ar1iv:2310. 12798, 2023b.
12

---

Daniel Lowe. Chemical reactions from us patents, 2017.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The Al scien- tist: Towards fully automated open-ended scientific discovery. arXiv preprint FrXiv22408.0629) 2024a.
Xingyu Lu, He Cao, Zijing Liu, Shengyuan Bai, Leging Chen, Yuan Yao, Hai-Tao Zheng, and Yu Li. MoleculeQA: A dataset to evaluate factual accuracy in molecular comprehension. arXiv preprint arXiv:2403.085 08992 2024b.
Benjamin P MacLeod, Fraser GL Parlane, Thomas D Morrissey. Florian Häse. Loie M Roch, Ke- van E Dettelbach, Raphaell Moreira, Lars PE Yunker, Michael B Rooney, Joseph R Deeth, et al. Self-driving laboratory for accelerated discovery of thin-film materials. Science Advances, 6(20), 2020.
Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Benedict Emoekabu, Aswanth Krishnan, Mara Wilhelmi, Macjonathan Okereke, Juliane Eberhardt, Amir Mohammad Elahi, Maximilian Greiner, Caroline T. Holick, Tanya Gupta, Mehrdad Asgari, Christina Glaubitz, Lea C. Klepsch, Yannik Köster, Jakob Meyer, Santiago Miret, Tim Hoffmann, Fabian Alexander Kreth, Michael Ringleb, Nicole Roesner, Ulrich S. Schubert, Leanne M. Stafast, Dinga Wonanke, Michael Pieler, Philippe Schwaller, and Kevin Maik Jablonka. Are large language models superhuman chemists?
Nature Chemistry, 17:984-988 2025. 10. 1121///415577-5-01555555
Mistral AL Nisttal-small-2A--nnsrruee / //hugggigg booggingfar acee. ca mwiittaaaa/iiitt........ saairrall-mmmllmmm 1 24B Inssrnctt225555
Hugging Face model repository, 2025. URL https:
Ac-
cessed: 2025-09-24.
Siddharth M Narayanan, James D Braza, Ryan-Rhys Griffiths, Albert Bou, Geemi Wellawatte, Mayk Caldas Ramos, Ludovico Mitchener, Samuel G Rodriques, and Andrew D White. Training scientific reasoning model for chemistry. arXiv preprint arXiv:2506. /7238, 2025.
NVIDIA. Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model, 2025a. URL httpt:t////// rxiv www.i..oo...aaoooooooo.oo.. wog///rr//////a/a 2508 214.55188844... 14444.
NVIDIA. Nvidia Ilama 3.3 70b instruct fp8. Hugging Face model repository, 2025b. URL https: 3..3-700 70B- Instrncttttuttsss Accessed: 2025-09-
/ //huggingface.......////// co wivii/iiiiii/iiii/iiiiiiaaiiiiai aLLlam 3.3- 24.
OpenAl. GPT-OSS:
Open-Weight Model Release (GPT-OSS-20B, GPT-OSS-120B). URL https://openai. htt.eanii.com/nnee///..
Model documentation, August
2025.
introducing-gpt-osst
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL URLhtttss://qwenlm
github. wttpii////l//oe qwen2. 5/.
Qwen Team. Qwen3 technical report, 2025. URL
httpgg//as/2555.00950 09388.
Philipp Renz, Dries Van Rompaey, Jörg Kurt Wegner, Sepp Hochreiter, and Günter Klambauer. On failure modes in molecule generation and optimization. Drug Discovery Today: Technologies, 32:55-63, 63, 2019.
David Rogers and Mathew Hahn. Extended-comeecivity fingerprints. Journal of Chemical Infor- mation and Modeling, 50(5):742-754, 2010.
Nicholas T Runcie, Charlotte M Deane, and Fergus Imrie. Assessing the chemical intelligence of large language models. Journal of Chemical Information and Modeling, 2025.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng. Yiping Wang, Sewoong Oh, et al. Spurious rewards: Rethinking training signals in RLVR. arXiv preprint 10940:::500..04477,, 2025.
Aarohi Srivastava, Abhinay Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, al. Beyond the imita- tion game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
13

---

Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. SciEval: multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 19053-19061, 2024.
Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, et al. Chemagent: Self-updating memories in large language models improves chemical reasoning. In The Thirteenth International Conference on Learning Representations, 2025.
Gary Tom, Stefan Schmid, Sterling G Baird, Yang Cao, Kourosh Darvish, Han Hao, Stanley Lo, Sergio Pablo-García, Ella M Rajaonson, Marta Skreta, et al. Self-driving laboratories for chemistry and materials science. Chemical Reviews, 124(16):9633-7733 9732, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0 092288, 2023.
Eric Wang, Samuel Schmidgall, Paul F Jaeger, Fan Zhang, Rory Pilgrim, Yossi Matias, Joelle Barral, David Fleet, and Shekoofeh Azizi. Txgemma: Efficient and agentic Ilms for therapeutics. arXiv preprint arXiv:2504.00 06/96, 2025a.
Miles Wang, Robi Lin, Kat Hu, Joy Jiao, Neil Chowdhury, Ethan Chang, and Te- jal Patwardhan. Frontierscience: Evaluating ai's ability to perform expert-level sci- entific tasks. Technical Report, 2025b. URL https: / cdn. openai. www./oo/////o.ooo / 0017733933effo/fronctee
pacpeeeppppprpppppp .dd..
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SciBench: Evaluating College-Level Sci- entific Problem-Solving Abilities of Large Language Models. arXiv preprint urXiv:2307.10635, June 2024a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.1 1/11,, 2022.
Yubo Wang. Xueguang Ma. Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo. Weiming Ren, Aaran Arulraj, Xuan He, Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances Neural Information Processing Systems, 97:95266-95290 2024b.
David Weininger. SMILES, chemical language and information system. 1. introduction to method- ology and encoding rules. Journal of Chemical Information and Computer Sciences, (8(1):31-36 1988. 10. 1021 / ci 000570000
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, al. LiveBench: A Challenging, Contamination- Limited LLM Benchmark. arXiv preprint arXiv:2406.1933 2025.
Yuyang WU, Jinhui Ye, Shuhao Zhang, Lu Dai, Yonatan Bisk, and Olexandr Isayev. Molerr2fix: Benchmarking I1m trustworthiness chemistry via modular error detection, localization, expla- nation, and correction. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 19365-19382, 2025.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, et al. Moleculenet: a benchmark for molecular machine learning. Chemical Science, 9(2); 513-530, 2018
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The Al scientist-v2; Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504. 250.:25000000666... 2025.
Geyan Ye, Xibao Cai, Houtim Lai, Xing Wang, Junhong Huang, Longyue Wang, et al. Drugassist: A large language model for molecule optimization. Briefings in Bioinformatics, 26(1) bbbb:6,,, 2025.
14

---

Botao Yu, Frazier N. Baker, Ziqi Chen, Xia Ning, and Huan Sun. LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset, August 2024. arXiv:2402.09391 [cs].
Z.ai. Glm-4.6: Advanced agentic, reasoning and coding capabilies, 2025. URL https://22 blog/ ht/mmmmmmmm44 4. 6. Blog post announcing GLM-4.6 model release.
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.0647) 2025.
Zheni Zeng, Bangchen Yin, Shipeng Wang. Jiarui Liu, Cheng Yang. Haishen Yao, al. ChatMol: interactive molecular discovery with natural language. Bioinformatics, 40(9):btae534, 2024.
Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, Dongzhan Zhou, Shufei Zhang, Mao Su. Han-Sen Zhong, and Yuqiang Li. ChemLLM: A Chemical Large Language Model, April 2024. arXiv:2402.06852 [cs].
Guojiang Zhao, Sihang Li, Zixiang Lu, Zheng Cheng, Haitao Lin, Lirong Wu, Hanchen Xia, Hengx- ing Cai, Wentao Guo, Hongshuai Wang. et al. Molreasoner: Toward effective and interpretable reasoning for molecular 1lms. arXiv preprint arXiv:2508.02066, 2025a.
Zihan Zhao, Bo Chen, Ziping Wan, Lu Chen, Xuanze Lin, Shiyang Yu, Situo Zhang, Da Ma, Zichen Zhu, Danyang Zhang, Huayang Wang. Zhongyang Dai, Liyang Wen, Xin Chen, and Kai Yu. Chemdfm-r: An chemical reasoner Ilm enhanced with atomized chemical knowledge, 2025b.
URL httph:tt//// http///axvvvv//. oro.o/////oooooo. abs/5/2 21990.
Zihan Zhao, Da Ma, Chen, Liangtai Sun, Zihao Li, Yi Xia, al. Developing ChemDFM as large language foundation model for chemistry. Cell Reports Physical Science, 6(4), 2025c.
10.101 6/ xcrp 2025. 102523.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instroction-follovinng evaluation for large language models, arXiv preprint 2tti::2311.09///1 2023.
15

---

MolecularIQ Appennixx
A MOLECULARIQ dataset details
17
A.I Molecular features
17
A.2 Sampling and task construction
21
A.3 Question templates and examples
25
B Details on the benchmark experiment
27
B.1
Models
27
B.2 Details on evaluation
27
B.2.1
Integration into - harness
27
B.2.2
System prompt
30
B.2.3
Verifiers and answer extraction
30
C Extended results
33
C.I
Extended performance breakdown
C.1.1
Overall performance and across benchmark dimensions.
33
C.1.2
Performance across molecular features
37
C.1.3
Sensitivity to SMILES representation
44
C.1.4
Response length analysis
48
C.1.5
Rollout ablation
49
C.1.6
Chemistry LLMs and base model comparison
49
C.1.7
Output type validity
49
C.2
Failure mode analysis
................................
51
C.2.1
Overview of systematic failures
C.2.2
Task-specific success patterns
51
C.2.3
Performance transitions under compositional stress ................................
C.2.4
Output format validity analysis
54
C.2.5
Failure analysis on substructures
55
C.2.6
Chain-of-thought failure analysis
56
C.2.7 Failure mode examples
64
D Use of Large Language Models
78
16

---

A MOLECULARIQ DATASET DETAILS
This section provides extended information molecular features included (Section A.1), the dataset creation procedure (Section A.2), and presents examples of the questions included (Section A.3).
A.1 MOLECULAR FEATURES
MOLECULARIQ covers 30 different molecular features across the six groupings introduced Sec- tion 3.2. Table Al summarizes the categorization, and in the following paragraphs, give short description of each feature.
Table A1: Overview of tasks in MOI.CULARIQ
Feature category
Name
Graph topology
Rings
Graph topology
Fused rings
Graph topology
Bridgehead atoms
Graph topology
Smallest or largest ring size
Graph topology
Chain termini
Graph topology
Branch points
Chemistry-typed graph topology
Aromatic rings
Chemistry-typed graph topology
Aliphatic rings Heterocycles
Chemistry-typed graph topology
Chemistry-typed graph topology Chemistry-typed graph topology Chemistry-typed graph topology Chemistry-typed graph topology Chemistry-typed graph topology Chemistry-typed graph topology Chemistry-typed graph topology
Saturated rings sp² carbons Longest carbon chain R or S stercocenter Unspecified stereocenter
E or Z stereochemistry double bond Stereachemistry unspecified double bond Stereocenters
Chemistry-typed graph topology
Composition
Carbon atoms Hetero atoms
Composition
Composition
Halogen atoms
Composition
Heavy atoms
Composition
Hydrogen atoms
Composition
Molecular formula
Chemical perception
HBA
Chemical perception
HBD
Chemical perception
Rotatable bonds
Chemical perception
Oxidation state
Functional groups
Functional groups
Synthesis and fragmentation
BRICS decomposition
Synthesis fragmentation
Template based reaction prediction
Synthesis and fragmentation
Murcko scaffold
GRAPI TOPOLOGY
Rings. Ring detection is a fundamental aspect of molecular structure analysis, cyclic motifs present in the majority of biologically active compounds. This task requires identifying whether molecule contains ring structures and determining their number. While trivial for cheminformatics toolkits, it challenges L.Ms to correctly parse connectivity patterns beyond simple linear chains. Performance on this task provides a baseline indicator of whether models can capture cyclicity, which underpins aromaticity, rigidity, and scaffold formation in medicinal chemistry.
Fused rings. Fused or condensed rings occur when two or more rings share one or more atoms, producing scaffolds such as naphthalene steroid backbones. Detecting fused rings requires models not only to identify individual cycles but also to recognize shared boundaries, which adds a layer of structural complexity. Correct reasoning here is crucial for capturing polycyclic scaffolds that dominate many drug classes. Errors indicate that models may treat rings independently without appreciating their overlap and fusion.
17

---

Bridgehead atoms. Bridgehead atoms are special positions where two often bicyclic or polycyclic systems. These atoms influence molecular strain, conformational rigidity, and reactivity. Identifying them requires models trace cycles through shared atoms and to distinguish simple substitution from ring-ring junctions. Success on this task indicates structural reasoning beyond mere ring counting, while failure highlights limitations in global graph under- standing.
rings intersect,
Smallest largest ring size. Ring size strongly influences chemical reactivity, strain, and biolog- ical activity. This task challenges models to determine the size of either the smallest or the largest ring molecule, requiring explicit counting of atoms within cycles. Such reasoning tests whether models can not only detect rings but also evaluate their dimensions, which are essential for un- derstanding macrocycles or strained small rings. Performance degradation here would suggest that models rely on heuristics instead of robust ring-perception logic.
Chain termini. Chain termini are the end points of linear carbon or heteroatom chain, typically corresponding to substituents or free valences. Detecting termini requires parsing connectivity to identify atoms degree one. While chemically straightforward, this task probes whether models can recognize basic graph-theoretic features such as leaf nodes, which are important for subsequent reasoning about branching, chain length, and attachment sites. Failures indicate difficulty in trans- lating molecular strings into graph structures.
Branch points. Branch points occur where linear chain splits into two or more substituents, creating tree-like structures. These positions affect molecular shape, flexibility, and lipophilicity, which in turn influence pharmacokinetics. Detecting branch points requires identifying atoms of degree three more in the molecular graph. Correct reasoning here demonstrates that model can capture local topology rather than treating molecules as simple chains.
CHEMISTRY-TYPER GRAPH TOPOLOGY
Aromatic rings. Aromatic systems are rings with delocalized r-electrons, following Hückel's rule (Hückel, 1931). They play a central role in stability, reactivity, and biological activity. Identifying aromatic rings more complex than detecting generic cycles, since resonance and hybridization must be considered. This task, therefore, probes whether model understands chemical typing beyond pure graph topology.
Aliphatic rings. Aliphatic rings are saturated, non-aromatic cycles such cyclohexane cy- clopentane. While structurally simple, they contribute unique conformational preferences and steric constraints. Distinguishing aliphatic from aromatic rings requires reasoning about bond order and hybridization, not just connectivity. Success on this task indicates a model's ability to classify rings by chemical type rather than by topological presence alone.
Heterocycles. Heterocycles contain at least one non-carbon atom in the ring, such as nitrogen in pyridine oxygen in furan. These motifs dominate pharmaceuticals, dyes, and agrochemicals due to their tunable electronic properties. Detecting heterocycles requires models to distinguish atom types inside rings and recognize their chemical implications. This task thus evaluates both structural perception and atom-level classification.
Saturated rings. Saturated rings are cycles with only single bonds, often flexible and conforma- tionally rich. They differ strongly from aromatic or unsaturated rings in chemical behavior and reactivity. Identifying them requires counting bond types within cycles and ensuring that no unsatu- ration is present. Accurate reasoning here shows that model can integrate bond order information with ring detection.
sp3 carbons. Carbons in sp² hybridization are bonded to four atoms via single bonds, leading to tetrahedral geometry. The proportion of sp² carbons is well-known descriptor of drug-likenes,, and scaffold diversity. Counting them requires distinguishing hybridization states based on connectivity and bond order. This task highlights whether a model can map molecular graphs to stereoclectronic properties.
Longest carbon chain. The length of the longest continuous carbon chain is fundamental in nomen- clature and structural description. It often defines the molecule's 's main scaffold in IUPAC naming. dentifying requires global graph traversal to find maximum-lena length paths, testing whether models perform non-trivial structural reasoning beyond local atom counting.
18

---

R or S stereocenter. Chirality is a key determinant of biological activity, with enantiomers often having drastically different pharmacological profiles. Assigning R/S configuration requires applying the Preloo rules (Cahn al., 1966) to rank substituents and determine stereochemical orientation. This task is demanding because it requires integrating atom identity, connectivity, and three-dimensional orientation. Strong performance indicates sophisticated reasoning about stereo- chemistry rather than two-dimensional connectivity alone.
Unspecified stereocenter. In many SMILES strings or molecular depictions, stereochemistry is left undefined even when a chiral center exists. Detecting such unspecified stereocenters is important for identifying ambiguity in data curation and preventing misinterpretation of molecular activity. Models must therefore distinguish between the presence of chirality and the explicit encoding of stereochemistry. This task tests awareness of missing information, not just positive structural fea- tures.
or stereochemistry double bond. Double bonds with restricted rotation can give rise geo- metric isomers, classified E (opposite) or Z (together). Correct assignment requires recognizing substituent priorities on both carbons and applying geometric rules. This property has significant effects on molecular conformation and bioactivity. The task evaluates whether a model can interpret bond stereochemistry beyond connectivity.
Stereochemistry unspecified double bond. In some structures, double bonds present without explicit E/Z specification. Identifying such cases is crucial for flagging ambiguous molecular data. This requires detecting when stereochemical information is missing, rather than misclassifying it as a defined configuration. The task complements stereochemistry classification tasks by emphasizing uncertainty detection.
Stereocenters.
Stereocenters are atoms (often carbons) that give rise to stereoisomerism by con- necting to distinct substituents. Counting them provides insight into molecular complexity and the potential number of stereoisomers, Models must integrate atom identity, connectivity, and sub- stituent uniqueness succeed. This task probes holistic reasoning about stereogenicity across the entire molecule.
COMPOSITION
Carbon atoms. Simply counting carbons may appear trivial, but it provides baseline check of whether a model parses molecular graphs correctly. Carbon count influences molecular weight, scaffold size, and lipophilicity. Models must map SMILES tokens atom identity and correctly handle ring closures and branching. Errors here indicate failure in basic atom-level parsing.
Hetero atoms. Heteroatoms such nitrogen, oxygen, and sulfur govern hydrogen bonding, po- larity, and reactivity. Counting them is critical for estimating drug-likeness and physical properties. The task requires distinguishing carbons and hydrogens from all other elements in the structure. This tests whether models encode chemical types accurately and can generalize across diverse element sets.
Halogen atoms. Halogens (F, CI, Br. I, At) are common substituents in medicinal chemistry, mod- ulating lipophilicity, metabolic stability, and binding affinity. Detecting and counting them requires recognition. While simple for cheminformatics libraries, this task challenges L.MMs to generalize symbol recognition across less frequent elements. Strong performance indicates robust chemical token handling.
Heavy atoms. Heavy atoms are defined as non-hydrogen atoms the molecule. This count correlates with molecular complexity and is often used as a simple descriptor for dataset stratifica- tion. The task is straightforward but requires models to systematically distinguish hydrogens from all other atom types. Failures reveal gaps in even the most basic structural parsing.
Hydrogen atoms. Hydrogen count is essential for deriving molecular formulae and understanding saturation. While hydrogens often implicit in SMILES, inferring their correct number requires applying valence rules. This makes the task more challenging than simply reading atom tokens. Success demonstrates that models can handle implicit chemical information, not just explicit tokens.
Molecular formula. The molecular formula condenses the atom counts of all elements into a canon- ical string, such as C,H6. Deriving it requires accurate atom counting across the molecule, includ-
19

---

ing implicit hydrogens. This is global summarization task that integrates several simpler counting tasks. It tests whether models can consolidate structural information into a chemically valid repre- sentation.
CHEMICAL PERCEPTION
Hydrogen bond acceptors (HBA). HBAs are atoms with lone pairs (commonly oxygen nitrogen) capable of accepting hydrogen bonds. They determine solubility, binding interactions, and pharma- cokinetics. Identifying HBAs requires applying chemical perception rules, not just atom types, since context (e.g., resonance, hybridization) matters. This task probes whether models encode functional group semanties.
Hydrogen bond donors (HBD). HBDs are groups (c.g., OH, NH) that provide a hydrogen atom for hydrogen bonding. They are central in protein-ligand recognition and solubility. Counting HBDs requires models to parse both atom type and bonding pattern. Accurate predictions reflect chemical reasoning about donor functionality rather than raw atom identity.
Rotatable bonds. Rotatable bonds provide conformational flexibility, which influences molecular dynamics and bioavailability. Identifying them requires excluding bonds in rings and terminal bonds while recognizing non-rigid single bonds. This task goes beyond simple bond counting, demand- ing perception of conformational constraints. It evaluates whether models can integrate structural context into bond classification.
Oxidation state. The oxidation state of an atom reflects the distribution of electrons in bonds and is critical for predicting redox behavior. Assigning it requires applying electron-counting counting rules con- sistently across a molecule. This task is symbolically verifiable but conceptually demanding, since it combines atom types, bonding patterns, and electronegativity rules. Performance here indicates whether models can implement algorithmic chemical reasoning.
FUNCTIONAL GROUPS
Functional groups Functional groups define classes of reactivity, such as alcohols, amines, and car- bonyls. Identifying them requires pattern recognition across atom types, bond orders, and connec- tivity motifs. This task encapsulates the chemist's 's notion of "chemical perception," " linking structure to reactivity. It tests whether models can map from subgraph motifs to abstract functional categories.
SYNTHESIS AND FRAGMENTATION
BRICS decomposition. BRICS (Breaking of Retrosynthetically Interesting Chemical Substructures (Degen et al., 2008)) rule-based fragmentation method that splits molecules into chemically meaningful building blocks. Performing this decomposition requires applying substitution rules and graph-partitionnng logic. The task challenges models to reproduce widely used cheminformatics procedure. Success indicates capability for rule-driven reasoning over chemical graphs.
Template-based reaction prediction. Template-basee methods encode chemical transformations as reaction rules (encoded SMIRKS), which can be applied to reactants to generate products. Each template specifies a substructure match and an atom-mapped rewrite; the model must (i) localize the reactive site(s) in the input molecule and (ii) apply the transformation to yield the product. This setup mirrors how rule libraries are used in medicinal chemistry and CASP, emphasizing structural reasoning and atom mapping rather than surface-level classification. To avoid condition-dependent or stoichiometric ambiguities, we restrict the benchmark to common, unimolecular, single-step re- actions with deterministic + mappings (e.g., oxidations, reductions, substitutions, additions, climinations, aromatic substitutions, rearrangements, protection/deproteciio. ring openings/cleav- ages). These constraints provide unambiguous ground-truth products and isolate model's ability to understand and manipulate molecular structure.
Murcko scaffold. The Murcko scaffold (Bemis & Murcko, 1996) is defined as the union of all ring systems and linkers, with side chains removed. It represents the core structure underlying molecule's chemical series. Extracting requires identifying which atoms belong to the scaffold versus substituents. This task tests whether models can abstract molecular graphs into canonical cores, key step in medicinal chemistry analysis.
20

---

A.2 SAMPLING AND TASK CONSTRUCTION
Corpus preparation. Starting from a pool of molecules, we filter them to have between 5-50 heavy atoms, SMILES strings shorter than 100 characters, and valid (parseable) SMILES repre- sentations, ensuring the benchmark focuses on small molecules of reasonable complexity while avoiding trivial small molecules or overly complex macromolecules that could skew property distri- butions. Further, we calculate target properties of these compounds using our implemented solver functions. To ensure representative coverage across molecular complexity, we stratify our dataset 'D of molecules into three Bertz complexity bins (Bertz, 1981): [0, 250), [250, 1000), and [1000, oo). Figure Al illustrates a few example molecules from each complexity bin. From the property table in- dexed by SMILES, we construct three task families-comme index, and generation- eecch in single- and multi feature variants. Sampling employs inverse- frequeccyyy weighting to promote diversity and prevent verrreprreenttttinn of common cases.
Single-feature count and index task generation. For each complexity bin and feature, we gen- erate 10 examples per bin using an tivers-frrquency sampling scheme designed maximize diver- sity. For each molecule, we calculate property frequency, invert it, and down-weight zero cases by 0.5 obtain meaningful examples that avoid trivial "absent" cases.
Each selected molecule-feature pair generates a count question. If the property has corresponding index mapping, additionally create a paired index question for the same molecule. This allows for full traceability of model capabilities across different task types, while maintaining the same molecule.
multti-eattue count and index task construction. To test multi-step complex reasoning, we cre- ate tasks requiring simultaneous analysis of n E {2, 3, 5) properties on the same molecule. Starting from molecules already used in single-property tasks, we:
1. Begin with properties already associated with that molecule using doubly frequecyy sampling with zero bias (as in single-property tasks)
2. Add additional properties using eiverse-frqueen logic to maintain diversity
3. Apply plausibility filters: reject samples with more than one zero-valued property to avoid trivial cases
4. Target 100 items per complexity bin and per property count
We maintain the same pairing logic between count and index tasks as before. This approach ensures multi-property tasks build naturally from single-property foundations while maintaining chemical plausibility. It also allows us to trace whether a model can handle multitask load meaningfully, ensuring we don't 't lose skills already demonstrated in simpler tasks.
Single-constraint generation. We derive exact equality constraints (property, value) from single- count tasks, supplemented by additional constraint tasks not present in the count case. We aim to generate the same number of single-constraint tasks as count tasks. However, finding enough examples for certain count properties is infeasible, so we supplement with the unique property set. If that is smaller than the target value, we take all; otherwise, we apply inverse-frequency sampling.
Multi-constraint generation sampling Our multi-constraint generation tasks require molecules to simultancously satisfy & E 3,5 property constraints, creating increasingly complex reasoning challenges. The sampling procedure addresses two key challenges: ensuring chemical satisfiability while maintaining appropriate task difficulty, and preserving lineage relationships to parent tasks.
Constraint candidate construction. For each molecule, we construct a prioritized pool of candi- date constraints following hierarchical scheme: (1) seed constraints from single-constraint parent tasks (mandatory for lineage), (2) properties from multi-property parent tasks, (3) additional molec- ular count properties, and (4) special properties including molecular formulas, Murcko scaffolds, and functional groups. This hierarchy ensures that multi-constraint tasks build upon simpler prereq- uisites while introducing novel property combinations.
informattio-thherref scoring. We score k-constraint combinations based on their collective in- formation gain and rarity. For a combination C {C1, ..., ), we compute the cumulative reduction
21

---

Complexity bin: 0-250
Complexity bin: 250-1000
335
434
443
500
508
577
686
760
954
Complexity bin: 1000-inf
1031
1097
1110
1214
1215
1414
1454
1611
1644
2170
Figure Al: Example molecules from each Bertz complexity bin, labeled with their Bertz complexity.
22

---

ratio:
Mez .... O Ma-il
R(C) II
MMa n... Mai
where Me, denotes the of molecules satisfying constraint cę. Constraints randomly ordered before computing per-step reduction ratios. This measures how much each additional constraint reduces the satisfying molecule set. We require each non-zero constraint achieve reduction ratio 2 1.1 (reducing the satisfying set by at least 10%), while zero-valued constraints must achieve 2 2.0 (50% reduction) to justify their inclusion. Combinations must satisfy between 10 and 1,200 molecules for k = 2, with relaxed bounds of 5-1,500 molecules for l 5, ensuring non-trivial but achievable tasks.
Diversity-aware selection. The final selection employs Monte Carlo sampling with rejection cri- teria ensuring: (1) at least one seed constraint for task lineage, (2) no redundant property pairs (e.g. molecular formula with elemental counts), (3) most one zero-valued constraint meeting the stricter reduction threshold, and (4) adherence to k-specific satisfiability bounds. Tasks achieving molecule counts within ideal ranges (60-220 for k 2, 20- 20-120 for 5) receive scoring bonuses, while combinations with prelevance rates below 5 % are prioritized for their rarity.
Solver reliability Our symbolic solver leverages RDKit (Landrum et al., 2006), the industry- standard cheminformatics toolkit. While we inherit RDKit's computational assumptions, this en- sures compatibility with existing workflows and reproducibility across research groups. Rather than resolving fundamental chemical ambiguities (c.g., tautomerism, aromaticity models), pri- oritize self-consistency: tautomers maintain distinct identities, aromatic and Kekulé representa- tions yield identical outputs despite different internal states, and undefined stereochemistry is ex- plicitly tracked. Validation on 10,000 randomly sampled molecules demonstrates 100% internal consiseency-strroccenitttt counts always sum correctly, property calculations remain determinis- tic, and all 50+ solver methods produce valid outputs without crashes. This approach acknowledges that different chemical representations can be equally valid while ensuring that our benchmark pro- duces reproducible results regardless of the underlying representational choices.
Quality controls and reproducibility Every generated task undergoes self-consistency valida- tion: property values computed during generation match those evaluation, constraint satisfiability verified against the source molecular dataset, and parent-child task relationships preserve prop- erty values across difficulty levels. To ensure reliable and reproducible benchmarks, we implement several quality measures:
Deterministic sampling: All random processes use fixed seeds
Usage tracking: Counters ensure balanced coverage across categories and properties
Deduplication: Tasks are deduplicated and assigned unique identifiers
Validation: All constraints are verified against source molecules using symbolic checkers
The resulting dataset is exported as a HuggingFace with complete test splits and task-type organiza- tion. Figure 2 provides a visual overview of the dataset's s structure- druuctue-iincluding task-type distribution, multitask load, and Bertz complexity-wii where the sunburst on the left depicts the three hierarchical levels and the stacked bars on the right show the feature-group composition across counting, in- dexing, and generation tasks. Figure A2 shows the distribution of molecular Bertz complexity for MOLECULARIQ compared with ChemCOTBench and ChemIQ.
23

---

600
Frequency
400
500
1000
1500
2000
2500
3000
3500
4000
600
Frequency
400
200
4000
Irequency
200
500
1000
1500
2000
2500
3000
3500
4000
Bertz complexity
Figure A2: Bertz complexity distribution of molecules MOLECULARIQ (top), ChemCOTBench (middle), and ChemlQ (bottom).
24

---

A.3 QUESTION TEMPLATES AND EXAMPLES
Questions generated from predefined templates (see Figure A3). Separate template sets are used for single- and multi-fcature counting, single- and multi-feature indexing, and constrained generation. Individual questions instantiated by replacing the template keywords constraint, count sype(s), and index rype(s) with the corresponding property descriptors. These descriptors are preprocessed by natural language formatter the final test set, we further refined the questions using API calls to both GPT-S-mini and Claude 4.5 Sonnet; employing multiple models mitigates potential bias toward any single model's phrasing or style. Question quality was verified through both automatic and manual checks, such as manually inserting SMILES strings and verifying the correctness of JSON keys.
grammatical correctness and consistency. For
task wask_tmpllated ates "iingee_contt;;
"single_ count";
"Count the number of @oount_typee in the molecule (smiles)", "Analyze (smiles) and report the number of (count_type)... "what is the (count_type) count in {smiles)?",
"multi_count":
"For the molecule (smiles), count the follewing features: { count_types).""
"List the counts for each of these features (smiles): count_types).
"Summarize the counts for (count_types) detected in [smiles)
},
"single_inder"
15
"Identify the indices of (index [index_type] in the molecule /smiles).", "Output the index list of [index (yde) in compound (smiles).", "Which atom indices represent [index_type in [smiles)?",
19
20
"multi_index""
"For (smiles), please lccate the indices of: (index_types). "For compound (smiles), locate: (index_types)... "Where are the (index_types) located (smiles)?",
23
24
25
constraint ger.riattggeeeaiiinn
"Generate molecule with (constraint)."
28
"Propose a molecular sucture that respects (constraint)." "Can you generate molecule that satisfies (constraint)?"
29
Figure A3; Task templates to generate single- and multitask questions for counting tasks, indexing tasks, and constraint generation.
Here, we show some example questions:
Question example for single count tasks
Question: How many carbon atoms are in CCCCCMM--CICCCCCCCCCCCC Return the nu- merical answer as a JSON object with the field carbon www.aattooouui
Question example for single index-based attribution tasks
Question: Which atoms in cICCM223cFPccccc//CCC-ccccc
part of saturated
rings? Please provide your answer using saturated wwaputtedddinnineeeee index for the list of atom indices.
25

---

Question example for single constraint generation tasks
Recommend molecule that fulfills exactly 13 hydrogen bond acceptors. Return the result as JSON object using the key 'smiles'.
Question:
Question example for multi count tasks
Question: How many unspecified B/Z (stereochemistry) double bonds are in what its molecular formula, and how many atoms make up the longest continuous carbon chain? Return the answers JSON object ing the keys stereochemistry unspecified double bond count, molecular formula count, and longest carbon chain count.
Question example for multi index-based attribution tasks
Question: Which atom indices in NHCCICNCCNICCICCOCO are:
hydrogen bond acceptors,
heteroatoms,
atoms in aliphatic rings,
branch points,
atoms in Z-double bonds?
Give the actual atom index positions for each feature and provide the answer a JSON ob- ject mapping hha index, hetero atom index, aliphatic ring index, branch point index, and pesttrrochemiitry_dobbblbb douulle.onnd__jieg
their respective index lists.
Question example for multi constraint generation tasks
Question: Build molecule containing exactly 5 S-stereocenters and exactly saturated rings. Respond with a JSON mapping whose key is 'smiles'.
26

---

B DETAILS ON THE BENCHMARK EXPERIMENT
B.I MODELS
This section provides details on the models included in the benchmark experiment, such as the full model name and its associated name on HuggingFace.
Table Bl: Model nomenclature mapping our abbreviations to official names and HuggingFace iden- tifiers.
Abbreviation Used
Model Name
HuggingFace Identilier
Chemistry LLMs
ChemLLM
LM (Zhao et al., 2025c)
LCCchen/ChemLLM-1B---
LlaSMol
(Yu et al., 2024)
usunlp/LiaSMul
MolReasoner-CCe Cap
MolReasoner (Zhao al, 2025a)
guojian/MMoRResoner (captioning file)
MolReasoner-Ger
MolRensoner (Zhao ol., 2025a)
guojianz/MolReesoner (generation file)
3Jama-3 Mollnst
Mol-Instructions (Fang al., 2024)
sjun/p/llmma iin.mmmma-ittmtcttmliinsnn.oooo
OpenDEFM/ChemmDPPPyyIIII mac.iinntt-olccle--..
ChemDFM-8B
ChemDDM (Zhao al., 2025c)
ChemDFM-13B
ChemDFM (Zhao al., 2025c)
ChemDFM- 14B
ChemDFM (Zhao et al., 2025b)
OpenDEMMCCeenDDFM
TxGemma-999 9B
TxGemma (Wang al., 2025a)
googletxgemma-9hhe ch-chat
TxGemma-27B
TxGemma (Wang al., 2025a)
googlotngemma--7--hht
EtberO
EtherO (Narayanan
2025)
futurchousc/ttherO
Generalist LLMs
Gemma-2 9B
Gemma-2-9b-i (Gemma Team, 2024)
google/gemma-22900
Gemma-2 27B
Gemma-2-276--4 (Gemma Team, 2024)
googlo/gemma-224
Gemma-3 12B
Gemma-3 12b-it (Gemma Team, 2025)
google/gemma-33 1---gemmmm---1---
3.2hh (Gemma Team, 2025)
google/gemma: 3.27b-it
LLaMA-2 13B
LLaMA-2. 3b-chat (Touvron et al., 2023)
1-b--hht
meta-llamai Inma-2-1 33-ccht-hff
LLaMA-3
Llama 8B Meta, 2024)
LLaMA-3.3 70B
NVIDIA Llama 3,3 70B Instruct FP8 (NVIDIA, 2025b)
Mistral v0.1
Mistral 7B (Jiang et al., 2023)
mistralai/Mistral7B..
Small
Smatii-Small-24B..o 200mall-2----nsuee---250 Instrruttt011 (Mistral AI, 2025)
Nemotron-Nano 9B v2
"IDDAA-Nmmttro-NNao-------4 (NVIDIA, 2025a)
SEED-OSS
Bed--OSS-36B-Imss (ByteDance Seed 2025)
ByteDance-SecdSeed-000 I-sstrutt
Qwen 2,5 7B
Qwen2.5 7B Instruci (Qwen Team, 2024)
Qwen/Qwen2.5-77 7B Instruct
Qwen-3
Qwen3-8R (Qwen Team, 2025)
Qwen/Qwen3.81
Qwen-3 14B
Qwen3-14B (Qwen Team, 2025)
Qwen/Qwen3-14B
Qwea-3 32B
Qwen3-32B (Qwen Team, 2025)
Owen/Qwen3-322
Qwen-3 30B
Owen3-30B-A3B (Qwen Team, 2025)
Qwen-3 Next 80B
Qwen3-Noxt-80B-4 (Qwen Team, 2025)
Qwen/Qwen3-Noxx--000 B-.-:-BB-----inkiiie
Qwen-3 235B
Qwen3-235B (Qwen Team, 2025)
Qwenu(Qenn-2335-A222 TeinB-Thiiiing- 2507-FPS
GPT-OSS 20R (low**)
GPT-oss (OpenAI, 2025)
openai/gnt.os--222
GPT-OSS 208 (Med*)
GPT-oss-20b (OpenAT, 2025)
openai/gpt-oss-20h
GPT-OSS 20B (High")
GPT-oss-20b (OpenAl, 2025)
openal/gtt-os--2000
GPT-OSS (Low:f)
GPT-oss-120b (OpenAI, 2025)
openai/gpt-os-
GPT-OSS 120B (Med*)
GPT oss: 120h (OpenAI, 2025)
openai/gpt.oss oss 120h
(High*)
GPT-oss- (OpenAT, 2025)
openai/gnt-oss-120hh
GLM-.66
GLM-4.6 (Zeng al., 2025; Z.ai, 2025)
2alorrgGGLMM4--FF55
DeepSeck-RI R1-0528
DeepSeek-HR R1- R1-0552 (Guo al., 2025)
mwddh/DeepSeel---115--PPP
Low/MedHHghh refer to reasoning effort settings, not separate models
B.2 DETAILS ON EVALUATION
We evaluate models using the 1m-eva mnreevaaeatiinn--areees framework (Gao al., 2024), which provides a uniform interface across local and API-served models, standardized task configuration, and comprehensive logging of prompts, seeds, and versions. We integrate MOLECULARIQ as cus- tom task to ensure reproducible evaluation while allowing model-specific optimization of prompting strategies and extraction methods minimize formatting bias.
B.2.1 INTEGRATION INTO EN-EVALUATION
Framework integration. We add MOLECULARIQ 1m. wwwallaaas//aeeeeeeeaaeaaea MUMARIII within the 1m-eva l i on-harnnes framework (Gao al., 2024). The dataset is hosted on the Hugging Face Hub and, upon publication, will be accessi- ble through the standard dataset loading interface.
YAML-configured task under
27

---

MOLECULARIQ task configuration.
task:
molecularig paaeculariigppss______ pass_at mass__t__
dataset_pathi molecularig
dataset_nad name: default
output_type: generate_until test_split: test
repeats:
process docs: _ocs:
!function task pask_processcrrc prrccescrr.roceese docs
doc_to to text: funcction task bask_processcr.o doc doc__ooott__cee_o to_text doc_to_target: target
_oc_tt_text::
process_rosults: !function
metric_list:
metric: avg_accuracy aggregation: mean tigher_is_beter ie_ttterr true
metric: estracted_answet aggregation: bypass higher_is_betten true
Generation parameters. To reflect realistic practitioner usage and model provider sug- gestions, we follow each model's native generation settings. For models that include generation.com contiin.confg..... json
json file, we adopt their specified parameters (temperature, top-p, top-k, etc.). Models without explicit generation configs fall back to backend defaults.
Flexible prompt and extraction strategies. Our architecture enables easy experimentation with different prompting and extraction approaches. Each model can have multiple configuration vari- ants to test different system prompts, extraction functions, or preprocessing strategies. We eval- uate models using standardized chemistry expert prompt by default (B.2.2), but additionally, practitioners provide model-specific prompts or alternative extraction methods, we test multiple configurations and report the best performance.
Model-specific processing pipeline. The evaluation system is model-agnostic, with processing functions defined per model configuration rather than at the task level. If none is provided, fallback to default implementations. An example skeleton config is presented below. For each model, we specify:
Controls how answers are extracted from model outputs. In Section B.2.3 we describe the standard extraction function we used for almost all models in this study.
inttractionffuncte
preprocessi in function: Optional model-specific input preprocessing (e.g., adding specific tags certain inputs like smiles) systemmprompt/im mlinnn nn.pppppt:: Model-appropriate instruction formats
API and local model support. The framework supports both local model inference and API-based evaluation through various backends. For API models, authentication and endpoint configuration are handled through the model configuration file, enabling seamless evaluation of commercial models alongside open-source alternatives.
28

---

Model configuration template.
Model configuration strusture (YAML)
model_name: smodel-identifie
modell: <framework>
e.g., vilm, openai, anthropic
chen_modellconfigge configs/<config-file) yaml model_args:
pretrained: -moell-path
tensor parallel size: <int>
gen kwargs:
temperature: <float>
0.0-1.0
0
top_p: <float>
0.0-1.0
top_k: <int>
top-k sampling
min_p: <float>
minimum probability
max_gen_toke <int>
generation tokens
until: []
stop sequences sampling enabled
do_sample: <bool>
system_ prompt: "xprappt-exxt
preprocessing function: <module>: <function>
extraction function: <module>: <function>
Multi-rollout evaluation. Stochastic variability is captured through the repeats:2 parameter, which automatically runs three independent generations per instance with different random seeds.
Metrics and aggregation. We report avg.accuracy the primary metric, computed as the mean score across the three rollouts for each instance. While the framework supports pass vass@k k evalu- ation (pass @ pass @ 3, etc.), we focus average accuracy for direct model comparison. Individual extracted answers are logged via the extracted answers metric for debugging and analysis.
Execution. Models are evaluated using wrapper that reads the model-specific configuration and constructs appropriate 1m-eval arguments:
MOLECULARIQ execution.
Evaluate local model
python evaluate_model.pp \
horriigg///morroonnnno-------2 yaml --task molccularigg_pas__uu
--model_conflle
--output_dir results/
Evaluate an API model (ensure API keys are set)
export OPENAI API BBA_AAAAKK""o"oo-oo------ python evaluate_ model.py
-modee_connig contigs/gpt-to-miil --task mwlecularrggppss_.t.. --output_dir results/
The wrapper automatically applies:
Model-specišic extraction and preprocessing functions
Appropriate system/inline prompts
Goncration parametors and rollout settings
API authentication and endpoint configuration
29

---

B.2.2 SYSTEM PROMPT
System prompt
You are an expert chemist. Answer molecular property, understanding, structural analysis and molecular generation questions precisely and accurately.
CRITICAL: Only content Sathiinccanse><//sswweee tags will be extracted. ALWAYS return JSON format.
KEY REQUIREMENT: Use EXACT key names from the question. Never modify or invent keys.
INDEXING: Atoms indexed from 0 to the end of the SMILES string from left to right. Only heavy atoms (skip [H], include [2H]/[3H])
Examples:
"CCOO" C(O), C(1), O(2)
"CCCCCOO"; C(O), C(1), C(2), 0(3)
""CC(=ONN"; C(O), C(1), 0(2), N(3)
ABSENT FEATURES: Use 0 for counts, [] for indices. Never null or omit.
ALWAYS USE JSON with EXACT keys from the question:
Single count (key from question: "aluoh:"_coumt)):
mnaswer>"alchhar count": 2</answer>>
<answer> "alcohol count": O</annser>> (if absent)
Single index (key from question: "ketone indicees):
"onswrr>"ketone indices": [5]</answer>> Canswer>""kttone "ittineiiddices"" /|<nseerr (if absent)
Multiple properties (keys from question: "ring count", "halogen indices")):
<answer>>"ing "ring count": 2, "halogen indices": [3, 7J<//nswer>>>
Canswer>"rrig_couu 0, "halogen_indices" indices": []<//nswer>> (if all absent)
Constraint generation <answer> "smiless: </anse"<//nswer>>>>>>>
Include ALL requested properties. Never null omit.
B.2.3 VERIFIERS AND ANSWER EXTRACTION
Design motivation. Requiring models to emit exact JSON schemas risks conflating chemical rea- soning capabilities with instruction- fowlltiionnooooiinnngg. proficiency. To isolate assessment chemical under- standing, our verifiers accept flexible output formats while maintaining semantic precision through key-specific matching. Specifically, we support JSON dictionaries, comma-separated values, var- ied bracket notations, and natural language property names (e.g., accepting both ("ring-count "; 3} and "number of rings: 3"), all normalized to canonical forms. This design is motivated by empirical evidence showing substantial variation
cetrucction-oollowinn across models (Zhou et al., 2023), and our observation that models may correctly identify chemical properties but express them in unexpected formats. By decoupling format compliance from chemical accuracy, we ensure that evaluation genuinely measures understanding of molecular properties rather than syntactic ad- herence, while our key-specific approach preserves the granular trackability necessary for detailed performance analysis.
Output parsing hierarchy. We extract answers using a four-tier cascade, applied in order until successful:
30

---

1. Explicit answer tags and JSON: If <answer>... //asswer> tags found and JSON-like object is present within them, we parse it as the answer.
2. Answer tags only: When only kanswer>... </anwwer tags exist (no JSON), apply lightweight parsing: split on common delimiters (commas, semicolons, line breaks), then coerce strings to appropriate types (numbers, booleans).
3. JSON without answer tags: Locate the final well-formed JSON object in the output. Apply repair heuristics: strip markdown code fences, fix trailing commas, normalize quote styles, then parse into structured data.
4. Raw text fallback: answer.
If everything else fails, we return the last continuous string as the
Content normalization.
All extracted values undergo standardization before comparison:
Text processing: Unicode normalization, case folding, whitespace compression, punctua- tion removal.
Type coercion: Convert numbers to appropriate type, standardize bracket notation [], map null/empty values to empty lists.
Value extraction and list normalization: Extract all values from JSON objects, discard- ing key names. Scalar values are wrapped in single-clement lists for uniform comparison (e.g., {a:1, b: [2, 31) becomes [ [[1], (1]3
Key-specific matching for performance trackability. To enable fine-grained analysis of model capabilities across varying task complexities, we employ key-specific matching that preserves property-level granularity. Given target dictionary T {(k] ) and predicted dictionary P = [Kk.. )=
we define correctness as:
correct(T, P) ]e [canon canon(kf)) A norm(r)) norm (1))
where canon(-) canonicalizes property names and norm(-) normalizes values. This formulation per- mits |P| 2 [T], though extraneous keys are ignored for scoring but kept for manual inspection; duplicate keys are resolved by retaining the final occurrence. An instance receives score of if all properties match, otherwise. This design crucially enables tracking of individual property per- formance within all the dimensions that MOLECULARIQ provides (complexity, task type, multitask load, etc.).
Task-specific verification. All tasks employ the key-specific matching framework with type- appropriate value comparisons:
Values must be integers or convertible to integers. For each canonicalized property key, we verify norm(v") = norm(ri (v') where both values are coerced to integers. Multi-count tasks require all properties in the target to have matching predictions.
Counting tasks:
Indexing tasks: Values are lists of atom indices. For each property, we compare sets of indices: set(rP) set(v"), where order within lists is ignored but property associations preserved (e.g., target { ""aromatic": [0, 23, "aliphatic": [1]} requires both properties match).
Molecular generation:
No predetermined exists. We extract the generated SMILES string from the predicted dictionary (searching for common key variations like smi les, molecule), then verify it satisfies all specified constraints through deterministic property validation (functional groups, molecular formula, ring counts, etc.). The score is binary: I if all constraints satisfied, 0 otherwise. Unlike prior work (Narayanan et al., 2025), we exclude purchasability requirements to ensure reproducibility and eliminate API depen- dencies.
31

---

Cross-complexity analysis benefits. The key-specific approach enables three critical analyses: (1) Subtask scaling tracking how individual property performance degrades as the total number of properties increases, (2) Cross-task
performance the same prop- erty across different task types (counting vs. indexing), and (3) Error porelltioo-iidentfying which property combinations prove partieularly challenging when evaluated together. This granular trackability transforms evaluation from binary pass/fail to actionable insights about which specific capabilities need improvement at each complexity level.
Reproducibility measures. We ensure reproducibility by pinning software versions (1m-eval 1 commit hashes, dataset versions), model configurations, and random seeds. Hardware-specific non- determinism may persist despite these controls.
32

---

C EXTENDED RESULTS
Here we provide the complete set of evaluation results for all models assessed on MOLECULARIQ, complementing the aggregate metrics reported in the main text. We present detailed breakdowns across the three benchmark dimensions-task category, multitask load, and molecular complexity-as well as analyses of feature families, SMILES formats, and representation variants. These results enable fine-gr -rainerandd inspection of the strengths and limitations of both generalist and chemistry- specialized LLMs.
C.I EXTENDED PERFORMANCE BREAKDOWN
This section reports complete performance profiles for all 38 evaluated models across the full MOLECULARIQ task suite. Whereas Section focuses on high-level trends and top- poppperfrmmin models, the tables and figures below provide model-by-model accuracy decomposed along task type (counting, indexing, constrained generation), multitask load, and molecular complexity. We addi- tionally include per-feature- results covering the six molecular-featte categories defined in Section 3.2.
C.I.1 OVERALL PERFORMANCE AND ACROSS BENCHMARK DIMENSIONS
Tables C1-C3 present the core performance breakdown across benchmark dimensions. Table C1 re- ports overall accuracy
the three task types for all evaluated models. Table C2 shows accuracy as function of multitask load, ranging from single-task queries to compositions of up to five simul- taneous subtasks. Table C3 compares performance across Bertz complexity bins. Together, these results reveal substantial variation in model capabilities: performance degrades with both compo- sitional complexity and molecular complexity for nearly all models, though the rate of degradation varies considerably.
For generation tasks, Bertz complexity cannot be computed directly since there no input molecule to measure. Instead, we assess task difficulty by the prevalence of molecules satisfying each con- straint set within our test pool. Figure CI plots the percentage of molecules fulfilling a given con- straint set against achieved model performance. A clear trend emerges: rarer constraints-those sat- isfied by fewer molecules in the pool-are substantially harder for models to fulfill, indicating that generation difficulty scales inversely with constraint prevalence.
Pavalance Rata Ch1
Figure C1: Model performance generation tasks as function of constraint prevalence. The x- axis shows the percentage of molecules in the test pool that satisfy given constraint set; the y-axis shows model accuracy. Rarer constraints (lower prevalence) consistently harder for models to fulfill.
33

---

Table CI: Results on the MOLECULARIO benchmark. We report overall accuracy as well as aceu- racy across task families (in %). Error bars indicate standard errors across instances. Model size refers to parameter count; is a flag for reasoning models.
Task family
Size
R
Overall
Counting
Indexing
Generation
Chemistry L.Mss
ChemLLM
7B
2.2 + 0.2
2.0 + 0.2
2,7 + 0.3
1.9 + 0.3
LlaSMol
7B
1.5 0.1
1.6
1,8 0.3
1.1 0.2
MolReasoner-Cap
7B
0.2
5.6
0.8 0.1
4.0
MolReasoner- Gen
7B
0.2
4.9
1.2
3.2
Llama-3 MolInst
8B
2.0
4.4
0.5
0.7
ChemDFM- 8B
8B
3.2
4.4
0.1
6.0
ChemDFM-13B
13B
2.2
4.1
1.2
2.7
ChemDFM-R-144
14B
0.7
1.89
2.8
10.5
TxGemma-
2.2
4.1
0.5
3.1
TxGemma-211
27B
0.2
7.0
0.3
EtherO
24B
6.5 0.3
3.2 0.3
0.1 0.0
17.5 0.8
Generalist LLMs
Gemma-2 9B
9B
+ 0.2
3.9
1.9 0.3
6.3 +
Gemma-2 2 27B
27B
6.9 0.3
6.6
0.2
13.9 0.8
Gemma-3 12B
12B
0.3
6.
2.8
8.4
Gemma-3 27B
27B
0.9
9.9
1.4033
1.87
LLaMA-2 13B
0.2
3.9
0.2
5.3
LLaMA-3 8B
8B
0.3
6.5
0.5
8.1
LLaMA-3.3 70B
70B
9.4
1..1
1.9
1..7
Mistral 7B v0.1
7B
1.4
2.2
1.1
1.2
Mistral-Son Small
24B
8.3
9.7
1.6
14.1
Nemotron-Nant Nano 9B v2
9B
1..1
1.22
6.5
14.8 0.8
SEED-OSS
24.3
2.9
0.9
20.0
Qwen-2.5 7B
7B
5.1
6.5
1.9
7.5
Qwen-2.5 14B
14B
6.6
7.6
1.3
11.5
Qwen-3 8B
8B
1..3
1.7
5.9
15.8 +
Qwen-3 14B
1..1
1.74
7.5
17.7 0.8
Qwen-3 32B
32B
15.6
17.0
9.6
21.0
Qwen-3 30B
30B (A3B)
2
23.0
16.5
29.4 1.0
Qwen-3 Next 80B
80B (A3B)
3..
30.7 1.0
2.9
Qwen-3 235B
235B (A22B)
0.6
37.1
3.05
46.7
GPT-OSS 20B (Low)
20B (A4B)
10.8
1.70
5.7
14.0 0.8
GPT-OSS 20B (Med)
20B (A4B)
27.2
27.9 1.9 0.9
32.1 2..2
GPT-OSS 20B (High)
20B (A4B)
+.6
39.1 1.0
32.8
35.2 1.0
GPT-OSS 120B (Low)
120B (ASB)
/
1..9
17.1 + 0.8
1..7
20.3 1 0.9
GPT-OSS 120B (Med)
120B (ASB)
26.5
29.1 0.9
2.83
28.0 18.0 1.0
GPT-OSS 120B (High)
120B (ASB)
47.5 0.6
46.8 1.1
42.5
53.7
GL.M-466
355B (A32B)
16.2
15.9
11.3
22.0 0.9
DeepSeck-R1-OSS
671B (A37B)
37.1 0.6
36.4 + 1.0
30.2 30,2 0.9
45.5 4 1.0
34

---

Table C2: Accuracy (%) on MOLECULARIQ by multitask load, the number of items requested per prompt (1,2,3,5). Error bars indicate standard errors across instances. Model size refers parameter count; R is a flag for reasoning models.
Multitask load
Size
R
1
2
5
Chemistry LLMss
ChemLLM
7B
2.8 0.2
0.3 = 0.2
0.1
0.0 0.0
LlaSMol
7B
2.8 2.8
0.2
0.0
0.0
MolReasoner-Cal
7B
6.4
1.3
0.3
0.1
MolReasoner-
7B
5.4
1.3
0.1
0.0
Llama-3 Mollnst
8B
3.3
0.2
0.1
0.0
ChemDFM
8B
7.4
1.0
0.1
0.1
ChemDFM-133
13B
0.3
0.5
0.3
0.0
ChemDFM-R-149
14B
16.4
3.6
1.2
0.3
TxGemma-99
5.1
0.2
0.0
0.0
TxGemma-277
27B
9.6
1.8
0.4
0.1
EtherO
24B
1..7
4.6
1.
0.3
Generalist LLMs
Gemma-2 9B
9B
7.0 0.4
0.7 0.2
0.1
0.0 0.0
Gemma-2 27B
27B
0.6
2.4
1.1
0.1
Gemma-3 12B
12B
1088
0.2
0.0
0.0
Gemma-3 27B
27B
1.
3.5
1.1
0.1
LLaMA-2 13B
13B
6.2
0.4
0.2
0.0
LLaMA-3 8B
8B
0.5
0.9
0.3
0.0
LLaMA-3.3 70B
70B
1.71
3.6
0.
0.6 061:3
Mistral-7B v0.1
7B
0.2
0.2
0.0
0.0
Mistral-Small
24B
1.79
3.5
1.0
0.2
Nemotron-Naan 9B v2
9B
2.77
5.0
0.9
0.5 0.2
SEED-OSS
36B
3.9
18.9 1.1
4.2 1.9
Qwen-2.5 7B
7B
8.9
1.6
0.3
0.1
Qwen-2. 2.5 14B
14B
1.7
0.5
0.8
0.4
Qwen-3 8B
8B
2.74
5.6
2.8
0.7
Qwen-3 14B
14B
2.86
6.7
3.0
1.3
Qwen-3 32B
32B
2.82
9.7
4.4
1.6
Qwen-3 30B
30B (A3B)
3..9
18.4 1.0
10.0 0.8
4.5 0.5
Qwen-3 Next
80B (A3B)
0.9
29.0 1.2
19.6
+ 0.8
Qwen-3 235B
235B (A22B)
0.9
36.8
13.1 2.2
GPT-OSS 20B (Low)
20B (A4B)
1.7
5.6
1.9
0.8 + 0.2
GPT-OSS (Med)
20B (A4B)
21.7 4..4
1..1
5.3 +
GPT-OSS 20B (High)
20B (A4B)
51.4 0.9
32.6 + 1.3
2.14
10.4 0.8
GPT-OSS 12BB (Low)
120B (ASB)
2..8
9.6 9.6:0.88 0.8
4.3
1.9 1 0.4
GPT-OSS 120B (Med)
120B (ASB)
4.81
21.4 1.1
12.9 0.9
6.0 1 0.6
GPT-OSS 120B (High)
120B (ASB)
6..5
46.7 + 1.4
36.8 1.4
21.2 1.2
GL.M4.66
355B (A32B)
2..6
10.8 + 0.8
4.4 0.5
2.3 0.4
DeepSeck-R110022
671B (A37B)
5.97
34.7 4: 1.3
22.7
12.0 0.8
35

---

Table C3: Accuracy (in %) on MOLECULARIQ by Bertz complexity bin. Error bars indicate stan- dard emrors across instances. Model size refers to parameter count; R is flag for reasoning models.
Size
R
Bertz Complexity
0-250
250-1000
1000+
Chemistry L.Ms
ChemLLM
7B
2,0 + 0.3
0.9
0.6 0.2
LlaSMol
7B
2.1 0.3
1.2 4 0.2
1.2 1 0.2
MolReasoner-Cap
7B
5.5
2.4
1.3
MolReasoner-Gei
7B
5.4
1.8 0.3
1.0 0.2
Llama-3 MolInst
8B
0.5
0.3
1.6 0.3
ChemDFM-8B
8B
0.5
0.1
1,5 0.3
ChemDFM-
3.2
0.3
1.3
ChemDFM-R-148
14B
1.99
7.2
4.3 0.6
TxGemma-999
9B
3.5
1.7 0.3
1.3 0.3
TxGemma 27BB
27B
0.2
3.2
2.7
EtherO
24B
0.3
1.3
1.1 0.2
Generalist LLMs
Gemma-2 9B
9B
3.0 1 0.4
2,2 02..0.3 0.3
1.5 1 0.2
Gemma-2 27B
27B
0.6
3.0
1.9 0.3
Gemma-3 12B
12B
6.7
0.4
2.0
Gemma-3 3 27B
27B
9.1
4.7
3.1
LLaMA-2 -2 13B
13B
3.4
1.3
1.3 0.3
LLaMA-3 8B
8B
5.2
2.4
1.710.
LLaMA-3.3 70B
70B
1.90
0.6
4.1 0.6
Mistral- 7B v0.1
7B
1.8
1.1
0.7
Mistral-Sman
9.
4.2
2.9
Nemotron-Nant Nano 9B v2
9B
1.99
0.6
4.5
SEED-OSS
36B
4..9
2..4
14.2
14.2 0.8
Qwen-2.5 7B
7B
5.6
2.4
1.7
Qwen-2.5 2.5 14B
14B
7.1
3.1
2.5
Qwen-3 8B
8B
17.
6.6
4.5
Qwen-3 14B
14B
19.7
8.4
4.9
Qwen-3 32B
32B
2..5+
1.73
6.6
Qwen-3 30B
30B (A3B)
34.9
15.0 0.9
9.5
Qwen-3 Next 80B
80B (A3B)
4.
1.1
1.220099
Qwen-3 235B
235B (A22B)
1.3
1.2
23.5
GPT-OSS 20B (Low)
20B (A4B)
16.0
0.6
4.9
GPT-OSS 20B (Med)
20B (A4B)
23.2 1.23
12.7 - 0.8
GPT-OSS 20B (High)
20B (A4B)
49.3
33.6 + 1.2
24.9 1.1
GPT-OSS 120B (Low)
120B (A5B)
1.1
10.8
6.2 0.6
GPT-OSS 120B (Med)
120B (A5B)
39,5 1.2
22.3 - 1.0
15.6 - 0,9
GPT-OSS 120B (High)
120B (A5B)
55.2
44.0 + 1.3
34.9 1.3
GL.M-466
355B (A32B)
22.8
10.6 + 0.7
6.9 + 0.6
DeepSeek-R1-0S3
671B (A37B)
V
50.9
28.5 + 1.1
20.5 1.0
36

---

C.1.2 PERFORMANCE ACROSS MOLECULAR FEATURES
Molecular feature groups. We analyze performance across the six molccular-featur categories defined in Section 3.2. Table C4 presents single-task accuracy aggregated across feature groups. Tables C5, C6, and C7 provide the corresponding task-specific breakdowns for counting, indexing, and generation tasks, respectively, enabling direct comparison of which feature families pose the greatest challenge for each task type.
Molecular features. Figures C2-C4 provide fine-grained performance breakdowns for individ- ual molecular features within each task category. Each heatmap displays the top-10 overall mod- els alongside the average across all models, with tasks ordered by difficulty (average accuracy). These visualizations reveal which specific molecular properties remain challenging even for top- performing models.
Table C4: Single-task accuracy (in %) on MOLECULARIQ by chemical feature category: graph topology (GT), chemistry-typed topology (CT), composition (C), chemical perception (CP), func- (SYN). Error bars indicate standard errors across
tional groups (FG), and synthesis/fragmentani instances. Model size refers to parameter count; R is a flag for reasoning models.
Chemical Features
SYN
Chemistry LLMs
ChemLLM
7B
1.9 1.9
0.5
0.9
2.2
4.8
5.5
LlaSMol
7B
2.5
4.0
0.7
1.9
5.2
4.0
MolReasoner-Cap
7B
5.6
0.7
5.0
4.
7.0
9.1
MolReasoner-GGar
7B
4.0
5.3
4.2
6.1
10.0
9.1
Llama-3
8B
3.4
5.1
22
37
1.6
2.2
ChemDFM-8B
8B
5.2
8.0
4.
5.3
3.4
19
ChemDFM-138
4.7
5.1
27
4.0
11.1
5.8
ChemDFM-R-
13
13
1.66
1.
25.6
24.1
TxGemma-9B
9B
5.0
0.6
2.
4.4
6.7
8.9
TxGemma-2271
27B
0.
11.00
6.1 0.9
92
16.3 233
15.2 2.0
EtherO
24B
9.8 1.2
11.8 0.9
144 1.5
7,3 1.2
14.1
15.9 2.1
Generalist LLMs
Gemma-2
9B
2.7
17.1
Gemma-2 27B
7.3
13.9
144
102
3.
23.8
Gemma-3 12B
7.3
11.7
8.7
92
148
19.8 2.5
Gemma- 3 27B
27B
10.4
1.2
128
10.0 1.5
18.5
28.0
L.aMA-2 13B
13B
6.0
5.3 1.3+:00 0.7
3.5
4.7 1.0
8.5 4
17.4
L 8B
88
7.3
9.0 088
7.9
8.6 1.3
11.9 + 3.2
18.9
LLaMA-3.3 70B
70B
14.5 14.51.6
18.9 + 1.3
16.9
10.6
24.8 4.5
23.1 2.7
Mistral-7B v0.1
7B
0.5
2.9
1.=
1.9
1.1 0.6
4.0 0.8
Mistral -mall
24B
11.3 2 1.3
17.2 - 1.1
15.0 1.4
10.8 1: 1.4
23.0 = 4.1
26.6 2.7
Nemotron Nano 9B
9B
15.1
1.4
20.4 +: 1.2
28.9 1.7
13.4 1.5
26.3 +: 4.0
26.3 2.7
SEED-OSS
36B
35.3
1.8
37.5
52.1 + 2.0
26.8 + 2.1
45.6 + 4.7
38.4 2.8
Qwen-2.5 7B
7B
6.8 + 1.0
9.. 0.9
6.4
7.6 +: 1.2
8.5 4: 2.9
19.5 2.4
Qwen-2.5 14B
14B
106 1.4
12.4 1.1
8.7
9.6 + 1.6
20.0 4.2
26.3
Qwen-3 8B
8B
14.6
19.5
25.8 1.7
14.9
26.7 4.0
30.2
Qwen 14B
14B
1844
23.3
29.1
17.2
30.4 4.3
30.8
Qwen-3 32B
32B
22.8
26.7 1.3
35.4
20.2
38.1 4.7
26.9 2.5
Qwen-3 30B
30B (A3B)
28.9
33.8 1.4
49.4
30.4 2.1
40.4 4.6
36.8
Qwen-3 Next 80B
80B (A3B)
43.7 2.0
44.2 1.5
6..8
36.1 2.2
46.7 4.7
44.2
Qwen-3 235B
235B (A22B)
46.7 2.0
$4.6
74.1
48.9
47.8
GPT-OSS 20B (Low)
20B (A4B)
23.2 1.3
17.0
25.1
GPT:OSS 20B
(A4B)
38.1
GPT:OSS 208 (High)
(A4B)
39.8
1.5
GPT-OSS (Low)
(ASB)
28.3 1.4
35.0
4.6
29.3
GPT-OSS 120B (Med)
120B (ASB)
3122 1.8
38.2 1.5
57.3
38.8
47.4 07.4-.777
39.6 2.8
GPT-OSS 120B (High)
120B (ASB)
2.1
59.7
1.5
83.04.
68.1211
= 4.8
47.6
GLM-4.6
35SB (A32B)
22.5
26.4 1: 1.3
35.6
20.8
38.1 444
31.4
DeepSeek R1-0528
671B (A37B)
47.2
49.4 +: 1.5
19.7.42..3
55.6 4.8
43.5
37

---

Table C5: Single-task counting accuracy (in %) on MOLECULARIQ by chemical feature category: graph topology (GT), chemistry-typet topology (CT), composition (C), chemical perception (CP), functional groups (FG), and synthesis/frammenttiion (SYN). Error bars indicate standard errors across instances. Model size refers to parameter count; R is flag for reasoning models.
Chemical Features
Size
R
GT
CT
C
CP
FG
SYN
Chemistry LLMy
CbemLLM
7B
3.1 + 0.9
0,9
11 0.6
2.8 0.9
4.4 1.6
1.1 0.8
LlaSMol
7B
3.3 + 0.8
4.1
=
28 1.2
4:
1.7
MolReasoner- Cap
7B
10.2 = 1.8
12.0
78 1.7
8.1
12.2 4.7
3.3
MolReasoner Gen
7B
7.2 2 1.4
9.8 + 1.3
7.6
10.0
10.0 4.3
4.4 1.7
Llama-3 3 MolInst
8B
7,2 1.5
12.1 + 1.6
4.8
8.6
4 4.6
3.3 1.7
CbemDFM-SB
8B
8.1
13.9 1
5.0 1.4
1.9
16.7
2.2
ChemDFM- 13B
13B
9.1 1.4
9.6 1.2
4.1 1.1
7.5 1.8
7.8
2.2
ChemDFM-R M-R-14B
14B
21.7 2.7
20.6 2.3
2.9
2..0
11.7 11.7
TxGemma 9B
9B
9.6 1.8
9.8 1.3
5.2
6.4 1.7
6.7 3.4
3.3
TxGemma 27B
27B
10.6
14.7 1
8.9
1
15.6 5.7
5.0
EtherO
6.9 1.5
11.0 1.3
2.8
1.9
0.0
1.1 0.8
Generalist LLMs
Gemma-2 9B
9B
8.1 1.4
7.5 = 1.0
5.6 1.2
7.8 1.6
6,7 3.7
4.4 1.9
Gemma-2 27B
27B
10.2
12.5 1.6
1..
1.5
8.9
5.0 1.9
3emma-3 12B
12B
1.2
16.4
9.4
11.7
18.9
4,4
3emma-3 27B
27B
2.6
2.19
144424
1.90
20.0
8.3
LLaMA-2
109 1.8
1.3
1.3
6.7
5.6
4,4
LLaMA-3
8B
10.4
14.0 1.6
1.
2.3
4.7
5.6
LLaMA-3.3 70B
2.8
2.30
15.6
16.1
2.66
6.1
Mistral-7B v0.1
7B
4.1
0.8
0.5
4.2
1.1 1.1
2.2
Mistral-Small
2.8
18.8
1..5
1.5
23.3
7.2
Nemotron-Naan
9B
1.9
32
37.3 2.4
3.0
3.78
18
Qwen-2.5
7B
13.7
8.9
1.4
11.1
2.2
Qwen-2.5 14B
2.70
9.4
10.8
16.7 16.7.69
3.3
Qwen-3
8B
21.5 2.
2.9
18.6
30.0 30.0722
11.1
Qwen-3 14B
14B
21.1
23.7 2.1
4.0
18.1 2.147
32.2 7.7
1000 28
Qwen-3 32B
32B
2.6
27.8 2.1
3.0
20.0
36.7 2 8.5
10.6 26
Qwen-3 30B
30B (A3B)
2.
2.3
5.443.1
2.
25.6 23.67J
22.2 4:
Qwen-3 Next 80B
80B (A33)
41.0 41.0 412
6.1
32.2 3.72
37.8 1.9
22.2 4.4
Qwen-3 235B
235B (A22B)
413
2.5
6..8
4..7.1.11
38.9 8.6
29.4 4,9
GPT:OSS 208 (1.ww
20B (A4B)
18.0 0.00220 244
26.5 2.1
20.0 + 2,5
21.9 + 3.2
26.7 + 6.9
6.7 + 1.9
GPT-OSS 20B (Med)
20B (A4B)
33.0 3.0
36.2 - 2.2
60.4 2.8
47.8 +
46.7 + 74
18.9 3.4
GPT-OSS 20B (High)
20B (A4B)
38.0
47.6 1.66:.1
76.5 2.6
58.1 3.6
53.3 = 8.7
21.1 11.1+40 4.3
GPT-OSS 120B (Low)
120B (A5B)
23.9
2.2
3.06
31.1 34
36.7 - 7.7
10,0 3.1
GPT-OSS 120B (Med)
120B (ASB)
33.3 330
2.3
57.8
3.6
53.3 8.5
22
GPT:OSS 120B (High)
120B (A5B)
41.7
55.1 +: 2.4
87.0 2.1
67.2
50.0 8.7
28.9
M-.6-4.6
355B (A32B)
23.9 2.7
24.7 1474:24
33.9
18.9
31.1
9.4
DeepSeck-R1-055
671B (A37B)
44.4
2.44
7.77
4.77
43.3
28.9
38

---

Table C6: Single-task indexing accuracy (in %) on MOLECULARIQ by chemical feature category: graph topology (GT), chemistry-typed topology (CT), composition (C), chemical perception (CP), functional groups (FG), and synthesis/frammenttiion (SYN). Error bars indicate standard errors across instances. Model size refers to parameter count; R is flag for reasoning models.
Chemical Features
Size
R
GT
CT
CP
FG
SYN
Chemistry LLMy
CbemLLM
7B
0.2 + 0.2
1.4
0.3 0.3
0.0 0.0
2.2
0.0
LlaSMol
7B
2.2 + 0.9
4,9 0,9
0.0
0.8 0.6
5.6
0.0
MolReasoner- Cap
7B
0.2 +: 0.2
1.3
0.8 0.6
0.3
0.0 0.0
0.6
MolReasoner Gen
7B
0.6 +: 0.4
1.2 0.4
1.7
1.1
2.2 2.2
0.0 0.0
Llama-3 3 MolInst
8B
0.0 + 0.0
0.2
0.0 0.0
0.0 0.0
1.1 1.1
0.0
CbemDFM-SB
8B
0.4
0.0
0.0
0.3
1.1
0.0
ChemDFM- 13B
13B
0.2
0.3
0.6
0.3
2.2
0.0
ChemDFM-R M-R-14B
14B
4.3
1.6
1.2
0.8
16.7 6.9
0.0
TxGemma 9B
9B
0.9 0.5
0.2
0.0
0.3
2.2 2.2
0.6 -
TxGemma 27B
27B
1.9
5.7
1.7 0.9
1.4
7.8 3.5
0.6 0.6
EtherO
0.0 0.0
0.1
0.3
0.0.1:0..0
0.0 0.0
Generalist LLMs
Gemma-2 9B
9B
0.2
1.4
1.4
0.3 = 0.3
7.8
0.6 0.6
Gemma-2 27B
27B
0.3
0.6
3.1
1.1
5.6
0.6
3emma-3 12B
12B
0.0
2.7
3.1
2.5
4.4
0.0
3emma-3 27B
27B
0.6
4.2
2.2
0.3
6.7
0.0
LLaMA-2
0.2
0.0
0.6
0.6
2.2
0.0
LLaMA-3
8B
0.2
0.3
2.
0.0
1.1
0.0
LLaMA-3.3 70B
1.1
1.1
5.
1.2
10.0
0.6
Mistral-7B v0.1
7B
0.5
0.4
0.0
0.0
0.0 0.0
0.0
Mistral-Small
1.8
4.2
2.2
5.0
0.0
Nemotron-Nor
9B
28.1
6.9
7.0
31.5
36.7
6..1
45.6 844
10.0
Qwen-2.5
7B
0.0
0.5
0.6
1.4
3.3
Qwen-2.5 14B
0.0
0.8
1.7
1..0
0.0
Qwen-3
8B
2.6
9.2
3..7
7.2
15.6
1.7
Qwen-3 14B
14B
7.
11.8 +: 1.6
28.1 3.3
114
22.2 7.0
1.7
Qwen-3 32B
32B
8.4
15.1 1.7
314 3.3
12.5 2.6
24.4
4
Qwen-3 30B
30B (A33)
163
25.9
40.8 4.6
23.6
3.7
722
Qwen-3 Next 80B
80B (A33)
33.7 3.2
36.3 2.4
63.9
33.6 3.7
36.7 1.9
20.0 1100::0 4.3
Qwen-3 235B
235B (A22B)
2.4 3.0
8.
40.3 3.9
2..1
GPT:OSS 208 (1.ww
20B (A4B)
4.3
13.7 +: 1.7
13.3 2.4 2.4
7.8 +: 1.9
13.3 + 5.2
0.0 + 0.0
GPT-OSS 20B (Med)
20B (A4B)
24.8 2 2.8
35.8 +: 2.2
56.1 3.3
36.7 3.7
41.1 + 8.3
11.7 11.73311
GPT-OSS 20B (High)
20B (A4B)
35.2
47.9 +
70.8 + 3.3
46.9
51.1 8.6
19.4 4.5
GPT-OSS 120B (Low)
120B (A5B)
9.1 19
19.6 0.6:200
3..
2.87
27.8 2 7.8
5.0 2.2
GPT-OSS 120B (Med)
120B (ASB)
33.0 2.6
66.4 4.4 3.4
33.9
3..0
1..4
GPT:OSS 120B (High)
120B (A5B)
44.8 3.4
37.9 +: 2.4
85.8 2.8
64.2 3.8
51.1 8.4
28.3
M-.6-4.6
355B (A32B)
10.4
1..4
38.6
10.8
18.9
4.4
DeepSeck-R1-055
671B (A37B)
38.1 3.0
2.3
3.11
38.9
40.0 - 8.2
17.2
39

---

Table C7: Single-task constrinne-ggnnratinnn accuracy (in %) on MOLECULARIQ by chemical fea- ture category: graph topology (GT), chemistry-typet topology (CT), composition (C), chemical perception (CP), functional groups (FG), and synthesis/fragmentttion (SYN). Error bars indicate standard errors across instances. Model size refers parameter count; R is flag for reasoning models.
Chemical
Size
GT
CT
C
CP
FG
SYN
Chemistry LLMs
ChemLLM
7B
2.9 + 1.2
=.=
4.2
3.5
11.2
LlaSMol
7B
1.3 +
1.
0.40.3
2.1
5.6
1.7 1.8
MolReasoner- Cap
7B
7.1 1.8
9.
49=12
6.7
8.
17.3 2:
MolReasoner Gen
7B
4.5 1.3
4.5
27
7.4
17.0 2.7
Llama-3 MolInst
8B
2.6
1.
0.8
211
0.0
2.
CbcmDFM-8B
8B
8.4
12.1
1.8
9.8
28.2 3..0
ChemDFM-IIP 13B
13B
5.8
5.1
2.
4.2
23.3
1122
ChemDFM-R--Bl M-B-14B
14B
1844288
20.9 3.1
148
17.9
4.9
40.0
45.2 4.9
TxGemma 9B
9B
4.2
7.0 1.4
2.9
7.0
11.14
17.0 28
TxGemma-227B
27B
9.7
13.7 2.1
6.2
2.66 2.66
19.5338
EtherO
24B
32.0 4.0
3.1
3.59
2.66
4.22
33.7 = 377
Generalist LLMs
Gemma-2 9B
9B
8.7 2.1
10.7 = 1.9
10.1 2.0
10.5 + 2.6
17.8 = 5.7
34.0 3.8
Gemma-2 27B
27B
14.6
3.3
2.
18.6
33.3 7.5
48.1 4.2
Gemma-3 12B
12B
1.0
20.9 2.8
12.1
144
21.1
40.1 4.4
Gemma-3 3 27B
27B
16.5 - 3.6
33.5 1.54:30
189
4:5
28.9 79
55.4
4.7
LLaMA-2 13B
13B
7
8.1 1.9
5.6 1.5
7.4
17.8 6.9
34,9 4.1
LLaMA-3 8B
8B
3.1
16.0
9.7 1.9
16.8 3.3
278
37.5 4.3
LLaMA-3.3 70B
70B
28.8 2.3
36.7 3.3
2..
15.1 3.4
42.2
45.8 4.7
Mistral-7B v0.1
7B
2.6 1.2
1.7 17:20.0 0.6
2.3 = 0.8
0.9
2.2 1.5
7.4 1.6
Mistral-Small
24B
22.3
39.0 4:
2.
1..3
36.7
7.9
53.2
Nemotron-Nand
9B
3.7
37,5 3.2
3.00
16.5 2.9
25.6
6.9
51.0
SEED-OSS
36B
4.0
3.1
3.27
3.3
47.8
66.0
Qwen-2.5
7B
15.6 3.06=300
8.0
2.4
5.6
40.7
Qwen-2.5 14B
21.424.1
2.39
13.6
1.09
33.3
54.8
Qwen-3
Qwen-3
32.7
7.6
Qwen-3
43.7
53.3
Qwen-3
(A3B)
42
5..5
3.3
4.0
61.1 8.0
62.2
Qwen-3
(A3B)
6.14
65.23.3
63.4
44. 4.2
65.6
7.6 7.6
70.8
Qwen-3 235B
235B (A22B)
65.7
77.6 1: 2.5
73.0 4: 3.0
64.9 +: 4.2
74.4
7.1
70.8 3.6
GPT-OSS 20B (Low)
20B (A4B)
23.0 3.7
34.8 +: 3.2
17.9 4: 2.7
22.5 +: 3.5
42.2
7.1
50.3
GPT-OSS 20B (Med)
20B (A4B)
45.6 4.1
52.0 +: 3.2
54.1 4: 3.2
48.8 4: 3.6
60.0 + 6.7
64.4 3.7
GPT-OSS 20B (High)
20B (A4B)
51.1 51.1-22 42
55.4 3.2
59.7 4 3.0
55.1 3.9
70.0 - 7.0
64.1 3.9
GPT-OSS 120B (Low)
120B (ASB)
31.7 4.17
46.9 3.5
40.9 3.3
25.6 3.7
$4.4
54.5 4.6
GPT-OSS 120B (Med)
120B (A5B)
40.5 = 4.2
54.8 3.2
50.0 + 3.1
36.5 1: 3.6
58.9 = 7.4
64.1 4.1
GPT-OSS 120B (High)
120B (A5B)
53.4 + 4.5
71.6 2.8
76.3 2.5
74.0 + 3.7
82.2 +: 6.3
69.6 4.2
G1.6-466
355B (A32B)
41.1 4.3
48.4 3.2
35.4 $5.4..35 3.2
35.8 + 3.8
64.4 + 7.0
59.6 4.0
DeepSeek-R R1-0228
671B (A37B)
67.6 3,9
72.1
70.2 3.0
64.6
83.3 -: 5.5
67.0 3.9
40

---

Count per-property performance
Steeecchemeetty Lispecttee
80-
Actureyy
40
100
Teetaro Aboms
Carbon Aloms
Saturaced Rings
115A
Aotumiy (N)
20
22 Stareashemitry Dousle
80
Accussc
40
20-
100
Funciional Groups
Rings
Oxidation State
Chain Temnin
80
Accuraty (60
60
RIS Stemooonters
Carbon Chain
Alphaticc Rings
Astursay
00
20
Bridgchozd Atoms
Sterepcenters
Murdo Scaffald
Unspeciled ed Stereooonter
Accumaay (14
80
80
40
Smallestt angest Siee
Molecuaar Formula
Rotorable
80
Actunio
20-
Rewnch
BRI'' Decompootiion
Accursty 7641
80
40
GPT-OSS 1208 (High)
DeeepSeek-R1-5522
Owen-3 Next 808 GPT-055 (Med)
GPT-OSS 1200 (Med) Owen:3 308
(ell models!
Owen-3
GPT-OSS 208 (High)
SFFD-OSS
Figure C2: Per-feature performance on counting tasks. Models (rows) are ordered by overall per- formance; tasks (columns) are ordered by average accuracy. The bottom row shows the mean across all models.
41

---

per-property performance
Sicreschenistey Unspodifed Double Bond
100
Ho ogen Alom
Heavy Ajoms
Hetero Aloms
Accumity NN6)
80
60-
800
Acorsey
40
100
Fuperianal Groups
Alphaatic Sings
Ocidation State
Acursty (K
50
Hetersoycles
Chen
Accurscy (16)
80-
40
Sterocceempryy Double Bend
Cerbon Chein
Smallegt Ring
Acc-ueey
80
70
Murcoo Scatfild
Unspectfer ed Ste mocember
Steveocenters
RIS Stemonenters
00
Acurso (56)
Retorable
80
Accursey
60
40
GPT-OSS 1208 (High)
DeepScck-RL R1- 05-8
Owen-3 3 Next 80B GPT:0S5 (Med)
GPT-OSS 1208 (Med)
Owen:3 30B
Avg (all models)
GPT-05S 206 (High)
SEED-OSS
GLM-4.6
Figure C3: Per-fcature performance on indexing tasks. Models (rows) are ordered by overall perfor- mance; tasks (columns) are ordered by average accuracy. The bottom row shows the mean across all models.
42

---

Generation per-property performance
Sherecheenteryy Linsperifed
langeet
80
Actureyy
40
100
Aecmattcc Bings
Murdo Scattsld
Saturaced Rings
Alphatic Rigs
Aotuniy (N)
80-
40
Accuscy 190)
80
40
100
Choin Temirs
Oxidation State
Ho og Abom
Heterooyc es
Accuracy
60
HBDs
Hetero
Stereocenters
Ascursty
20
HSAS
Fused Kings
Hudi ogon Atems
Branch Points
Accumsoy (14)
80
40
Sterochemiitry Double
Unspectfed Stemeoenter
Retocable Bonds
RIS Stemocentors
Actunio
80
20
Brtdgpheeed Atrere
Medeeniar Sermais
Decompostionn
Accurstt 2641
80
40
GPT-OSS 1208 (High)
DeepSeek-RL R1-022
Owen-3 Next 808 GPT-055 (Med)
GPT-OSS 1200 (Med)
Owen-3 308
(ell models!
Owen-3
GPT-OSS 208 (High)
SEFD-OSS
Models (rows) are ordered by overall performance; tasks (columns) are ordered by average accuracy. The bottom row shows the mean across all models.
Figure C4: Per-feature performance on generation tasks.
43

---

C.1.3 SENSITIVITY TO SMILES REPRESENTATION
To assess how strongly models rely on specific canonicalization conventions rather than learning representatiooniivaaain molecular reasoning, we compare performance across three SMILES vari- ants: canonical, randomized, and Kekulized. Tables C8 and C9 report numerical results for counting and indexing tasks, respectively. Table C10 further examines robustness to alternative ring-closure numbering schemes, where we modify ring-closure indices to random values between and 100 while preserving molecular identity.
Figures C5 and C6 visualize these degradation patterns. both figures, points below the diagonal indicate performance degradation under augmentation. Randomized and Kekulized variants con- sistently reduce accuracy, and systematic degradation under ring enumeration indicates that models often memorize syntactic patterns rather than performing reasoning over the underlying molecular graph. These results are further discussed in the context failure modes in Section C.2.
Table C8: Counting accuracy (in %) on MOLECULARIO under different SMILES representations. Error bars indicate standard errors across instances. Relative changes from canonical aromatic are shown as colored subscripts. Model size refers to parameter count; R is flag for reasoning models.
Counting
Size
R
Canonical
Canonical (kek.)
Randomized
Randomized (kek.)
Chemistry LLMs
ChemLLM
7B
2.0 + 0.4
2.1 + 0.7.ss
2.1 + 0.4
1.6 + 0.5 0.5.av
LlaSMol
1.3
1.4 0.4.rs
0.3.ars
3.0 0.8.
MolReewoner--ag
7B v
5.6
3.2 1.0.9s
6.2 2.2:: 0.7.11
6.7 1: 1.5.
MolReasoner-Gen
7B
v
5.0
2.6 0.8 0.8.
5.5 0.7.as
5.2 + 1.2,.s
Llama-3 Mollnst
8B
3.7
4,4 + 1.2,19999
5.0 0.7.1..
6.5 1.3.pes
ChemDFM-8B
0.7
5.1 1.2
3.7 1.1
ChemDFM-
13B
3.4
+ 1.0,77%
4.4 0.6,
1.1 1.1
ChemDFM-R--E
14B
12.8
1.3
12.2 + 2.2.9%
14.0 1.3
10.0 1 2.1 2226
TxGemma-99 9B
9B
2.5
4,5 1 1e9r
5.3 5.3:10 0.7
4.3 4.3 1.2.,0%
TxGemma-27B
27B
7.2
7.5
1.4.4
6.9 1
6.2 1.3 148
24B
3.5
+ 1.0
+ 0.5 11%
1.0
Generolist I.
Gemma-2 9B
9B
3.4 0.5
3.2 + 0.8
4.4 + 0.5
4.1 + 1.0.229
Gemma-2 27B
27B
6.3
3.6 : 1.0
8.0 : 0.9
6.0 1: 1,3
Gemma-3 12B
12B
4.8
4.7 1.3
8.0 1.0,ar
7.3 1.6.sis
Gemma-3 27B
9.1
1.6
1.1-209
9.2 1: 1.8.ps
LLaMA-2 13B
13B
4.1
4.4 1.2
3.7 0.5 11.
4.0 :1: 1.1
LLaMA-3 8B
8B
6.1
4.5 = 1.1
6.2 2 0.8,
6.7 4 1.4.
LLLMA-3.3 70B
70B
9.2
14.3 + 2.3
11.6 1.2
11.7 + 2.2,
Mistral-7B v0.1
2.4
0.5.3
1.7 0.3
0.7
Mistral-Small
24B
9.1
7.5 + 1.4.7%
10.6 1.0.11%
10.8 1.8.,95
Nemotron-Nano 9B v2
9B
12.7 -
9.9 = 1.7.22
13.2 1.1.45
9.7 + 1.7.wx
SEED-OSS
36B
/
2.5
20.7 + 2.3
29.9 1.5
21.1 2.5
Qwen-2.5 2.5 7B
5.5
4.8
12%
6.7 0.8 122%
6.8 1.5 259
Qwen-2.5 14B
8.0
= 1.7.16
8.2 1.0,25
5.2 + 1.5 35%
Qwen-3 8B
8B
1..1
8.9 = 1.7.22%
13.4 1.1
11.6 = 1.9 116
Qwen-3 14B
14B
14.9 1.2
11.3 +: 1.8
14.9 +: 1.2
14.1 + 2.2.5s
Qwen-3 32B
32B
17.8
1.3
14.6 2.0.
17.8
14.6 2.1
Qwen-3 30B
(A3B)
24.8 1.5
17.3 2.2. 309
24.5 1.4.s
17.9 2.2-ams
Qwen-3 Next 80B
80B (A3B)
V
33.5
1.6
28.5 2.7.195
31.4 1.5
21.7 2.4
Qwen-3 235B
235B (A22B)
V
1.7
36.5 + 2.8
39.2 1.6,3a
27.5 + 2.7
GPT-OSS 20B (Low)
20B (A4B)
V
13.9
11.9 + 1.9.199
12.8 + 1.0
12.5 2.1 Jos
GPT-OSS 20B (Med)
20B (A4B)
31.0
21.8 2.3 30%
29.7
18.6
GPT-OSS 20B (High)
20B (A4B)
V 41.8 1.7
32.1 2.7 2094
41.3 + 1.6.1%
30.3 2.7
GPT-OSS 120B (Low)
120B (ASB)
V 1.3
12.0 - 1.8.1.1..0 35%
18.0 + 1.2.a%
14.4 2.2
GPT-OSS 120B (Med)
120B (A5B)
V 3.61
23.1 +: 2.5
30.1 + 1.4.ps
19.8 1: 2.3 Joss
GPT-OSS 120B (High)
120B (ASB)
49.5
42.5 3.0
47.2 1.7 .m
41.7 11
GLM-466
355B (A32B)
15.9 1.2
15.6 + 2.0 29
16.2 1.1
15.1 + 2.1 ss
SeepSeck-R1-052
671B (A37B)
V 37.6 1.7
33.6 109
38.5 1 1.6
28.7 1 2.6 23%
44

---

Table C9: Indexing accuracy (in %) MOLECULARIQ under different SMILES representations. Error bars indicate standard emrors across instances. Relative changes from canonical aromatic are shown colored subscripts. Model size refers parameter count; R is flag for reasoning models.
Indexing
Size
Canonical
Canonical (kek.)
Randomized
Randomized (kek.)
Chemistry LLMs
C.
7B
0.1 + 0.1
0.3 + 0.2.
0.5 + 0.2.
0.7 + 0.4.
LlaSMol
7B
1.0
3.7 + 1.2.:s
2.2 # 0.5.1144
1.3 = 0.6.2y
MolReasoner-Cap
0.6 0.2
0.5 = 0.3.
+ 0.3
0.3 0.3
MolReasoner-Gen
1.3
+ 0,6
0.4,37%
+ 0.4.35%
Llama-3 MolInst
8B
0.1
0.7 # v.5
0.6 - 0.2
0.7 = 0.4.ops
ChemDDM-8B
8B
0.0
0.2 1 0.2 0.3 L 0.2
0.1 1 0,1
0.0 1 0.0
ChemDDM-133
13B
0.2
0.3 13.1.00 0.1,
0.0 0.0 pooss
ChemDDM-R
14B
0.6
1.1
0.7
1 220
TxGemma-90
9B
0.2
0,7 1 0.4,509
0.5 0.5 0.2
0.3 1 0.2 201
A.Gemma-27B
27B
1.2
1,5 1 0.7
2.3 0.5
1.8 0.8
EtherO
24B
0.0
0.0 + 0.0
0.1 0.1
0.0 + 0.0
Generalist LLMs
Gemma-2
1.3
1.24
0.5.
1.3 0.6.
Gemma-2 27B
27B
0.3
0.7 + 0.5.18
1.1 0.3.
0.5 : 0.4.
3emma-3 12B
12B
0.8
1.7 1: 0.9.1119
0.9 0.3
0.0 1: 0.0 moxs
Gemma-3 27B
27B
1.3
0.8 = 0.6
1.6 0.4.
0.2 = 0.2
LLaMA-2 13B
0.1
0.0 0.O norr
0.2 0.
0.0 0.0 moss
LLaMA-3
0.2
.39%
0.2 0.1
+ 0.2.33%
LLaMA-3.3 70B
70B
1.4
1,5 = 0.9.s%
2.2 0.6.,5m
0.5 + 0.5
Mistral-7B v0.1
7B
0.1
0.8 1 0.4.
0.3 0.1
0.2 + 0.2
Mistral-Small
24B
1.4
2.0 1 0.8
0.5
0.2 1: 0.2
Nemotron-Nano 9B v2
0.
4.4 1.3 40%
8.2 0.9
1.7 0.7 .798
SEED-OSS
36B
/
28.0
14.8 2,2 479
29.8 1.5
14.1 2.1 50%
Qwen-2.5 7B
7B
1.4
4.7 +: 1.4
1.5 0.4.
2.2 1 0.9 0.99.
Qwen-2.5 14B
14B
1.4
3.0 1.2, 1909
1.2 0.4,
0.0 + 0.0 poo6s
Qwen-3
6.6
1.3
7.0 0.8
0.6.49
Qwen-3 14B
14B
8.3
4.2 + 1.1.
9.0 + 1.0.es.
2.8 1: 0.8
Qwen-3 32B
32B
9.3
5.9 + 1.4.ys
11.4 + 1.1
4.1 +: 1.1.195
Qwen-3 30B
30B (A3B)
/
1.3
8.9 = 1.7
18.4 + 1.3
7.1 + 1.5
Qwen-3 Next 80B
80B (A3B)
31.4
15.7 2.3 .008
28.8 1.5 ar.
16.9 anis
Qwen-3 235B
235B (A22B)
38.5 1.6
25.6 2.6.33%
37.5 + 1.6.2%
20.4 2.5
GPT-OSS 20B (Low)
20B (A4B)
V 5.8
5.9 + 1.5.1%
5.8 + 0.8
3.0 + 1.0.aps
GPT-OSS 20B (Med)
20B (A4B)
V 26.7 1
13.0 1: 2.1
23.5 1: 1.4.
11.6 1.9.5%
GPT-OSS 20B (Hlggh)
20B (A4B)
37.4 1.6
25.6 2.7 .216
34.5 +: 1.6.au
18.4 2.4
GPT-OSS 120B (Low)
120B (ASB)
V
12.0
8.4
30%
11.6 1.1 1.1.
5.6 1 1.5 539
GPT-OSS 120B (Med)
120B (ASB)
V 27.115
13.3 2.0
23.6 11.6-10 1.4.68
10.3 1.8 62%
GPT-OSS 120B (High)
120B (ASB)
V
47.2 1.8
36.5 + 3.1
43,6 + 1.7
28.9 2.8
GL.M-.66
35SB (A32B)
11.9 + 1.0
7.6 + 1.5 37%
13.3 + 1.1 1.1.119
6.0 + 1.1 so%
DeepSeek-RI R1-0028
671B (A37B)
34.6 1.6
18.9 2.3
33.8
14.4 2.0.se%
45

---

Table C10: Counting and indexing accuracy (in %) MOLECULARIQ under different Ring Enu- merations. Error bars indicate standard errors across instances. Relative changes from original ring enumeration shown colored subscripts. Model size refers parameter count; R is flag for reasoning models.
Counting
Indexing
Model
Size
R
Original
Ring Enum.
Original
Ring Enum.
Chemistry LLMs
ThemDFM-R-144
14B
v
12.3 1 0.8
10.4 0.8. 15%
2.8 1 0.4
2.1 +L 0.4.25
TxGemma- -7B
0.5
5.5 0.5. .15
1.6 0.3
+ 0.3.0s
Generalist I.MMs
Gemma-3 12B
12B
6.2 + 0.6
5.7 + 0.5 8%
2.8 + 0.4
0.3 + 0.1
Mistral-Small
24B
9.3 + 0.6
7.8 + 0.7
1.2
0.6 + 0.2
Nemotron-Nano 9B
0.7
8.3 0.6.24%
0.5
3.8 0.4.23%
Qwen-3 8B
8B
V
0.7
8.4 0.6.258
0.5
2.7 1 0.4
Qwen-3 14B
14B
V 1.8
11.0 + 0.7-13%
0.5
5.4 3.4:1 0.5 7%
Qwen-3 30B
(A3B)
V 0.9
17.3 1.0.15%
13.2
13.0 + 0.9.29 2%%
Qwen-3 235B
235B (A22B)
V 1.1
33.0 + 1.1
31.5 1.1
30.1 + 1.1
GPT-OSS 20B (Low)
20B (A4B)
V 1.7
8,2 1 0.7.349
4.5
3.4 1 0.5. .55.
GPT-OSS 20B (Med)
20B (A4B)
V 0.9
23.5 + 1.0.ss
19.5
19.2 1 0.9.1
GPT-OSS 20B (High)
(A4B)
1.1
19.5 0.9 48%
3..5
14.1 0.7 54%
GPT-OSS 120B (Low)
120B (A5B)
v 0.8
10.5
0.8
9.7
6.3 = 0.6
GPT-OSS 120B (Med)
120B (ASB)
V
26.1
19.8
1.0.
20.0
14.3 0.8 29%
GPT-OSS 120B (Highh
120B (ASB)
43.7 1.2
1.1.s5
39.9 1.2
35.8 1.1.10%
Counting
Indexing
0r-000
GF 400 light
Stepltal 0128
DespBtcc
69,995 D001
14.
Figure C5: Impact SMILES representation on counting (left) and indexing (right) tasks. Each point compares accuracy under canonical aromatic SMILES (x-axis) to accuracy under augmented variants. Points below the diagonal indicate degradation under augmentation.
46

---

Figure C6: Effect of ring-closure enumeration counting and indexing tasks. Accuracy with original SMILES (x-axis) is compared to accuracy using enumerated ring indices (y-axis). Points below the diagonal indicate degradation.
47

---

C.1.4 RESPONSE LENGTH ANALYSIS
Figure C7 plots overall accuracy function of average response length across models, examin- ing whether verbose chain-of-though reasoning correlates with improved performance. Figure C8 provides the complementary view, showing response length distributions conditioned on accuracy level for the top-10 models. A consistent pattern emerges: failed responses (0% accuracy) exhibit substantially longer token lengths than successful ones (100% accuracy). This suggests that exces- sive verbosity may reflect unproductive reasoning-models generate longer traces when struggling rather than reasoning more effectively. The pattern holds across reasoning models of different sizes, indicating that this failure mode is architecture-indepenie
100
Non-messoning
228
1228
2350
ami-bbs 208 (High)
a0k
ChemLLLM Lis5Msl
ChembRM TxGemma
Gemma LLAA
Mistra Nemotron
Owen 2.5
Gen-065 1208
Owen
GLM-4.6
Gemma
LMMA 3.3
Figure C7: Overall accuracy as a function of average response length across all evaluated models.
Tolaos Length Dictrination by Accurasy (per Madict)
20.
Figure C8: Response length distributions conditioned accuracy level for the top- models. Each box shows the distribution of average token lengths for responses achieving given accuracy (0%, 33%, 67%, 100% across rollouts). Failed responses (0% accuracy, red) consistently exhibit longer token lengths than successful ones (100% accuracy, green) across the top- 10 performing models.
48

---

C.1.5 ROLLOUT ABLATION
Table CII compares performance between standard evaluation rollouts) and extended evaluation (8 rollouts) for a subsection of models, revealing overall consistency.
Table C11: Results on the MOLECULARIQ benchmark comparing 3 vs 8 rollouts. We report overall accuracy as well as accuracy across task families (in %). Error bars indicate standard emros across instances. Relative changes from to 8 rollouts are shown as colored subscripts.
Model
Size
Overall
Counting
Indexing
Generation
8
Chemsiary LLMs
ChemDFM & 14B
14B
87 1.0.4
86 0.4.u
12. 4 088
129 d: 0.8.x
28 1.04
26 0.4as
10.5 4 08
10.3 d: 0.8.s
Gonerelist LLMs
Nemotron-Niat Nino 98 v2
9B
11.1 0.4
11.1 + 0.4
12:2 + 0.7
12.0 + 06
6.6 0.5
6.4 0.5
14.8 +
15.1 = 0.8.s
308
30B (A38)
22.5
23.3
16.5 08
15.4
Qwen-3 Nest 80B
SOB (A3B)
32.3 0.6
30.5 0.5%
30.7 1.0
29.3
09
26.9.000
25.4 09
40.0 1.1
37,5
1.0c
(A22B)
10.0
GPT-OSS 120 (Low)
120B (A5B)
0.5
16.1 05.
17.
178 09
10.7 1:56
1044 07.
20.3 1:09
20.5
1.0,1
120 (Med)
0...6
09
09.s
1.0.s
GPT-DSS 1208 (High)
120B (ASB)
47.5 0.6
45.4 : 0.6
46.8
1.1
45.4
L.
42.5
4: 11
41.3 + 11.s
53.7 1.1
49.8
1.0-
1.1.6 CHEMISTRY LLMS AND BASE MODEL COMPARISON
Table C12 compares
L.MSs against their corresponding base models, quanti- fying the impact of domain-specing fine-tuning on molecular reasoning.
Table C12: Comparison of chemistry LLMs to their base models. Reported values are overall accuracy (in %) with standard errors on MOLECULARIQ.
Chemistry LLMs
Base Model
LlaSMol
1.5 - 0.1
1.4 0.1
Mistral-7B v0.1
MolReasoner-Cai
0.2
5.1 0.3
Qwen-2.5 7B
MolReasoner-
0.2
5.1
Qwen-2.5 7B
Llama-3 MolInst
0.2
4.8 1 0.3
LLaMA-3 8B
ChemDFM-SP
0.2
4.8 0.3
LLaMA-3 8B
ChemDFM- 13B
2.2
3.1 0.2
LLaMA-2 13B
ChemDFM-
8.7
6.6 0.3
Qwen-2.5
TxGemma 9B
0.6
4.0 0.2
Gemma-2 9B
TxGemma-27B
0.2
6.9
+ 0.3
Gemma-2 27B
EtherO
6.3
8.3 0.3
Mistral-Small
C.1.7 OUTPUT TYPE VALIDITY
Table C13 reports type validity-the the fraction of outputs satisfying expected syntactic constraints for each task category (integers for counting, integer lists for indexing, valid SMILES for genera- tion). Type validity provides an upper bound on achievable accuracy, as malformed outputs cannot be correct; the relationship between type validity and actual accuracy is further analyzed in Fig- ure C13. Table C11 compares performance between standard evaluation (3 rollouts) and extended evaluation (8 rollouts) for a subsection of models, revealing overall consistency.
49

---

Table C13: Type-validity results on the MOLECULARIQ benchmark. We report overall type-valid rate well rates across task families (in %). Error bars indicate standard errors across instances. Model size refers to parameter count; is a flag for reasoning models.
Task family
Size
R
Overall
Counting
Indexing
Generation
Chemistry L.Mss
ChemLLM
7B
33.8 + 0.5
55.6 = 0.8
18,8 + 0.6
25.6 + 0.7
LlaSMol
7B
28.7 0,4
35.1 0.7
25.8
24.7 + 0.7
MolReasoner-Cap
7B
61.1 0.6
72.8 0.9
7..5 0.8
30.6 1 0.8
MolReasoner- Geen
7B
53.1
61.1 0.9
69.8
25.3 1 0.7
Llama-3 MolInst
8B
58.4 0.6
0.5
0.
1.4
ChemDFM- 8B
8B
7..0
7.88
6..8
71.1
ChemDFM-13B
13B
0.5
6.85
24.1 4.9
ChemDFM-R-144
14B
72.6
0.80
1.1
53.9
TxGemma-
6.77
27.2 7.7 7.7
TxGemma-211
27B
87.6
87.7
0.7
89.3
85.6 0.5
EtherO
24B
27.4 0.6
9.0 + 0.5
0.0 + 0.0
78.8 + 0.7
Generalist LLMs
Gemma-2 9B
9B
56.1 0.5
60.6 0.8
0.7
53.3 0.8
Gemma-2 2 27B
27B
91.6 0.3
93.1 0.6
9.3
84.3 0.7
Gemma-3 12B
12B
32.7
1.2
18
31.6 1.0
Gemma-3 27B
27B
71.5
8.9
5.
69.2
LLaMA-2 13B
84.5
0.5
8.6
71.5
LLaMA-3 8B
8B
80.7 0.4
9.6
0.7
65.5
LLaMA-3.3 70B
70B
0.4
9.66
9.5
81.0
Mistral 7B v0.1
7B
37.6
0.7
0.7
0.5
Mistral-Son Small
24B
90.2
0.6
0.
82.8 0.7
Nemotron-Nant Nano 9B v2
9B
75.8
0.64
0.4
40.7 0.9
SEED-OSS
74.4
88.9
9.56
37.6 1.0
Qwen-2.5 7B
7B
73.6
93.4
65.7 1.0
59.5 0.9
Qwen-2.5 14B
14B
75.6
0.7
6.2
75.6 1.1
Qwen-3 8B
8B
75.0
83.7 = 0.7
0.5
48.1 1 1.0
Qwen-3 14B
14B
0.5
90.5 0.6
0.5
54.1 0.9
Qwen-3 32B
32B
8.44
90.6 + 0.6
91.5 0.5
63.1 0.9
Qwen-3 30B
30B (A3B)
8
92.5 + 0.6
97.9 0.3
74.6 0.8
Qwen-3 Next 80B
80B (A3B)
89.3
90.9 + 0.6
94.9
81.2 1. 0.7
Qwen-3 235B
235B (A22B)
92.7 0.3
93.1
98.6 0.2
85.8 0.6
GPT-OSS 20B (Low)
20B (A4B)
81.9 0.4
93.6 33.600 0.5
89.0 0.5
60.7 0.9
GPT-OSS 20B (Med)
20B (A4B)
83.0
88.9 + 0.6
93.5 0.4
64.5 + 0.9
GPT-OSS 20B (High)
20B (A4B)
75.0 0.5
76.9 0.9
86.0
0.6
60.7 1.0
GPT-OSS 120B (Low)
120B (ASB)
V
1.5
93.6 + 0.5
97.6 17.6.000 0.2
56.3 1 1.0
GPT-OSS 120B (Med)
120B (ASB)
86.5 1 0.4
92.6 - 0.6
98.2 18.20... 0.3
66.6 1 0.9
GPT-OSS 120B (High)
120B (ASB)
86.1
84.8 0.8
0.56
79.4 0.8
GL.M-466
355B (A32B)
70.0 0.5
73.9 + 0.8
81.8 + 0.6
52.4 1.0
DeepSeck-R1-OSS
671B (A37B)
89.7 0.3
89.8 + 0.6
94.3 4 0.4
84.5 4 0.6
50

---

C.2 FAILURE MODE ANALYSIS
Despite evaluating MOLECULARIQ across a diverse set of LLMs, substantial room for improvement remains. This section systematically analyzes failure patterns to identify systematic performance gaps and inform future model development.
C.2.1 OVERVIEW SYSTEMATIC FAILURES
More than 1000 questions receive no correct answer from any evaluated model, indicating persistent reasoning challenges. Figure C9 summarizes the distribution of these failure cases: the sunburst plot shows how unanswered questions distribute across task types, multitask load levels, and Bertz complexity ranges, while the accompanying bar charts display the proportion of feature groups associated with failed questions each task category. Figure C10 presents heatmaps of success rates for indexing tasks across these dimensions, complementing the counting task analysis in Figure of the main text.
Types Counting (366)
100
13%
20%
Graph Topology
Groups
Chemistry-Typce
General (215)
Topology
Multitask Load (middle)
Camposition
Load (240)
Chemica Perception
Load
12%
Functional Groups
n=1777
Load (485)
Share
Synthesis
Bertz Complexity (outer)
0-250 (162)
1000+ (063)
51 NIA (Generation only, 216)
22% 24%
Counting
Indexing
Generation
Figure C9: Distribution of questions answered incorrectly by all models. Left: Sunburst diagram showing the breakdown of failure cases (n 1,176) by task type (inner ring), multitask load (middle ring), and Bertz complexity (outer ring). Right: Stacked bar plots showing relative frequencies of feature groups among failed questions counting, indexing, and generation tasks.
C.2.2 TASK-SPECIFIC SUCCESS PATTERNS
Complementing the analysis of universal failures, examine how success rates vary across bench- mark dimensions for specific task types. Figure C10 presents heatmaps of success rates for indexing tasks, analogous the counting task analysis Figure 3 of the main text. Unlike the sunburst anal- ysis above, which identifies questions no model solves, these heatmaps show the average success rate across models for each combination of Bertz complexity, multitask load, feature group, and functional group family. This reveals systematic difficulty gradients: certain feature groups (c.g., synthesis-related properties) and functional group families (e.g., organosulfur compounds) yield consistently lower success rates, even when individual questions are not universally failed.
C.2.3 PERFORMANCE TRANSITIONS UNDER COMPOSITIONAL STRESS
key design feature MOLECULARIQ that each molecular property appears across multiple multitask loads and task types with matched instances. For any given property (e.g., counting hy- droxyl groups), the dataset contains corresponding questions at loads through 5, as well as sibling tasks (counting, indexing, and generation) that query the same underlying property. This structure enables fine-gre meeggraieed tracking of how model performance on specific sub-property evolves under compositional stress, rather than merely observing aggregate accuracy drops. Figures C12 and C11 leverage this matched design to analyze performance transitions at the property level.
Figure C12 tracks whether models that succeed on a property in single-task settings maintain success as multitask load increases. For each property, we identify whether the model's outcome transitions from success to failure (or vice versa) as additional subtasks are composed together. Figure C11 examines cross-task transfer using the same matched instances, showing how success counting task for given property influences performance on the corresponding indexing task. Together, these results reveal systematic degradation patterns: counting tasks show modest degradation under load, indexing tasks exhibit strong brittleness, and success-to to-s-to-ffrre to-fuiuure transitions consistently dominate over Balure-to-succes transitions.
51

---

a) Bertz
b) Functional Groups
<) Multitask Load
d) Feature Types
GPT-05S 1208 (High)
Owen-3 2358
13
31
24
25
23
37
31
23
25
33
37
26
48
32
22
12
02
40
41
26
DespSeebs81--528
28
28
70
Success ratio (96)
(High)
Q
GPT-055 208 [Med]
EPT OSS 1208 (Mcd)
15
22
16
13
23
25
18
16
10
65
34
SEED-OSS
42
13
22
28
25
23
15
50
-00
Ower-3 308
10
20
<250
250-1k
>ik
RON
Sm
Figure C1O: Success rates (%) for indexing tasks across Bertz complexity, functional group fam- ilies, multitask load, and feature groups. Unlike Figure C9, which shows universal failures, these heatmaps display average success rates across all models. Feature group abbreviations: (com- position). CP (chemical perception), CT (chemistry-tyyre topology). FG (functional groups). SYN (synthesis). Functional group abbreviations: ROH (alcohols/phenols), π--Ys (T-systems), S/SO, (organosulfurr), Hal (organohalides), Ar (aromatics), ROR (ethers/epoxides), C=O (carbonyls), Alk (alkyl groups), NR (amines), C=N/N=O (nitriles/nitro groups).
Load
Load
Load
Load
S7%
CUS
ITV
Counting
Indexing
Counting
Indesing
Counting
Indexing
Counting
Indexing
Tel
Figure C11: Cross-task seccess-flow from counting to indexing under increasing multitask load. For each load level, panels display how models transfer success from counting tasks (left bar) the corresponding indexing tasks (right bar) on matched property instances. Analysis done for top-10 models.
52

---

Load 2
Load
Load
Counting
2333
03%
ICB
2355
Indexing
626
Po
222
014
Load
Load
Load
Load
Lead
Lead
Generation
AmA
2014
120
125
299
35%
Success
Fail
Success-SSurcess
Fall-Success
Success-FFal
Fail-Fail
Figure C12: Success-flow analysis from single-task to multitask settings across counting, indexing, and generation tasks. Each panel tracks performance transitions on matched property instances multitask load increases. Analysis done for top-10 models. Colors indicate transition types: sustained success, sustained failure, success fficlur, failure-ssuccess.
53

---

C.2.4 OUTPUT FORMAT VALIDITY ANALYSIS
To disentangle reasoning errors from formatting failures, we analyze the relationship between type validity and task accuracy. Type validity measures whether a model's output conforms to the ex- pected syntactic format for each task category, independent of correctness. We apply several layers of relaxations during answer extraction to handle common edge cases such as different bracket types, nord-to-number conversions, and minor formatting variations. (see Section B.2.3)
Counting tasks: Valid if the extracted answer is an integer.
Indexing tasks: Valid if the extracted answer is list of integers or an empty list.
Generation tasks: Valid if the extracted string is syntactically valid SMILES that can be parsed by RDKit. Unlike counting and indexing tasks, the extracted answer is inherently a string, rather than applying heuristic checks for "near-valid" SMILES, we simply verify parseability.
Since malformed outputs cannot be correct, type validity provides strict upper bound achiev- able performance. Models near the left edge are limited primarily by formatting errors, while mod- els below the diagonal exhibit reasoning emror performing models achieve both high validity and high accuracy; many mid-sized models show high validity but low accuracy, demonstrating that correct formatting is necessary but insufficient for strong molecular reasoning.
when outputs syntactically valid. High-
Overall
Counting
Accrtry (K)
ACCFR (40
20
20-
60
100
0-
22
60
130
Indexing
Generation
Arcunity (9)
30
ACcuacy
80
0-
Waldlty (5)
Tyse Wa (S0
Figure C13: Type validity task accuracy across overall, counting, indexing, and generation eval- uations. The diagonal indicates the theoretical upper bound where performance is limited only by format correctness. Each point represents single model instance.
54

---

C.2.5 FAILURE ANALYSIS ON SUBSTRUCTURES
To investigate structural failure modes in molecular reasoning, analyze model performance with respect to chemically coherent families of functional groups. Because individual functional groups oceur too sparsely to support statistically meaningful conclusions, we consolidate the original set of SMARTS- groups into broader families and retain only those families that appear at least ten times in the dataset to ensure sufficient statistical support. This results in the following functional group families:
Carbonyl and thiocarbonyl compounds (C=0): Functional groups centered on C=O or C-S units, including aldchydes. ketones, carboxylic acids, and their classical derivatives such as esters, amides, anhydrides, and acetals.
contoxy-containing groups in which oxygen is bound to sp', aromatic, vinylic carbon, covering alcohols, phenols, and enolic OH motifs.
Alcohols, phenols, and enols (ROH):
Ethers and epoxides (ROR): Neutral C-O-C linkages, including open-chain ethers and their three-membered cyclic counterparts (epoxides).
Amines and related nitrogen bases (NR): Neutral or cationic nitrogen-based function- alities containing N-H and/or N-C bonds, such as amines, ammonium ions, hydrazines, hydroxylamines, enamines, and guanidines.
Functional groups fea- turing C=N C=N linkages, or nitrogen in highly oxidized or energetic environments, including imines, nitriles, isocyanates, nitro groups, azides, and diazo species.
C-N multiple bonds and oxidized nitrogen groups (C=::N=O):
Organosulfur and sulfonyl groups (S): Sulfur-containing functionalities ranging from thiols and thioethers sulfoxides, sulfones, sulfonic acids, sulfonamides, and sulfonate- type leaving groups.
Aromatic ring systems (Ar): Conjugated cyclic r-systems that satisfy aromaticity criteria, including benzenoid rings and common heteroaromatic scaffolds.
x r-systems and conjugated motifs (r-sys): Unsaturated and conjugated carbon frame- works, including alkenes, alkynes, allyl and benzyl substituents, and electrophilic conju- gated motifs such as Michael acceptors.
Organohalides (Hal): Carbon-halogen functionalities in which sp', sp², or aromatic car- bons are directly bonded to F, CI, Br, I.
Alkyl substituents and carbon-class patterns (Alk): Simple alkyl groups and local car- bon substitution environments (primary, secondary, tertiary, quaternary) defining the im- mediate carbon topology.
For each functional group family, compare its total number of occurrences in the full dataset with its occurrences in the subset of questions a model answers successfully (defined as achieving the correct answer in least 2 out of 3 rollouts). The resulting success ratios (see Figure (b) and C10 (b)) quantify how reliably a model handles questions involving each structural family relative their prevalence in the benchmark.
55

---

C.2.6 CHAIN-OF-THOUGHH FAILURE ANALYSIS
To characterize reasoning failures qualitatively, subsampled 300 examples (100 each from count- ing, indexing, and generation) where no model achieved a correct answer, stratified by Bertz com- plexity and multitask load. We used GPT-5-mini to evaluate sfain-of-thougeee traces along di- mensions: chemistry knowledge and general reasoning capabilities, The rubric prompts are shown Boxes C.2.6 and C.2.6 respectively. These assess model responses across 9 chemistry-speeinf axes (e.g., molecular parsing, ring topology, sterenchemistry) and 9 semantic reasoning axes (e.g., task fidelity, reasoning flow, error awareness). Each axis is scored from (excellent) with "N/A" assigned when the capability
(failed), not evidenced in the model's 's response.
To ensure fair comparison across models, we apply consensus-based filtering: for each evaluation axis, we only include questions where at least 50% of models demonstrated the capability (i.e., received a non-N/A score). This filtering ensures we evaluate models on capabilities that were ac- tually required by the questions, rather than penalizing models for capabilities that were irrelevant to specifie problems. Coverage rates vary by axis-chemistry capabilities range from 13-100% (e.g., molecular parsing: 82- 22-98, stereochemistry: 43-51%). while semantic exhibit consis- tently high coverage (91-100%). Chemistry domain experts validated a subset of these automated judgments.
Figure C14 summarizes chemistry capability failures, including deficits in SMILES parsing, func- tional group recognition, and stereochemistry understanding. Figure C15 presents reasoning capa- bility failures, such as counting errors, index tracking mistakes, and logical inconsistencies. In both figures, lower scores indicate stronger performance (I=xxcellent, 5=failed).
GPT-055 1208 (Hight
1.34
1.42
1.49
1.64
1.70
1.74
2.08
1.90
Owwn-3
2.67
2.52
2.38
2.1
4.5
2.41
3.39
2.43
2.75
2.36
2.67
GPT-065 200 (High)
1.64
1.77
1.96
2.04
2.07
2.00
224
Model
808
2.87
2.54
GPT OSS (Mod)-
2.48
2.57
2.79
2.33
2.47
2.16
2.19
2772
277
2.5
2.0
SEED-O55
2.64
2.65
2.55
2.20
2.49
2:33
2.30
2.71
238
15
1.
GUM-4 6
3335
3.17
3.75
2.m
2.92
2.70
2,41
321
318
Pariing
Beotogi
Nosa
Armadiua
Mecigaitoo
Breding
Vakence
nundimall
Propeity
Steat
Chemical Reasoning Capability
Figure C14: Chemistry capability failure modes for top- models, evaluated on 300 question-trace pairs using GPT-S-mini. Higher scores indicate weaker performance.
56

---

GPT OSS 1206 HHihh
1.53
1.96
1.39
1:47
1:81
225
2.03
3.58
2.28
3.0
DespSesk-RLE RLL 0528
2.59
2.94
2.54
2.77
3:30
284
3.85
3.00
2.39
4.0
3.5
Model
Dee--3 Nest 03B
2.33
2.40
2.33
2.76
3:27
2.72
3.41
2.84
2.45
25
GPT-055 1200 (Med)
2.19
2.59
2.73
2.85
2.79
3.39
3.57
2.47
2.41
15
Owion 308
2.45
3.06
2.50
3.16
3.39
2.00
3.85
3.32
2.48
4.05
Fdelly
Canavenn
Creuniig
Meshod
Reatping
Plow
Preciion
Refersncc Diedptoor
qualety
Reasoning Quality Dimension
Figure C15: Reasoning capability failure modes for top- 10 models, evaluated 300 question-trace pairs using GPT-5-mini, Higher scores indicate weaker performance.
57

---

Evaluation prompts thought traces.
The following prompts used for automated evaluation of chain-of-
System prompt: Chemistry Capabilities Evaluation
You are expert evaluator of chemical reasoning traces.
TASK SPECIFICATION
Evaluate the quality of model's chemical reasoning based its generated trace. Focus on assessing the demonstrated capabilities and reasoning patterns, independent of answer correctness.
EVALUATION FRAMEWORK
Assess the model's 's demonstrated chemistry capabilities from trace evidence on 9 axes.
1 Excellent Clear mastery with nuanced understanding
* 2 Good - Correct approach with minor limitations
3 Adequate Basic competence with notable gaps
. 4 Poor Significant errors affecting reasoning
5 Failed Fundamental misunderstanding
N/A Capability not evidenced in trace
Note: Evaluate based on the model's expressed reasoning, not on independent analysis of the problem.
EVALUATION AXES
STRUCTURAL UNDERSTANDING
1) molecular parssin SMILES interpretation as demonstrated in trace
Evaluate: Quality of structural parsing discussion and graph construction
N/A: No structural parsing demonstrated
Explicit graph/adjacency building with complex notation handling
2: Correct parsing of standard notation
3: Functional parsing with minor uncertainties
4: Character-pateem reliance; structural confusion
5: Fundamental parsing failures
Indicators: Ring closure handling, bracket atom interpretation, branch recognition
2) ring topology Ring system understanding as expressed
Evaluate: Recognition and analysis of ring relationships
N/A: No ring topology discussion
1: Correct identification of fused/bridged/spprro with cycle basis understanding
2: Accurate common ring system handling
3: Basic ring counting with fuzzy fusion concepts
4: Oversimplified heuristics (e.g., "unique digits = rings")
5: Inability identify ring systems
Indicators:
Fusion/bridge distinction, cycle enumeration accuracy
3) connectivity counting Graph traversal and enumeration quality
Evaluate: Consistency in counting and traversal operations
N/A: No counting/traversal performed
1: Verified counts with explicit deduplication
2: Accurate counting with self-correction
3: Generally correct with minor drift
4: Off-by-one errors double-counting
5: Systematic enumeration failures
Indicators: List-total agreement, traversal logic clarity
ELECTRONIC & BONDING
4) aromaticity recognition Aromatic system identification approach
Evaluate: Criteria and reasoning for aromaticity
N/A: No aromaticity assessment
58

---

1: Correct criteria (Hückel, planarity) with heteroaromatic handling
2: Standard aromatic recognition
3: Limited to benzene-like systems
4: Lowercase-one heuristics
5: Incorrect principles (e.g.. "alternating aromatic")
Indicators:
Recognition of pyridine/furan/thioheene electron counting
5) bonding-statee Bond order and hybridization analysis
Evaluate: Understanding of bonding and hybridization
N/A: No bonding discussion
1: Accurate bond orders with resonance awareness
2: Correct standard cases
3: Basic single/double distinction
4: Bonding confusion affecting reasoning
5: Fundamental bonding errors
Indicators: Amide resonance handling, sp2/sp3 assignment
6) charge.valence Formal charge and valence handling
Evaluate: Feasibility and accuracy of electronic structures
N/A: No charge/valence discussion
1: Correct formal charges with protonation state recognition
2: Standard charge handling
3: Neutral molecule competence
4: Valence violations or charge errors
5: Chemically impossible structures
Indicators: Quaternary N recognition, protonation state handling
FUNCTIONAL PROPERTIES
7) functional identification - Functional group recognition
Evaluate: Accuracy in identifying functional groups
N/A: No functional group identification
I: Complex group identification (imides, lactams, enols)
2: Common group recognition
3: Simple group identification only
4: Some misidentification
5: Systematic recognition failures
Indicators: Ester/ether distinction, amide/amine differentiation
8) property attribution - Chemical property assignment
Evaluate: Property prediction accuracy and reasoning
N/A: No property discussion
1: Nuanced properties with exception handling
2: Standard property assignment
3: Basic property recognition
4: Property confusion or misattribution
5: Systematic property errors
Indicators: HBD/HBA assignment, rotatable bond identification
9) stereochemistry 3D configuration and symmetry reasoning
Evaluate: Stereochemical notation interpretation and symmetry recognition
N/A: No stereochemistry discussed
1: CIP rule application with symmetry awareness
2: Correct @/@ interpretation
3: Basic stereo acknowledgment
4: Stereo notation misreading or symmetry oversight
5: Unfounded stereochemical claims
Indicators: E/Z recognition, equivalence identification
INPUT STRUCTURE
59

---

terompt_text"/ "Original task instructions"
"model "model_trace": "Complete model output including reasening and answer",
"verification_insoo" "Detailed verification results, including ground
truth value and detailed verification results
of the generated answer"
OUTPUT SPECIFICATION
2
modell_trcce"
"scores":
"molecular pttpecuuaarpprssigg.... <1-5 "N/A"?,
"ring_topology": <1-5 "N/A">, ''ommectivity__couniin"" <1-5 or "N/A">,
"aromaticity_rreconttion <1-5 or "N/A">,
"bonding_states": <1-5 or "K/A">,
"charge_valencee <1-5 or "N/A">,
"functicnal_ideni iittaaiiaiiiiiiiiiii..... <I-5 or "N/A"?, "property <1-5 or "N/A">,
"steroochemistry": <1-5 or "N/A">
"coverage";
"axes_scored"; <count of non-N/A>, "axes_total"; 9,
"confidence"; "chigh/medi um/low>*
"summary": "Concise characterization of the trace's chemical reasoning strengths and weaknesses"
EVALUATION GUIDELINES
Base assessments solely evidence within the provided trace
Consider impact: errors affecting conclusions score worse than isolated mistakes
Use the full scoring range maximize discriminative power
Document N/A axes to track evaluation coverage
System prompt: Reasoning Capabilities Evaluation
You expert evaluator of model reasoning behavior for eedecular-structure tasks.
TASK SPECIFICATION
Evaluate the quality of model's reasoning patterns based on its generated trace. Focus on assessing the demonstrated reasoning behaviors and execution quality, independent of answer correctness.
EVALUATION FRAMEWORK
Assess the model's 's demonstrated reasoning capabilities from trace evidence on 9 axes.
Excellent Exemplary reasoning with systematic approach
2 Good Solid reasoning with minor limitations
3 = Adequate Functionnl competence with notable gaps
4 Poor Significant flaws affecting outcomes
5 = Failed - Fundamental reasoning breakdown
N/A Capability not evidenced in trace
60

---

Note: Evaluate based the model's expressed reasoning, independent analysis of the problem.
EVALUATION AXES
TASK UNDERSTANDING
1) task Requirement interpretation demonstrated in trace
Evaluate: Accuracy in understanding and adhering task requirements
N/A: Task interpretation not observable
1: Precise restatement with complex operator handling (AND/OR)
2: Clear understanding with minor terminology drift
3: Generally on-task with some scope confusion
4: Partial misinterpretation affecting answer direction Fundamental misreading; solves wrong problem
Indicators: Task restatements, focus consistency, operator logic
2) constraint.tracking Multi-requirrment management quality
Evaluate: Completeness in tracking all conditions and subtasks
N/A: No multi-constraint task
1: Explicit constraint verification with systematic checklist approach
2: Main constraints tracked with minor omissions caught
3: Most constraints handled with some drift
4: Lost constraints; conflated outputs
5: Major constraint failures: multi part task breakdown
Indicators: Checklist behavior, subtask completion, requirement coverage
3) evidence grounding
speccic verification approach
Evaluate: Use of local evidence versus generic reasoning
N/A: No verification opportunity
1: Systematic local checks with explicit enumerations for this molecule
2: Strong local verification with minor generic reasoning
3: Mixed local checks and generic rule application
4: Predominantly generic rules with minimal instance verification
5: Pure analogy without local evidence
Indicators: Atom-level enumeration, path traces, molecule-specific calculations
REASONING PROCESS
4) nethood_selction- Approach appropriateness and rigor
Evaluate: Quality of chosen reasoning method
N/A: Method choice not apparent
Principled algorithm with explicit assumptions and limitations
2: Sound method with minor shorteuts
3: Acceptable heuristic appropriate for case
4: Brittle shorteut with limited applicability
5: Random exploration without systematic approach
Indicators: Method justification, stated assumptions, alternative consideration
5) reasoning flow Efficiency and convergence quality
Evaluate: Balance between thoroughness and decisiveness
N/A: Insufficient reasoning visible
1: Efficient exploration with clear convergence; loops premature jumps
2: Well-balanced progression with minor repetition
3: Some meandering but ultimately converges
4: Notable overthinking with loops rushed premature closure
5: Extreme overthinking with endless reversals or snap judgment on complex task
Indicators: Reasoning depth vs complexity, revisit patterns, decision points
6) error awareness Belf-monitoring and correction capability Evaluate: Detection and handling of uncertainties and errors
N/A: No errors uncertainties present
61

---

1: Proactive detection with calibrated uncertainty expression
2: Effective self-correction with reasonable confidence levels
3: Occasional checking with mostly aligned confidence
4: Missed obvious errors with confidence miscalibration
5: No error detection; false confidence despite contradictions Indicators: Corrections made, hedging language, consistency verification
EXECUTION QUALITY
7) quantitative_preeiin
Numerical and counting accuracy
Evaluate: Precision in arithmetic and enumeration operations
N/A: No counting arithmetic performed
Verified counts with list-total agreement and clean propagation
2: Accurate quantitative work with minor corrections
3: Mostly correct with non-critical drift
4: Off-by-one errors count-total mismatches
5: Systematic miscounting with cascading errors
Indicators:
Enumeration completeness, total verification, arithmetic chains
8) eeference.discipline Index and label management consistency
Evaluate: Consistency in handling references and identifiers N/A: No index or reference handling
Consistent references with explicit base conventions; no drift
2: Clear reference system with rare slips
3: Generally consistent with occasional confusion
4: Reference drift with base confusion out-of-range issues
Systematic index errors throughout
Indicators: Index usage patterns, entity naming consistency, base convention clarity
9) output quality Answer completeness and format compliance
Evaluate:
Completeness and organization of final answer
N/A: Output format not specified
I: Complete, well-formatted answer addressing all required keys
2: Complete answer with minor formatting issues
3: Mostly complete with some organizational gaps
4: Missing keys or significant format violations
5: Partial or jumbled answer; unparseable output
Indicators: Format compliance, key coverage, structural organization
INPUT STRUCTURE
"prompt_text"; "Original task instructions",
"nodel_trace"; "Complete model output including reasoning and answer",
"verification "Detailed verification results, including ground
truth value and detailed verification results of the generated answer"
OUTPUT SPECIFICATION
"evaluation_target"" "model_trace", "scores":
"task_fidelity": <1-5 or "N/A">, coonttriin_traacinng <1-5 or ">/">> "evidence
"ericencee_goouddnng" grounding"; <1-5 or "N/A">, "methoa wtw.ttoiosseeeciion". <1-5 "N/A">,
62

---

"reasoning_ <1-5 or "N/A">, "error_awareness": <1-5 or "N/A">,
"quanciiatlve__pecciiinn'"" <1-5 or "N/A">, rrefereece_diiscpliine <1-5 "N/A">, "output_qqulity"": <1-5 or "N/A">
,, "coverage":
"axes_scored" <count of non-N/A>, "axes_total"; 9,
"conficence"; "shigh/medium/ www.mmedddmmmmmeeemm umwwow>" }
sunmary": "Concise characteri cattrrizatiinn of the trace's reascning strengths and weaknesses"
EVALUATION GUIDELINES
Base assessments solely evidence within the provided trace
Consider impact: errors affecting conclusions score worse than isolated mistakes
Use the full scoring range maximize discriminative power
Document N/A axes to track evaluation coverage
63

---

C.2.7 FAILURE MODE EXAMPLES
This section provides examples failure modes. The accompanying scores quantify the degree of failure, where higher values correspond more severe failure grades (see system prompts above).
GPT-OSS 120b (Medium) aromatic rings
Question
Type:
Complexity: 1000-inf
How many aromatic rings are present in the molecule D=CC-CCCNNCC-CCCCCCCCCCCCCCCCCCC +CCC==CCCCCCCCCC---CCCCCS------- =CC=))=N2)=C1? Please provide your answer as a JSON object using aromatic ring count.
Model Response
< channell Saaamnel/>aaan D>anaaaaansaeaaaa Imessage >ee need count aromatic rings in given SMI LES. Need parse aromatic atoms: lower case? Provided uses uppercase C etc with etc but aromatic indicated by lower case? The SMILES but aromatic rings count maybe based on aromatic pattern? Usually aromatic indicated by lower case c. Here it's all uppercase, so not aromatic. So aromatic.ring.coutt <answer> Bttpwerr.</enddii stttd (ansser>("armmtticzzria"
cl-cc-cc... that's s not aromatic lower case,
0. Provide JSON within |>fina<< |message 0}</anserr>
Ground Truth
Extracted Answer
Aromatic Ring Count:
10
Aromatic Ring Count: O
Chemistry Scores
4.0
Reasoning Scores
3.2
Molecular Parsing
4.0
Task Fidelity
2.0
Ring Topology
N/A
Constraint Tracking
2.0
Connectivity Counting
N/A
Evidence Grounding
4.0
Aromaticity Recognition
4.0
Method Selection
4.0
Bonding States
N/A
Reasoning Flow
Charge Valence
N/A
Error Awareness
4.0
Functional Identification
N/A
Quantitative Precision
5.0
Property Attribution
N/A
Reference Discipline
2.0
Stereochemistry
N/A
Output Quality
Analysis Summary
The trace demonstrates a minimal SMILES-level heuristic (noting lowercase atoms denote aromaticity) but substantive structural parsing ring analysis. Aromaticity reasoning is simplistic and reliant character case only, failing consider kekulized (uppercase) SMILEES perform any graph/ring traversal. No discussion ring topology. connectivity, bonding, charges. functional groups, properties, or stereochemistry is provided.
Chemistry:
Reasoning: The model correctly understood the prompt and produced the requested JSON output, relied on an overly simplistic heuristic (lowercase "c" => aromatic) and failed perform molecule- colecule-speccffc verification or ring enumeration. Reasoning was concise but prematurely convergent and overconfident, showing little error awareness. Execution produced formatted answer but the counting method was brittle and led to systematic, high-impact quantitative emor.
64

---

GPT-OSS 120b (Medium) stereochemistry
Question
Task Type: Index
Load: 1
Complexity: 250- 250-1000)
Which atoms CIC@ cicceeccccccc-ccccccccoooo- [C @ CHCCCHHHACCCCCCCCCCCCCCCCCCCCCCC [C@@ @ [][O)[[@ @ I2IC are R Rteteooccetees? Provide your answer with __sttreeccetter r iindex containing the indices of these positions.
OH
Model Response
< Danaannei/aaaaiiiisssse mmssagel>>ße need to parse SMILES, identify atoms that are stereocenters. Need output JSON with key w..s.ttereocenter.rt indee" Provide indices of stereocenters. We must compute sherreccheisstry from SMILES: с[саен] ICCC-c2c 30C (=0) [CØ0H] (C 4CC+ (CS-N [C8H]3]Се (0) [Cee]21C. need identify chiral centers with specified absclute configuration.
SMILES, '0' indicates chirality: '6' is anticlockwise (S?) Actually convention: In SMILES, 'Ør denotes counterclockwise (S) for tetrahedral? According to spec: For tetrahedral, 'p' is anticlockwise when looking from the viewer toward the chiral '00' is clockwise. But many teols treat 'a' as S and "Ge' as R? Let's s recall: For example, Iralannne SMTLES: C[C@H] (N)C(=0)0 -> I-alanine is configuration. So CICEEJ (=0) corresponds to S. Good.
with the lowest priority substituent away.
So rer => Then "ép"
Thus we need to find atoms with '@e- that are R. Also could be '3' with if conventions reversed? Let's trust above: 'Be' -> R. Now parse SMILES and index heavy atoms.
SMILES: c[cε@н] 1cCC=C2c(cHH] 30C (-0) [сеен] [C[NH+] 4CCN (C5-NC-CC-NS) CC4) (сCн]) [C88]2IC.
We need to include brackets for chiral: [са8H] etc. Atoms: Let's enumerate.
I'll write string with tokens. Start: C atomo (C)
Next: [CеeH]] bracketed carben with H, chiral. Atoml is C (the bracket). It has chiral rear -> R. Index1. The "1" is ring digit, not atom.
Next: C atom2
Next: C atom3
Next: C atom4
Next: =C2 carbon double bend: 'c' atom5, with ring digit 2 after? Actually /=02' means carben atom5, double bended, and ring opening digit 2 attached to this carbon.
Indices are heavy atom indices starting from 0. Confirm indices count: Let's s recount to ensure correct numbering.
List with indices:
C (first 1 (bracket [с(@H]1)
(=C2) C (after C2) 7 (1Ce8)3) o C (carbonyl) 4 (carbenyl 0) c
(Iс86H) (...)) 12 C (C attached to that) 13 N (INH+14) 14 15 c 16 N (N(...)) 17 (C5) 1 19 c 20 с 21 C 22 N 23 C 24 C 25 C (Iсен]3) 26 C ([се@н) )0)) 27 0 28 c (IC88]21C) 29 C (final C) Check that after [сен]3 (atom25) we have [Cа2H] (0) (atom26) then (27) then [cee]21c (atom28) then C (atom29). Yes. Thus indices: atoml, atomll, atom26, atom28. Return
JSON.< >< Dassisttate cchaneel| >final< nessage <<aassee>>> ["LssttoroccttorsLLno"" [1, 11, 26, 2
65

---

Ground Truth
Extracted Answer
Stereocenter R Index: [1. 7, 26, 28]
Stereocenter Index: [1, 11, 26, 28]
Chemistry Scores
1.7
Reasoning Scores
2.1
Molecular Parsing
1.0
Task Fidelity
2.0
Ring Topology
2.0
Constraint Tracking
3.0
Connectivity Counting
1.0
Evidence Grounding
Aromaticity Recognition
N/A
Method Selection
1.0
Bonding States
3.0
Reasoning Flow
2.0
Charge Valence
2.0
Error Awareness
4.0
Functional Identification
2.0
Quantitative Precision
2.0
Property Attribution
N/A
Reference Discipline
Stereochemistry
1.0
Output Quality
3.0
Analysis Summary
The trace shows strong low-level SMILES parsing and atom-indexing skills (explicit tokenization, ring-digit handling and complete atom index map). Connectivity counting is precise and self-consistent. The model correctly explains the SMILES stereochemical notation (@ vs @ @ and uses it to identify R-centers, demonstrating good stercochemical notation handling. Weaknesses: limited discussion of aromaticity and broader property attribution, only basic treatment of bonding (single/double and carbonyl recognition but hybridization resonance analysis), and only standard charge tandlin//functioall identification (recognized [NH+] and carbonyl but did not name more complex groups). The approach relies on the SMILES @ @ @ tokens rather than performing CIP-centred absolute-configuraate determination from substituent priorities, which can lead to missed incorrect final assignments (the trace's final list omits chiral position).
Chemistry:
The trace shows a clear, systematic approach: the model restates the task, explains the SMILES chirality convention, and enumerates atoms with explicit, molecule-specific indexing. Method selection and grounding are strong it principled SMILES SMILES-parsing algorithm and lists atom-level evidence, Execution mostly precise and references are consistent. Weaknesses: the model exhibits incomplete constraint tracking (it omitted an R center at index 7 in its final output) and limited awarenwressess (no self cofrectoottnn after the omission). Output formatting is correct but the answer is missing required element, reducing completeness.
Reasoning:
66

---

GPT-OSS 120b (Medium) stereocenter constraint generation
Question
Task Type: Generation
Load:
1
Complextyy: 1000-inf
OH
OH
Provide one SMILES for molecule achieving exactly R-stereocenters. Give the generated molecule a JSON object keyed by 'smiles'.
OH OH
Model Response
<Ichannell Danaaysiss [messagel >>e >Wee need to output SMILES with
exactly 4 R stereocenters. Must be in JSON with key "smiles".
Provice SMILES. Need to ensure stereocenters all R. Use wedge notation with B? SMILES, absolute configuration is indicated with for anticlockwise (S?) Actually CIP: "a" denctes
anticlockwise (counter) relative to SMILES order, which corresponds to S? In SMILES, "Q" is anticlockwise (counter) which corresponds S for typical tetrahedral. "@" denotes anticlockwise (counter) which yields if the pricrity order follows SMILES order. "g" and "0e" are opposite. But we need R. So we can use "00" for clockwise which yiclds R. So we need 4 stereocenters with "06" each. Provide reasonable molecule like tetrahydroxy but with chiral centers.
Example: C[Cен] (0) [сан] (0) [CеH] (0) [CGH] (0)C maybe? Let's count: Starting SMILES: eC? Actually need chiral centers. Let's design: "C[CeH] (0) [cea] (0) (C8H] (0) [Cеa] (O)C". Let's index: start C (0),
chiral, (4), [Cea] (5) chiral, (8), (9). That's chiral centers (the four [C@H]?). Actually there are [C2H] each with "g" That's s anticlockvise (S). We need R, so use "ea" for each; " "[cee]] " So SMILES: "c[ceen] (0) [cеeн] (0) [C22н] (0) [Cе6н] (O)C". That has R centers. Provide JSON.
[CHH] (1) chiral, 0 (2), [C0H]
(6), [Cea] (7) chiral,
Check: Are there any other stereocenters? The terminal C's are not chiral. So exactly 4. Good. Return: (answer>(ssmilees "C(сеeн] (0) [сеен] (0) [Се8H] (0) [cеeн] ///cc) <</nd)) <lstart channel >> finial |
assistant< chaannee
les"; "C[C00H] (0) [CеeH] (0) [саан] (0) [C00H] </)ser<</nserr>>
Generation Results
Reward: 0.0
Generated SMILES: {" ["smile""; "C[се0н] (0) [Cêен] (0) [се8н] (0) [сêен] (O)c")
Validation: Valid SMILES: /
Property
Expected
Actual
Constraint Satisfaction:
Stereocenter R Count 4
67

---

Chemistry Scores
3.7
Reasoning Scores
3.0
Molecular Parsing
3.0
Task Fidelity
Ring Topology
N/A
Constraint Tracking
3.0
Connectivity Counting
4.0
Evidence Grounding
4.0
Aromaticity Recognition
N/A
Method Selection
4.0
Bonding States
N/A
Reasoning Flow
Charge Valence
N/A
Error Awareness
5.0
Functional Identification
N/A
Quantitative Precision
3.0
Property Attribution
N/A
Reference Discipline
1.0
Stereochemistry
4.0
Output Quality
Analysis Summary
The trace shows some practical familiarity with SMILES stereochemical notation (recognizing @ vs @@@ and using explicit chiral labels) and the ability to assemble reasonable linear polyol SMILES. However the reasoning incomplete: the model assumed that placing four "@@ labels automatically yields four distinct stereocenters without checking CIP priorities or molecular symmetry, which led to an incorrect final claim. Molecular parsing is functional but not deeply validated (score 3). Comnctivityyccouttng failed to account for equivalence/symmete (score 4). Stereochemistry handling shows basic token-leve level knowledge but misapplies it to absolute R assignment and lacks verification (score 4). Other axes not addressed in the trace.
Chemistry:
The trace shows clear understanding of the prompt and returns the requested JSON key with candidate SMILES, and it uses consistent notation. However the reasoning relies simplistic rule about SMILES stereochemical markers without mnstance-specifie verification (no atom-level checks or CIP application). and the model shows no error detection or calibration; as a result the produced SMILES is likely incorrect with respect to the explicit constraint despite appearing to meet it syntactically. Overall: task understood and output formatted correctly, but weak grounding, verification, and error- arro--wwareness
Reasoning:
method is brittle shorteut (placing four
markers)
DeepSeek-R1-0523 bridgehead atoms
Question
Type:
Complexity: 1000-inf
Which atoms in RCCC-CCCC--CCCCCCCCCCCCCCC -CCCCCCNNEEBBCCCCCCCCCCCC OOC--C COCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
HO
serve as bridgehead positions in this polycyclic structure? Please
provide your answer JSON object with bridgehead index listing these atom indices.
68

---

Model Response
<think> we are given SMILES string: PC16=C2030-00-0 (C=3) C3C (NVANN N3CCC (oCcCC (0C2-CC-1F) C) CC3) [сеH] (OC (C) (C) (-010)). we are to identify bridgehead atoms this polycyclic structure. Bridgehead atoms are defined as atoms that are part of more than one ring and are shared by at least two rings. In Sused polycyclic systems, they are the atems
Let me try to sketch the structure:
The molecule has a complex polycyclic system. From the SMILES: - We see ring closures: 1, 2, 3, 4 (and also 3 and 4 appear multiple times, which is allowed in SMTLES)
note: the same ring closure number (e.g., 3) appears twice: once "C3C= and then again "CJC=C4N=CI...". This indicates that the ring closure is used for two different rings? Actually, this is common In SMILES, when ring closure number is reused, it forms ring. So we have:
to form fused rings.
Ring closure 1: connects two atoms: one after the first F (atom index 1) and the last part (in the fragment "OC2=CC-1F"), closure 2: appears in "C=C2" and then "OC2=CC=1F", se it connects the after the first 'zr (in C=C2) and the atom in the fragment "OC2=..."
Ring
first use: "C3C=CC-C(C-3)" -> this forms a ring: from the atom labeled by the first '3' to the atom at "C=3". Then later, "c3C=C4N=C..." -> this is new ring closure 3? Actually, it is the same ring closure number but for different ring?
Ring closure
SMILES, when ring closure number reused, starts a new ring. So the first ring closure 3 is between two atoms (let's call them A and B) and then the next clesure 3 is between two other atoms (C and D).
This complexity suggests Given the complexity, I will use the following appreach: 1. Write out the atom sequence (heavy atoms only) and note the ring closure numbers and their pesitions.
need to map the atoms.
But note: the SMILES string is long. Iet break it down: Atoms (heavy atoms only, ignoring hydrogens) and their indices (starting at 0):
Index 0: F (attached to C at index 1) Index 1: C (from "FC1") Index 2: с (2rom "C=C2" -> so it's the second C? Actually: after F (index0) comes (indexl), then C (index2), then (double bond), then (index3). But wait, the string "FCIC=C2" meaning:
FCIC=C2
Then the next part: "C3C-C4" -> this is new ring closure? but note: the ring closure number is reused. So start new ring closure Actually, it's new ring. So:
10: C (ring closure 3 attached) [from "C3"] 11: C (double bond)
12: C4 (ring closure 4 attached) [from "c4"] 13: = .... etc.
wiil list every atom in the order they appear (depth-first. traversal):
Then after the branch: the next token is 'C3'
carbon?
index10: C with ring closure 3 (new ring) Then 'C' -> indexll: with ring closure 4 Then 'N' -> index13: N Then f-r -> bond Then 'c' -> index14: с Then First: C( C( ... ...
Then '=r -> bond Then 'C4' -> index12:
'( -> branch: Inside: C( ...
This too complex. There are heavy atoms? me count:
Thus, the sinal answer. <answer> ("bridgeheadddnneex" [1, 3, /ans<rr
69

---

Ground Truth
Extracted Answer
Bridgehead Index: [4, 8, 10, 17, 19, 22]
Bridgehead Index: [1. 3, 10]
Chemistry Scores
3.6
Reasoning Scores
3.6
Molecular Parsing
4.0
Task Fidelity
2.0
Ring Topology
4.0
Constraint Tracking
Connectivity Counting
5.0
Evidence Grounding
4.0
Aromaticity Recognition
3.0
Method Selection
4.0
Bonding States
3.0
Reasoning Flow
4.0
Charge Valence
N/A
Error Awareness
3.0
Functional Identification
3.0
Quantitative Precision
Property Attribution
N/A
Reference Discipline
4.0
Stereochemistry
3.0
Output Quality
3.0
Analysis Summary
The trace shows a partial, informal attempt parse the SMILES identify ring junctions but contains substantial errors and inconsistent bookkeeping. The model tries to tokenize the SMILES and assign atom indices (evidence of attempted moolcuuarrppassing but repcatedly mis-parses ring-closure usage and index assignments, parsing quality is poor. Ring topology reasoning is present (recognizes fused rings and reused closure digits) but demonstrates misunderstandiins about ring-closure semantics fusion, so topology handling is weak.
Chemistry:
Connectivity/counting is systematically unreliable (many inconsistent index assignments and off-by-many errors), leading to failed score. Aromaticity and bonding are only cursorily acknowledged (mentions "benzene like' and double bonds), showing limited competence. Functional group recognition basic (identifies benzodiazeeineetrrazoooo features) but not precise. Charge/valence and property attribution are not addressed. Stereochemistry tokens are noticed but not meaningfully interpreted. Overall, the trace reveals partial domain knowledge but poor execution and tracking, causing incorrect conclusions despite some correct high-level vocabulary.
core and side-chain
Reasoning: The trace shows an overall correct high-level understanding of the task (identify bridgehead atoms and return JSON) and attempts to parse the SMILES and enumerate atoms. However, the execution is ad hoc and error-prone: the SMILES parsing is inconsistent, ring-closure mapping and atom indexing are handled heuristically with many speculative steps, and the trace contains back and nortforth reasoning and counting mismatches. The model expresses some uncertainty but fails to perform systematic instance-leve verification or produce reliably tracked indices. The final output is fol-tterttted JSON but incorrect and under-justified. Strengths: task recognition, some local enumeration attempts, hedging about difficulty. Weaknesses: brittle method, poor constraint tracking, inconsistent indexing counting, limited concrete atom-level checks.
DeepSeek- MaabeekkkR1-0522 R1-0528 constraint generation (rotatable bonds and molecular formula)
Question
Task Type: Generation
Londl: 2
Complexity: 250 1000
Formulate molecule engineered satisfy exactly rota- table bonds and exactly C16H1 8N403 molecular formula. Respond with JSON mapping whose key is 'smiles'.
H&N
Model Response
<think> We are to generate a SMILES string for a molecule with: Exactly 6 rotatable bonds - Molecular formula exactly C16A18N403 Constraints: Rotatable bonds: typically defined as any single bond that is not in ring and is not terminal (i.c., connected to non-terminal hoavy atom).
70

---

Approach: need structure that has 16 carbons, 18 hydrogens, nitrogens, and oxygens.
Let's break down the formula: C16818N403. The degree of unsaturation (DU) can be calculated: DU (2C 2 H x N)/2 (2+16
4)/2 (3242-18+4)
20/2 10.
This suggesta a high degree of unsaturation, which could come from rings, double bonds, or triple bonds. However, note that each ring and each double bond counts one DU, and triple bend counts as two.
must also have exactly 6 rotatable bonds. Therefere, we should avoid too many rings because rings reduce the number of rotatable bonds. But note: if we have a chain with many single bonds and have rotatable bonds only en the single some double bonds, in rings and not terminal.
bonds that
How about:
Central benzene: C6H4 (attached at 4, Each chain:
-C2-CH2-CH2-MB--OO -NE2 Carbons: (from CH2+3) - (from
C-0) (from NH2? Actually, the carbonyl carbon one) -> 5
carbons per chain? But the chain: -CH2-CHI-CH2-MM-CC000
Atoms: C, C, C, N, c, o, N, H's Formula for one chain: C4H9N20
(because: C: 4 (from 3 CH2 and one carbonyl H: (from CH2)
62 But each CH2 has 2H2 Actually: -CH2- CH2 (2H) -CH2-
CH2 (2H) -CH2- : CH2 (2H) -NH-
(1H) (C : carbonyl (no H)
-NH2 NH2 (2H) So total Н: 2+2+2+1+2 9 But wait: the first
CH2 is attached to the benzene and the last NH2 is attached to the carbonyl. So the chain is: Benzene CH2 CH2 CH2 NH C(=0)
NH2 The formula for the chain without the penzene attachment is: C4H9N20, but we have to remember that the first carbon replaces one H. from benzene.
C4H9N20 C8H18N402 Plus the benzene ring (C6H4) -> total: C14R22N402. But need C16H18N403. No: matching.
So two chains:
We need to add two carbons and remove 4 hydrogens and add one oxygen.
Alternatively, we can use:
[-[N+] ([O=]) (NC (=0) CNc2cce (CCN) cc2) eol -> this has the middle nitrogen as tertiary (no H) and we had 17H. But need 18H. So must have the middle nitrogen with one H. Therefore, the correct
The nitro group is represented with charge separation, but overall the molecule is neutral. This should be acceptable. Let me double-check the rotatable bonds in this new design: Bonds: 1. First ring to the first nitregen counted. 2. The amide bond: N-C -> not counted. 3. The bond from the carbonyl to the CH2: CC=0)-C (single bend) -> counted. 4. The bond from the CH2 to the middle nitrogen: C-[NH] -> single bond, counted, 5. The bond from the middle aitrogen the second ring: [NH]-c2 -> single bond, counted. from the second ring to the CH2 in the ethylamine: c2-CCN -> counted. 7. The bond in the ethylamine: between the two CH2: C-C -> counted. 8. The bond from the last CH2 to the NH2: terminal not counted.
NC(-0)) -> single bond,
The bond
We have bonds 1, 3, 4, 5, 6, It matches.
6.
So we are done. </think> { (smmi "smiles":
"O-[N+] ([O-]) clcce (NC (-0) [[[NN]]2ced (CON) ec2) ccl
71

---

Generation Results
Reward: 0.0
Generated SMILES: { "amiles";
"O-[N+] ([O-]) cleee (NC (-0) [NH)] c2ece (CCN) cc2) ccl
Validation: Valid SMILES:
Property
Expected
Actual
Constraint Satisfaction:
Rotatable Bond Count
=
Molecular Formula
C16H18N403
C16H18N403
Chemistry Scores
2.1
Reasoning Scores
2.2
Molecular Parsing
2.0
Task Fidelity
Ring Topology
2.0
Constraint Tracking
2.0
Connectivity Counting
2.0
Evidence Grounding
2.0
Aromaticity Recognition
2.0
Method Selection
2.0
Bonding States
2.0
Reasoning Flow
2.0
Charge Valence
2.0
Error Awareness
3.0
Functional Identification
2.0
Quantitative Precision
3.0
Property Attribution
3.0
Reference Discipline
3.0
Stereochemistry
N/A
Ouiput Quality
2.0
Analysis Summary
The trace shows solid, chemically informed reasoning: the model demonstrates good practical SMILES handling (including explicit nitro representation). correct recognition of aromatic rings and how ring bonds influence rotatability, and clear understanding of amide/carbonyl bonding and why amide bonds are typically excluded rotatable. It also breaks the molecule into fragments and performs explicit atom and hydrogen bookkeeping with seff-correction. Weaknesses: despite careful reasoning the model miscounted rotatable bonds (verification shows off off-by-one by-one error), indicating mistake in the final enumeration of which single bonds meet the rotatable-bond criteria. Overall competent chemical reasoning with isolated, but consequential, counting/assigmmet error in property attribution.
Chemistry:
The trace shows well-focused, systematic approach: the model carefully restated the task, computed degree of unsaturation, and explicitly tried to design structure meeting both formula and rotatable-bont constraints. used molecule- molecule-specifie reasoning (atom counts, enumerated rotatable bonds) and iterated to fix an initially detected hydrogen-count mismatch. Method selection was principled (building rigid aromatic core with fiexible linkers) and the reasoning flow was coherent and convergent. Weaknesses: the model mis-evaluates the rotatable-bone count (final
Reasoning:
SMILES failed the rotatable-bond constraint per verification). indicating off-by-one conceptual slip in which single bonds are rotatable (some N-aryl or other bonds were counted/miscounted) There was some ambiguity in SMILES-level hydrogen/valeee /allncc handling (explicit [NH] vs implicit H) and missed final verification step against the precise rotatable bond definition. Overall: competent, systematic reasoning with notable execution count).
the key constraint (rotatable-bon
72

---

GLM-4.6 constraint generation (stereocenters and saturated rings)
Question
Tisk Type: Generation
Load:
Complexit; 0-250
Build molecule containing exactly S-stereocenters and exactly saturated rings. Respond with a JSON mapping whose key is 'smiles'.
HH
Model Response
<think>I need to create molecule with exactly S-stereocenters and exactly 5 saturated rings.
Let me think about this step by step:
1. Saturated rings are rings that contain only single bonds (no double bonds the ring) with configuration
S-stereocenters are chiral centers
I need to design a molecule that has both of these features. Let think about structure that could work: One approach could be to where can introduce chiral centers at bridgehead pesitions or other suitable positions.
polycyclic structure with 5 rings,
let me try to build something with 5 saturated rings first: - A cyclopentane ring (5-membered saturated ring) cculd then fuse bridge additional rings Actually, let me think a more systematic approach. could use a steroid-like backbone or a polycyclic structure.
it
For 5 saturated rings, I could use something like: A bicyclic system fused to another bicyclic system, plus more ring Let me try to construct molecule with fused saturated rings and rnduee 5 s-stereocenters.
One possibility could be pentacyclic structure like:
C1---C2 C2
C5
C3
C5
C7
C8
c10
c11
C12
C13
C14
c15
C16
C17
C18
C19---C21
But this getting complex. Let me think of a simpler approach. Actually, let me use a known polycyclic structure and modify it. could use a decalin-like structure extended to have 5 rings. Let try: structure with 5 fused eyclohexane rings (like pentacyclof....]] structure. I'll focus creating complex polycyclic molecule with specific stereochemical requirements. The key challenge is introducing five s-stereocenters while maintaining five saturated rings. I'll need to carefully design the molccular architecture to ensure precise stereochemical configuration.
73

---

Model Response
The molecule will involve a sophist ring system where each chiral center is strategically positioned. I'11 wedge and dash notations to control the absclute configuration, ensuring all stereocenters are S-configured. The structure will require meticulous attention to bond angles and spatial arrangement to achieve the desired stereochemical outcome.
My approach involves constructing polycyclic framework with carefully placed substituents that define s-configuration at each stereogenic center. The complexity 1ies in simultancously managing ring saturation and stereochemical precisicn.</think <answer> ("siir>>> "C[CeeH] [C@H] (CCH]20C[e0 [ceH] 3CC [cеeн] ACCICCCH] 4CC|C (CCC333020054322111 </answer>
Generation Results
Reward: 0.0
Generated SMILES; {" smmiles": "C[CeеH] [C@H] 200 [CeH] [CGe)) [CeeH] 5CC [ceaH] 504c302c1"
Validation: Valid SMILES: /
Property
Expected
Actual
Constraint Satisfaction:
Stereocenter Count
5
3
X
Saturated Rings Count = 5
5
Chemistry Scores
3.2
Reasoning Scores
3.1
Molecular Parsing
3.0
Task Fidelity
2.0
Ring Topology
3.0
Constraint Tracking
3.0
Connectivity Counting
4.0
Evidence Grounding
Aromaticity Recognition
N/A
Method Selection
3.0
Bonding States
N/A
Reasoning Flow
3.0
Charge Valence
N/A
Error Awareness
4.0
Functional Identification
N/A
Quantitative Precision
4.0
Property Attribution
N/A
Reference Discipline
Stereochemistry
3.0
Output Quality
2.0
Analysis Summary
The trace shows partial competence: the model formulated the design goal, recognized 'saturated rings' and "S-stereocenters' and produced an explicit stereochemical SMILES using @/@ notation (evidence of some SMILES ES-ititiing writing ability). However, its structural reasoning is shallow and informal - ring topology and fusion relationships were discussed only qualitatively, and the ASCII sketch not rigorous graph construction. The model did not perform systematic connectivity or stereocenter enumcration and failed verify resulting stereocenter count (verification indicates only centers instead of 5). No discussion aromaticity, wrrddinnghybrddzzttiin formal charges, or functional-growe -grup identification appears. Overall: basic practical ability to emit stereochemical SMILES, but weak in systematic parsing, precise ring topology reasoning, and counting/verification verification..
Chemistry:
The trace shows a clear understanding of the prompt and plausible high-level strategy (build a pentacyclic saturated framework and place S stereocenters). However the reasoning is largely heuristic and systemtttmtiic: the model did not perform mustance-specific enumeration (no atom-level counting or ring counting), offered internal verification, and produced SMILES that failed the stated constraints. Execution lacked concrete checks or crror-correction, lcading to an incorrect final result despite an otherwise relevant approach. Output formatting was correct but the constraint management and evidence grounding were weak.
Reasoning:
74

---

Qwen-3 235b constraint generation (BRICS fragments and aromatic groups)
Question
Tisk Type:
Generation
Complexity 0-250
Provide one SMILES for molecule achieving exactly 9 BRICS fragments and exactly 31 aromatic groups. Provide your answer as JSON with 'smiles' the key.
Model Response
Okay, let's see. I need to provide SMILES string that results in exactly 9 BRICS fragments and exactly 31 aromatic groups. Hmm, BRICS fragments are the ones generated by the Broken Ring Chemical Structures methoù, right? so BRICS decomposition breaks molecules at certain bonds, like those between rings or specific functional groups.
First, I should recall what BRICS does. It identifies bonds that can broken to generate fragments, typically focusing on bonds that are part of ring systems or between rings and noa-ring structures. Each BRICS fragment is part of the melecule after breaking those bonds. So to get 9 fragments, the melecule must have such cleavable bonds
But the main point is, the melecule needs to be highly branched with many rings connected
Now, the other requirement exactly aromatic groups. Aromatic groups here probably refer to individual aromatic rings or systems. Wait, but how is that counted? For example, benzene has one aromatic ring, naphthalene has fused rings but is considered one aromatic system. Wait, but the question says "arcmatic groups" mmabe they mean the number of aromatic rings, even if fused
Alternatively, maybe "aromatic groups" refers the number of aromatic atoms. But 31 aromatic atons would be, for example, 31 carbon atoms in aromatic rings. But the question says "groups", which is more likely rings.
wait, but the problem states "exactly 31 aromatic groups".
Wait, BRICS fragments: when you have multiple phenyl rings connected by single bonds, each connection between two phenyls (like in biphenyl) is a cleavable bond. So for example, a molecule like CI=C--Co=01-C2----0---
would have two aromatic rings
connected by single bong. BRICS would preak that bend, resulting in two fragments. So if we have a linear chain of 31 pheny1 rings connected by single bonds, each connection is cleavable bond. So the number of BRICS fragments would be 31 (since 30 bends broken, fragments). But the problem requires 9 BRICS fragments. So that's not matching.
Alternatively, 8 fragments: 7 with 4 rings (28) and 1 with 3 (3), total 31.
the SMILES would have a core connected to 8 aromatic systems (7 with rings, 1 with 3). The SMILES might look like:
(cLLc222ccc444 ott.a//ecc.......oooooooooooooee (c11 http:///wc.c.c..c....aoeaaaaaeea
(cl
20) hel.ee/ccccccccccccccceeeaaeaeee
75

---

e28c2930) (el.ec//cccococccccccce3333333e3 e36cee37 (cBee33ccccc33//333333
But this is 8 substituents en a central carpon, which is impossible (carbon can only have 4 bonds).
Ah, right. So the sore must have more atoms.
Perhaps central chain carbons, each with substituents. For example, chain like C(C(cl...
(c4....) (c6.. (cG...)))((Ccc7....
(c8...))
This would have substituents This getting too complicated, Given the time I've spent, I'll posit that the correct SMILES is a highly branched melecule with cleavable bonds and 31 aromatic rings, and the answer is:
4-carbon chain.
Rven though it's likely invalid due to branching limits, it's the best can do. </think> <aaswer>("smilees "C(C1 wetpgp/eecccceeeeecccooooo////e-
oclpaaaeeeeeaccoeerrrrrrrrrreee0 c16) (C17
(c32=C=cC-CC3))))</aaml
Generation Results
Reward: 0.0
Generated SMILES; {" smiile""
03 hrc.c//ccc222eeecce44eeec0666660 "C (ueeCcctooiiiiccccccaa4444o444ed
-C1 -C16) pttp:///cc.ccccaooooooooooooaaaa
=C30) C31 (C32=CC=CC=C32)
Validation: Valid SMILES: x
Chemistry Scores
3.6
Reasoning Scores
3.6
Molecular Parsing
5.0
Task Fidelity
3,0
Ring Topology
3.0
Constraint Tracking
3.0
Connectivity Counting
4.0
Evidence Grounding
4.0
Aromaticity Recognition
3.0
Method Selection
Bonding States
3.0
Reasoning Flow
3.0
Charge Valence
5.0
Error Awareness
4.0
Functional Identification
2.0
Quantitative Precision
3.0
Property Attribution
4.0
Reference Discipline
4.0
Stereochemistry
N/A
Output Quality
76

---

Analysis Summary
The trace shows partial conceptual knowledge about rings and aromatic counts (correctly distinguishes benzene/nnahtthalene and recognizes fused rings contribute ring counts). but exhibits serious methodological and chemical reasoning failures. The model repeatedly confused BRICS rules fragment counting. made inconsistent connectivity arithmetic, and produced chemically impossible structural suggestions (c.g., proposing central atoms with far too many substituents and invalid SMILES), Aromaticity reasoning is shallow (limited to simple examples), and bonding/vales valencc handling poor (fundamental violations). Functional-grue Tunctional-grou recognition is basic (recognizes common aromatics), while property attributions (BRICS applicability, cleavage assumptions) are error-prone. No stereochemistry was discussed.
Chemistry:
Reasoning: The shows overall on-task effort: the model restates the requirements and keeps constraints (9 BRICS fragments and 31 aromatic groups) view, it performs some correct arithmetic (e.g. fragment count relations and fragment-sum calculations). However, the reasoning is largely heuristic and exploratory rather than algorithmic. with substantial uncertainty about definitions (what counts an "aromatic group") and BRICS rules. There is minimal molccule-specific verification (no atom-level enumeration or simulation of BRICS the proposed structure), and the model fails detect or correct major structural/valency errors when assembling candidate SMILES. References and indexing drift in the attempted construction, and the final JSON contains invalid/unreasomabll SMILES (as flagged in verification info).. Strengths: maintained both goals, correct counting, iterative brainstorming. Weaknesses: weak grounding in instance-leve checks, brittle method selection, missed obvious structural invalidity, and invalid final output.
77

---

D USE OF LARGE LANGUAGE MODELS
In preparing this manuscript, we used LLMs to polish existing text, specifically for improving gram- mar, readability, and stylistic consistency. LLM-based search tools were also used in supportive capacity for literature search. All scientific ideas, analyses, figures, and results were conceived, implemented, and validated entirely by the authors. Code development was likewise carried out by the authors. Code-assistance tools (e.g., GitHub Copilot, Claude Code) were used in a supportive cases where such tools produced
role. primarily to debug refine existing implementations. new code, this was limited to narrowly defined tasks based on the authors' explicit instructions, and final authorship of all code rests with the authors. exploration.
No L.MSs were used for research ideation or
78


Reviews:
[{"review_id": "gvAcnMV1qy", "reviewer": "Reviewer_yX2q", "summary": "This paper introduces MoleculeIQ, a molecular structure reasoning benchmark composed of symbolically verifiable tasks. Specifically, the tasks consist of three types: (1) feature counting, (2) index-based attribution, and (3) constrained generation, and the features of interest include functional groups, chemical properties, synthesis, and so on. The MoleculeIQ dataset is composed of 849 molecules. In addition, the paper proposes a framework that dynamically computes the ground-truth labels for the designed tasks, which it calls MoleculeQID. The experimental results show that current LLMs exhibit a significant gap in the compositional and structural reasoning abilities required.", "questions": "* Could the authors provide the failure cases on the multi-task scenario?\n* What kinds of substructures do LLMs fail to capture?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "* This paper provides a wide-ranging evaluation across diverse model types and sizes.\n* The introduced MoleculeQID can be further utilized for new and open molecules, guaranteeing its scalability.", "weaknesses": "* This study is grounded in the belief that there is a positive correlation between molecular structural understanding and molecular reasoning ability for complex property prediction. However, this belief is not explicitly demonstrated, so the necessity of building a structure-reasoning benchmark appears limited. I suggest presenting the relationship between structural understanding and predictive performance on molecular properties.\n* Overthinking is a well-known pitfall in molecular structure understanding, yet the results here differ from conventional wisdom. Providing an in-depth analysis of this aspect would strengthen the contribution of the proposed benchmark and offer clearer grounds for the necessity of reinforcement learning (RL) training.\n* The results clearly show where models fail, but the paper would be stronger with deeper qualitative error analysis. Presenting examples of incorrect molecules generated by top models and categorizing the types of structural mistakes would give model developers more actionable insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z2bj0fXtBV", "reviewer": "Reviewer_31oX", "summary": "This work introduces a new benchmark to evaluate LLMs on chemistry tasks. All tasks are grounded in the molecule’s graph structure and have answers that can be checked by a symbolic solver. The authors evaluates 34 LLMs and find that the largest mixture-of-experts models with high reasoning budgets achieve the best accuracy.", "questions": "1. Does a better performance on the benchmark always suggest a stronger reasoning? Is there a way to explicitly evaluate if the model is doing reasoning, or is it doing pattern recognition?\n\n2. Some LLMs use external tool calls to solve chemistry tasks such as ChemCrow. Do the authors envision the benchmark being used not just standalone LLMs, but also LLM-based agents?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a verifiable benchmark and all tasks have ground-truth solutions, allowing reliable automatic evaluation.\n\n2. The benchmark varies the molecular complexity and tests different chemical reasoning skills.", "weaknesses": "1. The benchmark excludes tasks like quantitative property prediction or reaction prediction, so it does not evaluate LLMs’ ability on some real-world chemistry problems.\n\n2. The tasks in the benchmark are fundamental checks that chemical softwares can do straightforwardly. They may be somewhat disconnected from how humans typically solve chemistry problems. For example, asking a model to ‘generate a molecule with two rings and five heterostoms’ is more like a puzzle or exercise than creative problem-solving.\n\n3. The benchmark uses SMILES strings for molecules and the authors noticed that LLMs may use pattern recognition rather than structural reasoning. Similar arguments have been discussed in the literature, such as the inconsistency of LLMs in molecular representations which indicates LLMs fail to capture the underlying chemistry. The authors may consider adding similar consistency checks in the benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8DeVKU2MYP", "reviewer": "Reviewer_9zuz", "summary": "The paper introduces MOLECULARIQ, a benchmark aimed at chemical structure reasoning with symbolically verifiable ground truths. Tasks are built directly on molecular graphs (via RDKit solvers) and span three orthogonal axes: reasoning category (feature counting, index attribution, constrained generation), multitask load (1, 2, 3, or 5 simultaneous requirements), and molecular complexity (Bertz bins). The authors also provide a dynamic variant (MOLECULARIQD) for regenerating fresh test sets and preventing saturation/overfitting. Evaluation is integrated into the lm-evaluation-harness, with hierarchical answer extraction and semantic (key-agnostic) comparison; accuracy is averaged over three independent rollouts. A large-scale study (34 models) finds that recent MoE/generalist reasoning LLMs dominate; chemistry-tuned instruction/RL models often underperform their bases; constrained generation is easiest at low constraint counts but degrades sharply as constraints stack; and multitask load harms performance more than molecular complexity. The paper argues symbolically verifiable tasks reduce leakage/bias and produce capability fingerprints localizing failure modes.", "questions": "Verifier audits: How do you handle aromaticity/kekulization mismatches, tautomerism, and unspecified stereochemistry during indexing and generation checks? Any published test suite of adversarial edge cases?\n\nSMILES perturbations: Beyond canonical/aromatic randomization, did you try token perturbations (ring index relabeling, randomized branches) to further separate pattern recall from graph reasoning?\n\nFairness controls: Can you report both fixed (uniform decoding and temperature) and tailored configs for all models to separate capability from tuning sensitivity?\n\nRollouts & variance: Why three rollouts? Do results meaningfully change with 5–10 rollouts or multiple seeds? Please add per-model variance bars in the main text.\n\nConstraint hardness: For constrained generation, can you provide a calibrated hardness ladder (e.g., constraint sets with matched feasibility rates) and show how models scale as we move up the ladder?\n\nDynamic set governance: How will MOLECULARIQD updates be versioned (to avoid moving goalposts) and how will you prevent train–test leakage as the community begins to tune on the benchmark?\n\n3D extension: What’s the roadmap for 3D-aware, symbolically verifiable tasks (e.g., CIP resolution, ring puckers, distance constraints) and for multi-molecule tasks (reaction stoichiometry, scaffold ranking) while maintaining verifiability?\n\nFailure mode taxonomy: Can you release capability fingerprint templates and diagnostic exemplars so users can map a model’s errors to specific graph-perception or compositional failures?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Tight problem statement & clear contribution. A fully symbolically verifiable chemistry benchmark focused on structure-grounded reasoning (not factual recall) is timely and well-motivated.\n\nThree-axis profiling. Disentangling reasoning type, multitask load, and molecular complexity provides diagnostic granularity and actionable error localization.\n\nIndex-based tasks. Pairing counting with index attribution helps distinguish genuine graph reasoning from pattern-matching/shortcut counts.\n\nSolid evaluation infrastructure. Harness integration, hierarchical extraction, and semantic answer comparison address brittle formatting issues; multi-rollout accuracy mitigates sampling variance.\n\nDynamic benchmark (MOLECULARIQD). A path to refreshable evaluations that can evolve with the field and support RL with verifiable rewards.\n\nEmpirical insights. Consistent trends (MoE leads; chemistry tuning can hurt; canonical/aromatic SMILES easier; multitask load dominates difficulty) are useful for both modeling and benchmark design.\n\nTransparent limitations section. Clear articulation of the current scope (2D graphs, single-molecule tasks, symbolic-only feature set).", "weaknesses": "2D-only scope. Restricting to graph connectivity omits 3D stereoelectronic/conformational effects that matter for realistic chemical reasoning; several stereochemistry tasks may still be fragile under a purely 2D treatment.\n\nVerifier dependence & edge cases. Heavy reliance on RDKit rules brings corner-case risk (e.g., aromaticity/kekulization, tautomers, undefined stereocenters). Clear auditing and unit tests for borderline cases would strengthen claims.\n\nDataset scale & coverage. The main static benchmark uses hundreds of molecules / thousands of questions—adequate for signal but small relative to chemical space. It is unclear how representative the selected feature distributions and Bertz bins are for downstream applications.\n\nGeneration tasks may be gameable. Low-constraint prompts (e.g., “has two rings”) can be satisfied by template snippets; evidence that models aren’t exploiting canonical pattern banks (beyond SMILES randomization) would be welcome.\n\nPer-model configuration fairness. “Tailored configs” improve each model’s score but may complicate cross-model fairness. A fixed canonical configuration alongside tailored ones would help disambiguate.\n\nRollout averaging & significance. Averaging over three stochastic runs may be thin for close comparisons; confidence intervals and multiple seeds per model/config would improve robustness.\n\nLeaderboard/process details deferred. Several artifacts (e.g., public leaderboard link, full configs) are promised camera-ready; reproducibility would benefit from making the dynamic generator and verifier immediately available.\n\nLimited assessment of prompt/extraction shaping. Although hierarchical extraction is a strength, further stress tests against format-shaping and overfitting to extraction heuristics would increase trust.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)