Unleashing Scientific Reasoning for Bio-experimental Protocol Generation via Structured Component-based Reward Mechanism
Authors: Haoran Sun
Yankai Jiang
Zhenyu Tang
Yaning Pan
Shuang Gu
Zekai Lin
Lilong Wang
Wenjie Lou
Lei Liu
LEI BAI
Xiaosong Wang
Venue: ICLR 2026 Poster (2026)
Keywords: AI for biology
Protocol Generation
Scientific Reasoning
Large Language Model
Reinforcement Learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We introduce SciRecipe, a large-scale protocol dataset, and the SCORE mechanism with the “Sketch-and-Fill” paradigm, enabling Thoth to generate precise, executable scientific protocols that surpass existing LLMs.

Abstract: The foundation of reproducible science lies in protocols that are precise, logically ordered, and executable. The autonomous generation of these protocols through natural language queries could greatly improve the efficiency of the reproduction process. However, current leading large language models (LLMs) often generate incomplete or inconsistent protocols, limiting their utility. To address this limitation, we first introduce SciRecipe, a large-scale dataset of over 12K structured protocols spanning 27 biological subfields and encompassing both comprehension and problem-solving tasks. To further improve protocol generation, we propose the "Sketch-and-Fill" paradigm, which separates analysis, structuring, and expression to ensure each step is explicit and verifiable. Complementing this, the structured component-based reward mechanism evaluates step granularity, action order, and semantic fidelity, aligning model optimization with experimental reliability. Building on these components, we develop Thoth, trained through a staged Knowledge-to-Action process that progresses from knowledge acquisition to operational reasoning and ultimately to robust, executable protocol generation. Across multiple benchmarks, Thoth consistently surpasses both proprietary and open-source LLMs, achieving significant improvements in step alignment, logical sequencing, and semantic accuracy. Our approach paves the way for reliable scientific assistants that bridge knowledge with experimental execution.

UNLEASHING SCIENTIFIC REASONING FOR BIO- EXPERIMENTAL PROTOCOL GENERATION VIA STRUC- TURED COMPONENT-BASER REWARD MECHANISM
Haoran Sun Yankai Jiang!" Zhenyu Tang 1.3
Yaning Pant.2,
Shuang Cu Lei Bai', Xiaosong Wang lss
Zekai Lin", Lilong Wang', Wenjie Lou', Lei Liu?,
"Shanghai Artificial Intelligence Laboratory Fudan University
Shanngaai Jiao Tong University
{ [jiangyankai,
ABSTRACT
The foundation of reproducible science lies in protocols that are precise, logically ordered, and executable. The autonomous generation of these protocols through natural language queries could greatly improve the efficiency of the reproduction process. However, current leading large language models (LLMs) often generate incomplete or inconsistent protocols, limiting their utility. To address this limi- tation, we first introduce SciRecipe, a large-scale dataset over 12K structured protocols spanning 27 biological subfields and encompassing both comprehen- sion and problem-sole solving tasks. To further improve protocol generation, we pro- pose the "Sketch-ane poeechhann--ae aanh--nnd-Fiii Fill" paradigm, which separates analysis, structuring, and expression to ensure each step is explicit and verifiable. Complementing this, the structured component-basee reward mechanism evaluates step granularity, tion order, and semantic fidelity, aligning model optimization with experimental reliability. Building on these components, develop Thoth, trained through a process that progresses from knowledge acquisition to operational reasoning and ultimately robust, executable protocol generation. Across multiple benchmarks, Thoth consistently surpasses both proprietary and open-source LLMs, achieving significant improvements step alignment, logi- cal sequencing, and semantic accuracy. Our approach paves the way for reliable scientific assistants that bridge knowledge with experimental execution. Code publicly available at https::/githuu..com//t http:///whhuu.cooo/////m///////o
staged Knowledge-to-Actinn
INTRODUCTION
The planning and execution scientific experiments hinge on protocols that serve not merely as textual guidelines but as operational blueprints detailing procedures, materials, and logical depen- dencies (Freedman al. 2013) Goodman al. 2016). well-structured protocol ensures that experiments are reproducible, safe, and scientifically valid (of Sciences et al. 2019). which is es- sential for cumulative progress in the life sciences. While recent advances large language models (LLMs) have greatly expanded their role in biomedical research, ranging from literature-based dis- covery to domain-specifit question answering (Devlin et al. [2019] Brown et al.) 2020 Lee et al.) Beltagy et al., 2019), their ability produce reliable experimental protocols remains under- developed. Existing datasets and benchmarks are typically confined to comprehension tasks (Liu) al. 2025 Jiang et al. 2024), thereby neglecting the dimensions of planning and problem solv- ing required to support reproducibility and practical execution. As result, researchers often find that models offer fragmented recommendations experimental procedures but fall short of produc- ing concise, logically ordered protocols that can be directly implemented in laboratory workflows
2020
(0 Donoghuu al. 2023
Yi et 2024).
"Equal Contribution. "Cooresppoding authors.

---

Currently, proprietary models (Achiam et al.) 2023 Comanici et al. 2025 Jaech et al., 2024 OpenAI 2025) such as GPT-5 demonstrate strong capabilities in procedural reasoning. while domain-specifte scientific systems complement LLMs with curated knowledge bases tool-assisted pipelines (Huang et al. 2025 Jin et al. 2025). Despite these advances, the generated protocols of- contain unordered steps, redundant operations, factual inconsistencies, hallucinated actions (0 O'Donoghue et al.) [023] Yi et al. 2024, Jiang et al. 2024), undermining both reproducibility and scientific credibility. Moreover, evaluation remains central bottleneck: metrics such as BLEU (Pa- pineni et al.. 2002), ROUGE (Lin 2004), and BERTSScore (Zhngg et al. 2019) capture only superfi- cial lexical overlap. failing to reflect whether the generated action sequence logically consistent, semantically faithful, and practically executable. While "LLM-as-a-judge" frameworks align better with human preferences, they introduce prohibitive costs to reinforcement learning (RL) pipelines, limiting scalability (Liu et al., 2023; /Zheng et al.) 2023). Existing reward designs further neglect the structured and verifiable nature of protocols, producing outputs that are linguistically fluent but experimentally unreliable (Zeng et al. (023). These limitations highlight the need for an approach that integrates structured and efficient evaluation.
In this work, we introduce a comprehensive framework that advances both data and modeling for protocol generation. At its core lies the "Sketch-and-Fill paradigm, which formulates protocol gen- eration a structured reasoning process: each step is decomposed into essential components and expressed in natural language with explicit correspondence, ensuring logical coherence and exper- imental verifiability. To support this paradigm, we curate SciRecipe. large-scale dataset of over 12K protocols spanning diverse domains and covering both Protocol- ComprececCom and Problem- Solving tasks. Building on this foundation, we propose the Structured COmponent-bass (SCORE) mechanism, the central innovation of our framework. SCORE provides a structured re- ward and evaluation scheme that captures three complementary dimensions: step granularity (con- trolling scale and avoiding redundancy), action ordering (ensuring logically consistent sequences for reproducibility), and semantic fidelity (verifying alignment between predicted and reference actions, objects, and parameters). By jointly modeling these dimensions, SCORE moves beyond conventional text-based metrics to directly assess whether protocols are executable, interpretable, and scientifically sound. It serves both as effective RL training signal and as reliable metric for evaluation. Based on these components, we develop Thoth, a pettocol-generatioon model trained to combine structured reasoning with SCORE-guider evaluation. Extensive experiments demonstrate that Thoth outperforms SOTA models, particularly in step alignment, logical sequencing, and se- mantic accuracy. Just as importantly, the protocols generates are concise and reproducible, which are qualities often absent from existing systems. Our main contributions are as follows:
REward
We curate SeiRecipe, a large-scale, multi- dataset covering over 27 biological subfields, designed serve foundation for both training and evaluating on protocol generation.
We introduce the Sketch-and-Fill paradigm, reasoning framework that aligns with the logic of experimental design by converting open-ended queries into verifiable protocols.
We propose the SCORE mechanism, structured reward and evaluation framework that jointly measures step granularity, order consistency, and semantic fidelity, ensuring that protocols are not only linguistically fluent but also experimentally executable.
We develop Thoth, a achieves SOTA performance across protocol-specifcc and broader scientific benchmarks.
model with strong reasoning abilities, which
RELATED WORKS
LLMs in the Life Science Recent progress in life science LLMs has shifted from general- -prppos models. BioBERT and SciBERT (Lee et al., 2020 Beltagy et al. 2019). pretrained on large-scale biomedical corpora, substantially improved tasks such as information ex- traction and question answering (QA). Autoregressive models like BioGPT (Luo et al.) (022) fur- ther advanced generative abilities, achieving strong and consistent results on PubMedQA (Jin et al.) 2019). These works collectively demonstrate the effectiveness of domain-specific pretraining and task adaptation, yet remain limited to knowledge- bbsed tasks (e.g.. QA, summarization) and cannot produce executable protocols. Related
systems to domain
goal, BioPlanner (0 DoDogghe et al., (022) proposes pseudocode- based evaluation framework for assessing biological protocol planning. While valuable for benchmarking L.M reasoning, it focuses pseudocode reconstruction rather than generating
2

---

natural-language, experimentally executable protocols, making it complementary to work. In
contrast, proprietary models (OpenAl 2025| Comanici et al. 2025 Jaech et al. 2024) such as GPT-
5 leverage massive corpora, parameter scales, and reasoning mechanisms to generate preliminary
multi-step protocols. More recent efforts (Huang et al.) 2025 Jin et al. 2025: Zhang et al.) 2025c
Gao et al. 2025), including Biomni and STELLA, integrate external knowledge and tools sup- port task-oriented solutions. Nonetheless, current models still struggle with redundancy, misordered steps, and hallucinations, thereby limiting their practical usability. To address this gap, we propose a biologically oriented model trained on real protocols, designed to generate rigorous experimental procedures through a structured scientific reasoning paradigm.
[+]
The protocal mis gel oxhrdding prosix with 25 pl. of 10% sencniun preszifate for brain elice, how should ocale the and what volomes of APS and TEMED should use?
Drden-S
2.5 TEMED If only
Thoth
Step CCaction: "eooaree "objecte" l'gel embreding provic]] "paranttare" ["1 3
Step ("action': "ade" "obyects" (1 0% aemonium parcalfate "paraneters": (1S prD
"Concise and Logically Ordered
Step ("action': add" "oby ebyecte' [ttaeed"] "paranetere:: (M0.5 ((1)
Step ('action': was. "ohects': exbeddnng solution) "paranaters:: ('Hoorighhy
Semantic
ChanSPT-4o
Step ('actios "ransurn" hobjette I'gel anthedinng pevorie) J. "paconeterr (11 n'D
Step (action" "ransurrn "objecte]] anveonur permilate" "caranstere"" 21S pT
"Aiction Miceaach"
Step aaction "add"" "obyecte'; "tamed' 'paraneterr'' 25 JlI "Factudl Incorsistenca)
BLUB-AYS
RETZOR
Step "lcction! "ra "chects'ss "ell arbending premx', " arvoniium parsalfate" "taredf paramstera""
['until havogsneouul 1]
Execctional Inaccuracies"
ReugeL
Raw Protocols
Extraction & Cleaning
Structuring
Protocol Metadata
Construct
Check
Protocal.
coo-protoool
MinerU
[BagSxij Rule
exp navi: abstruct:
Comprahamalon
purvies sand
Fornat Check &
Repair Modile
Sg protocols.io
Data Fittering
xaterrials:
Sie ScientiFo
Dedkslication
baxed
equiprennts:
proredures
plaking
natureprooccoss
Export Restima
Quality Control
Problen-Soliving
Expert Resteu
Figure 1: (a) Thoth's performance advantages over other models, shown through qualitative evalua- tion (left) and quantitative evaluation (right). (b) The construction pipeline of the SciRecipe dataset.
Reward Design and Evaluation for Open-Ended Generation Designing effective evaluation mains a key challenge in open-ended generation. Unlike short-text tasks with clear ground truth, long-text generation such experimental protocols lacks definitive answers and must therefore be assessed along multiple dimensions, including coherence, factuality, and executability (Becker et al.
2024 Que al.] (2024). Metrics such ROUGE, BLEU, and BERTScore (Lin) 2004 Papineen et al,
2002 Zhang et al. 2019) capture surface similarity but fail to fully reflect reasoning depth and often
diverge from human judgment. The L..MMaas--juudee paradigm provides closer alignment with
human preferences (Gu et al. [024]] Zheng et al. 2023). yet it usually introduces prohibitive com- putational costs in RL training (Shao al. 2024] Schulman et al.) [2017; AAhmadian et al. 2024 Hul 2025). Alternatiee approaches, including smaller evaluator models or reward mechanisms such as PretBERT, Direct Reasoning Optimization, and LoVeC (Li al. 2025 Xu et al. [205 Zhang et al,,
2025a), can improve efficiency or reliability but still require repeated model calls. To address these issues, we propose a rule-based evaluation that extracts key protocol elements through structured reasoning, thereby offering direct and verifiable reward signals with efficiency and interpretability.
METHODS
This section outlines our approach to leveraging high-quality experimental protocol data and a component-wis reward mechanism to enhance the model's ability to generate experimental so- lutions. We first propose the "Sketch-and-Fill" reasoning paradigm, which aligns with experimental execution logic to guide the model in organizing experimental steps effectively. Building on this, we construct SciRecipe, dataset encompassing diverse experimental tasks and scenarios. Subse- quently, we develop the SCORE mechanism to evaluate protocol quality across multiple dimensions, serving as a reward signal to improve the model's 's generalization in protocol generation tasks. Ulti- mately, by leveraging SciRecipe and a multi-stage Knowledge-to-Acoe learning strategy, we train the Thoth model, which exhibits strong reasoning capabilities for protocol generation.
3

---

3.1
SCIRECIPE DATASET
In life science research, protocols serve as essential documents to ensure experimental reproducibil- ity and reliability, providing detailed records of materials, equipment, procedures, and critical notes. Recently, platforms such as Nature Protocols, Bio-protocol, and Protocols.io io (Wikipedia contribu- tors 2025: Bio-protocol, LLC 2025 Teytelman et al., 2016) have compiled vast number of stan- dardized workflows, offering rich resources for the scientific community. From these sources, we collected over 23K protocols spanning 27 subfields including neuroscience, molecular biology, and cancer biology. After cleaning and structural processing, approximately 12K high-quality data were retained as the foundation our dataset (the preprocessing pipeline is detailed in the Appendix A.I. Building on this foundation, we introduce the SeiRecipe dataset, designed to improve and evaluate LLMs in experimental protocol understanding and generation. SciRecipe comprises eight task types, grouped into two categories: targeting global summarization and fine-grained analysis, and Problem-Solving Tasks (retrieval, planning. troubleshooting, constraint, scaling, and safety), simulating typical challenges encoun- tered throughout experimental workflows. Together, these tasks form a complementary "understand- ing-application" loop. The construction pipeline of SciRecipe is illustrated in Figure [ and further elaborated in the Appendix [A.4] Notably, benchmarks focused on scientific experimental protocol generation are currently scarce (Liu et al. 2025). To address this gap. we introduce SeiRecipe-Eves built with a similar pipeline, with dataset splits and difficulty levels detailed in the Appendix B
Comprche-Cnn Tasks (overview and specific),
3.2 "SKETCH-AND-FILE REASONING PARADIGM
To transform protocol generation tasks into an executable and evaluable form, we propose the "Sketch-and-Fill" reasoning paradigm. This paradigm is centered on structured three-stage out- put: <think> + <key> + <orc>, with an additional <note> section dedicated to laboratory safety precautions. The core idea is organize outputs in the sequence of reasoning, structur- ing, and expression, thereby enabling the design of a structured reward mechanism for RL. An overview of this paradigm is provided in Figure 2 Specifically, <thi nk>, <key>, and <orc> denote the reasoning process, the extraction of key information, and the final natural language out- put, respectively. In <think>, the model decomposes sub-goals, identifies sequential dependen- cies, and justifies the necessity of the proposed experimental steps, ensuring that the protocol is grounded in scientific reasoning. The parsed steps are then organized through a two- two-phase process of "Sketch" and "Fill." In the "Sketch" phase, represented by <key>, the strategies from <think> are transformed into a sequence of atomic, machine- readable steps. Each step constrained to verb, "objects":
a single JSON dictionary representing one action unit: ("action": [...]]. This abstraction reformulates natural language steps into cebficat--objet--dvvete triplets, bridging free-form instructions with structured sequences. The subsequent "Fill" phase, represented by <orc>, expands these steps into fluent natural language instructions, ensuring readability and executability. Formally, <key> is defined Y:
[...], "parameterss
(y1.-..,y') .ym), Y: (a;; P.)),), where a; the experimental operation, O2 denotes the objects being acted upon, and P. speci- fies the parameters (e.g., temperature, concentration). Consistency constraints, such enforcing Aune-Actio-Per-Steee
and maintaining uniform parameter application across objects, are applied to standardize the data format. In the "Fill" phase, represented by <orc>, the elements of <key> are rendered into human-readable natural language. A strict one-to-one correspondence in step count and semantics is enforced, ensuring no information is added or omitted, with the focus solely on readability. Overall, the "Sketch-and-Fiil" paradigm standardizes scientific protocol generation by grounding open-ended language output in executable structural space. This not only supports stable RL training but also provides a consistent foundation for automatic evaluation.
3.3
SCORE MECHANISM
3.3.1
OPTIMIZATION OBJECTIVE
In scientific research, experimental protocols are not merely narrative texts but highly structured action guidelines refined through long-term practice. A qualified protocol must describe steps com- pletely while conveying experimental logic: what to do (action), on what objects (objects), and under what conditions (parameters). Traditional text generation metrics (e.g., ROUGE, BLEU,
4

---

Can Jun the muin
Rollout
worktloe for propartng sell cofture extracte aconitace te-ge plases only to proten spantiiiaation
sitge.
Gate
Concictancy Gote
-key Pansing
Antion aox Parseettrn: 11
Query
soric...
Rocord parsodaa
Key--Orn aligueent
"Seetch-and F
Rollost
Rollost
Rollout 3
Policy Model
Reasoning Paradign
Soep 1: Ournatt cultures, laffer, col))
foantringe
Step Phareord, lets settrr
Step (arvast,, pellats in water, resuapara)
"Hanwor" "Seain
"Ong"
pollat,,
-Bys,, pollets, bol,
Action Library
"Reeeppeed
"Bes"
"yya""
Step S: (quertity, proton sntract, ussegl
Step fittii,, odl peliets, des)
Step flon, cell pollat,, boll de Step foentrriggg, lysetu rexupaand?
Scalk
Staps
f Stope
Staps F
4 Staps
siseatch
natcc
mismatch
Ground Truth
Order fanvect centellggn Jann clair csanctly)
b harvast, qardit, yos, centr Page
(harvoot lqpa, queebfuu
Action Order:
Step
[harwaat lyne, cantrifuus, ssurtifl
[herwsst fyon, contrinte contrrfgge quartiful
[hansect, kyon, cenorliin, subeeitgl order Inconaletency
kyph, subeequencs
Sementio:::
intonsisteney
Step 1: Darneat, call cuttures, wish 3a, pls, 4 Semantics
Aigned ovtion paine [1 31,5,
Aligned action pairs: (17 12,
Aigned astion pairs: 13, 13, 13
Sezter
Step 2: (1ydo, eell pollett, triton citvato baffer.
dla ditrate bullee, "x3
(cantritag,, lguate, 116,000ks,
lusata
Secteon
Gel pullets
Stop (parnilt catment, accudly
canartice algrmmnt
sumeecicc Inconalatanoy
semertica partial conalitaanyy
signal
Combine
Semsrttica)
More
Figure 2: Illustration of the SCORE mechanism and the "Sketch-and-Fill" reasoning paradigm. Three representative rollouts are subjected to concise analysis across step scale, order consistency, and semantic consistency, while detailed computations are provided in Section 3.3.2
BERTSccore) often overlook such structure, rewarding lexical overlap even when action sequences are disordered. To address this, we propose the SCORE mechanism. Its optimization objective emphasizes both local alignment of protocol content and global logical consistency, defined as:
max (a..AA.))) J-D E. (-
Ruoeal V3) @ Rylohally, v²)
(2)
which defines No as the policy model with parameters 0, as the context, A as the action library, and y" the ground-truth protocol. Riocal and Rglobal denote reward functions at differrnnt granularities, combined by & (details in Section 3.3.2). The core idea is to shift evaluation from superficial similarity to multidimensional alignment, including logical order and execution precision. Unlike metrics focusing only on text overlap or embeddings, SCORE directly measures the operability of generated protocols, providing scientific, interpretable, axprriment-aligneed optimization signals.
3.3.2 REWARD DESIGN
First, a gating stage ensures basic structural and consistency requirements. Then, fine-grained rewards for step scale and step semantics are combined into the final reward signal. This approach encourages the model to fol- low rational reasoning paradigm and enhances both coherence and executability of the generated protocols. Format Gate: The output must contain the four sections <think>, <key>, <orc>, and <note> in the correct order. Each step in <key> must follow the format Step x: {ison),, explicitly specifying act i on, objects, s, and pa parameters ensuring parseability for subsequent reward calculation. Consistency Gate: This gate verifies step-by-step correspondence between <key> and <orc>, Every action, object, and parameter in <key> must appear in <orc> with at least 95% coverage (see Appendix C.I for details), ensuring the protocol practical guide rather than hollow framework. Only protocols passing both gates are used for reward computation.
The SCORE mechanism adopts a progressive design (see Figure 2}.
The quality of protocol depends on proper granularity, as too few steps cause omissions while too many lead to redundancy. We design step scale reward that measures the gap between generated and gold step counts, with one atomic action per step guaranteed by the "Sketch-and-Fiin paradigm. Matching counts yield score of 1, and deviations are penalized by cosine decay that drops to zero beyond threshold. length penalty also applied, and when the average word length per step exceeds a limit, the score is proportionally reduced to discourage verbosity. Formally:
Step Scale:
f(d)
md cos
0.
ZN
<
M f o(i)
L,
1. > L }
S
)
Taale 1o1N Nootal)
g(L)
(3)
5

---

where Npred and Ngoia are predicted and gold step counts, and M max(1, g0.6 is the deviation threshold. I, denotes the average text length per step. f(d) penalizes step mismatch via cosine decay, while g(L) penalizes verbosity. Together, they yicld the step scale reward Tscale-
Step Semantics The step semantics reward is central to SCORE, as it reflects the core execution logic of protocols. It consists of two components: Order Consistency and Semantic Consistency.
a) Order Consistency, denoted as Order(.), assesses whether the generated sequence of experi- mental actions matches the ground truth. We adopt a "Strict" mode (see details in Appendix (C.2) rewarding only if the predicted and ground-truth sequences identical mutually subsequences, and zero otherwise. This design mirrors laboratory reality since some steps may he repeated or omitted, but disordered sequence renders the protocol invalid regardless of textual similarity.
b) Semantic Consistency is then calculated on the basis of action alignment, since actions function the anchor of experimental steps (see Appendix C.3). In practice, researchers first decide what operation to perform (e.g., "incubate", "add"), and only then specify the relevant objects and condi- tions. Without the action anchor, textual similarity cannot guarantee executability. Thus, we align predicted and gold actions step by step. For each aligned pair (i.j), the overlap of object sets is measured using interseetion-over-unioo with subword-bare objects differ but are semantically related. Parameters are compared only if object overlap 2 0.5, reflecting the principle that condition descriptions are meaningless for incorrect objects. Formally:
similarity used as compensation when
P 2, =
O no Obj(i,)) O UO
Pa
O, IKCPL [CPP))K)()]]]
P. = P VV
(4)
otherwise
where O and P3 are the predicted object and parameter sets at step Oj and Pj the corre- sponding ground truth sets at step and XC(-) extracts subword sets for reliable comparison. To better incorporate positional fidelity, we apply a decay factor Maj max (0. - (|i - jl/D)* D)^), where D is the number of ground truth steps. This reduces the score when predicted actions, though correct in type, appear far from their reference positions. The final score is:
T Tsemantics = Order(a, a²) +
1
Σ m.y (O.j(Ob(ii))) + ( Par(i,,))
(5)
DW
(K.))W
where W the set of aligned action pairs. Unlike the step scale (equation 6b. we adopt an additive combination to avoid over ver-penalization. This allows partial credit when objects or parameters are imperfectly matched, as long the action sequence remains reasonable. Such tolerance improves training stability and aligns with how real researchers judge protocols, where minor detail errors may be acceptable if the experimental logic preserved. Finally, the overall SCORE is defined as:
SCORE(y, v/") = Iforma(()) (1) llcons (v) : TTscate 3²) *'semantics (y, y²),
(6)
where Iformat and Ilooss are gating functions. This design ensures that only protocols satisfying struc- tural, step scale, and semantic requirements obtain high rewards Wang et al. 2025). thereby mitigat- ing reward hacking and more faithfully simulating how protocols are evaluated in practice. SCORE also serves as a multidimensional evaluation framework, with details in "Evaluation Metrics" of Section 4.
3.4 KNOWLEDGE-TON TO-TIIO LEARNING STRATEGY
Inspired by curriculum learning (Bengio et al. 2009 Wang et al. 2021). we propose a three-stage Knowledge-to-Actien Learning framework that progressively enables the transition from textual knowledge to protocol generation. The framework parallels human learning, progressing from knowledge accumulation to standardized operations and finally to exploratory optimization. In the first stage, pre-training, the model learns the semantic structure and operational logic of experimen- tal language from large-scale protocol texts (see Appendix J.2 for further experiments). The second stage, supervised instruction tuning (SFT), is conducted on data following the "Sketch-and-Fiin paradigm, incorporating subtasks such as parameter filling, step ordering, and error correction (Liu et al. 2025). SFT both injects domain knowledge and provides a cold start for RL, aligning outputs with the designated paradigm. The third stage applies RL with the GRPO algorithm (details in Ap- pendix D & J.1.1) (Shao al. 2024). By removing entropy loss and reducing the KI. penalty,

---

enhance exploration and avoid premature convergence. Combined with SCORE rewards, this stage improves generalizability and robustness, ensuring more reliable and executable protocol generation.
EXPERIMENTS AND RESULTS
4.1 EXPERIMENT SETTINGS
Benchmarks & Baselines To evaluate our model, we adopt two types of benchmarks. The first fo- cuses on real-world protocol generation, represented by SciRecipe-Eval, which emphasizes linguis- tic quality and practical executability. The second covers broader scientific reasoning and question answering across domains, including Humanity's s Last Exam (HLE), LAB- BBB-ennch, and PubMedQA (Phan et al. 2025 Laurent et al. 2024 Jin et al. 2019). (Additionally, BioProBench sub-tasks (Liu et al. 2025) are used as supplementary experiments for protocol comprehension.) We further com- pare against proprietary, open-source, reasoning, and scientific LLMs (see Appendix E for details).
Implementation Details We used GPT-5 Chat (OpenAl) 2025) (temperature 0.6) to construct SciRecipe and Gemini 2.5 Flash (Comanici al.) 2025) (temperature 0.2) for validation. In the SCORE mechanism, hyperparameters L = 30 and X
1.5 were determined through iterative experiments. For training Thoth, we adopted Qwen3-8B as the base model YYngg et al., (025). Pre- training and SFT used LoRA fine-tuning via LLaMA-Factun parameter tuning was conducted with the VeRL framework (Sheng al.] 2025). All experiments ran on eight Nvidia H100 GPUs. Further hyperparameters and data ratios are given in Appendix F
(Zheng et al. (024)) while RL full-
4.2 EVALUATION METRICS
We designed metrics tailored to different task types. For natural language generation, BLEU, ROUGE, METEOR, and keyword matching (Liu et al. 2025) were used assess surface-level semantic similarity (Papineni et al., 2002 Lin 2004; Banerjee Lavie 2005 Grootendorsi 2020). Based on the SCORE mechanism, (Step-M), Order-LSC/S/Tauu and Monnantic-Alinmmeen (Semantic-A), to evaluate step scale, order, and semantic fidelity For multiple-choice, classification, and ranking tasks, employed Accuracy, F1, Exact Match, and Kendall's Tau (Abdi) (007).
introduced five executability metrics, namely Step-MATCH
Although SCORE originally introduced as reward mechanism for reinforcement learning, its structured design naturally yields a set evaluation metrics for protocol generation. Because each step protocol is decomposed into an action, an object, and a of parameters, the resulting met- rics allow the assessment of executability and structural fidelity beyond surface-level text similarity.
Formally, let the predicted key-step sequence be $ (81, ... 8n.) and the reference sequence be s* = (si...., ),.), with corresponding action sequences à and a". Let W = (x) } denote the aligned action anchors (See Appendix C.3). Based on this structure, we introduce five metrics.
Step-M measures step-level completeness through strict length matching:
Step-M(s,s*) = Ill | = =ss*]].
Order-S enforces exact sequential agreement of actions:
Order-S(ā. a²) I a*).
(8)
Order-LCS provides more tolerant measure based on the normalized longest common subse- quence:
2 a(,,a"))
Order-LCS(á, a²)
+
Order-Taar evaluates the relative order of aligned anchors using a Kendall-style correlation:
C D'
Order-Tau(W)-
(10)
C D
where C and D' denote concordant and discordant anchor pairs, defaulting to 0 when + D' 0.
Finally, Semantic-A corresponds to the semantic consistency described in Section 3.3.2 It evaluates object correctness, parameter fidelity, and positional consistency over the aligned anchors, providing structured measure of semantic accuracy and executability.

---

4.3 MAIN RESULTS
Table Main results on SciRecipe-Fiee Metries left of the dashed line evaluate executability, those on the right measure lexical similarity. Bold denotes the best score (see Appendix 1.2 for details).
Methods
Semantie-A
Order-LCC
Order-S
Order- Tan
Step-M
BLUE-AVG
ROUGE4
METEOR
KW-FI
AYG
ChatGPI-40
40.04
73.27
24.00
70.33
44.00
38.95
48.42
44.06
52.05
48.41
GPT-5
27.79
58.12
11.35
53.55
18.79
21.31
32.96
32.55
39.17
32.84
29.57
Claude Sonnet 4
39.35
71.97
20.83
70,00
35,83
34.24
44.27
80.97
49.40
45.21
Opns Gemini Flash
36.35
70.61
20.00
70.33
32.33
33.19
42.91
39.26
48.07
43.62
Gemini 2.5 Pro
35.80
72.68
21.83
70.17
32,00
31.37
44.16
45.59
18.58
14.69
Doubao-1.5-pr
40.34
72.88
21.83
71.33
47.50
30.81
48.02
43.82
51.98
47.61
Open-Sourzz
Owem2.5-72-Immiruut
36,40
70.82
21.00
69.17
42.17
29.55
46.06
43.44
49.73
45.37
Swem3-2391-A22--1mmsa
35.68
72.07
20.03
69.12
37.73
32.48
44.37
44.30
47.89
44.85
DeepSeck-V3
GPT-05S-1208
32.86
69.97
17.67
64.17
27.83
30.72
43.44
42.00
49.92
42.
Llama-3.1 3.1-105B-Insterr
35.92
69.46
18.00
67.78
39,23
36.42
44.86
18.09
14.66
Kimi-K Sami--22-nssraation
36.99
20.83
69.83
40.00
33.81
44.49
42.99
49.00
45.53
Recsoning Models
DeepScek-R1
45.86
44.83
Grok 3
37.40
73:27
21.92
69.73
39.25
34.72
46.04
46.21
48.59
46.35
Grok 4
36.73
72.08
20.25
65.34
34.66
37.21
46.17
40.18
51.81
44.94
OpenAL-o1
35.68
OpenAI-o3
35.40
70.38
15.38
65.05
24.08
28.62
43.08
44.33
30,26
41.84
Qwen3-415
24.74
Thoth-mini
11.28
74.68
25.33
70.83
52.67
43.32
49.23
46.41
53.13
$1.10
Qwen3-8B
28.89
63.51
11.17
58.67
24.33
16.66
32.31
34.72
38.63
34,32
Protocol Generation We evaluated protocol generation across two di- mensions: executability, which em- phasizes consistency in objects and step order, and lexical similarity, which reflects surface-level match- ing. As shown in Table [ Thoth achieves SOTA results across all met- rics. Compared to baselines. Thoth and Thoth-mini improve average per- formance by 17.78% and 22.01% %, underscoring the effectiveness of our approach. Thoth also surpasses much larger proprietary models, outper- forming ChatGPT-4o by 3.69% on average. Against the strongest open- source model, DeepSeek-V3, Thoth achieves gains of 4.88% 4.06%, and 11.29% on Semantic-Alinnment Order-S, and Step-MATCH, respec- tively, showing clear advantages in step alignment, logical order, and ac- tion fidelity. General-pune purppoee SOTA models show notable capability, likely due to large-scale pre- training, but mainly achieve superficial similarity and lack real-world executability. Reasoning- oriented models also perform poorly, often producing overly complex outputs unsuited for labo- ratory workflows (see Appendix I.I for qualitative cases). Overall, Thoth demonstrates consistent advantages across metrics, bridging knowledge-based text generation with executable protocols.
HLE: Biomedicine
LAB-Bench: ProtocolOA
175
:5.0
12.
:3.0
K.2
16.20
Intern matel
Ioter-33 21
Thath-inn minl
Thath
hntrr Sintel votel
Intern-33
Theth-inl miel
Thuth
PubMedOA
Average
36.33
31.00
33,77
+000
Accurary
Intam Sn-i
Iiter-33
Thath-eial Thath
Scont
Theth-eiet
Thath
Figure Results on other scientific benchmarks. Slash shading denotes baseline of Thoth models.
Fundamental Scientific Tasks Beyond protocol generation, we evaluated Thoth three out-of- domain biomedical benchmarks against recent scientific LLMs (Intern-S1 and SciDFM), with re- sults shown in Figure Thoth consistently outperforms models fine-tuned on biomedical corpora, exceeding Intern-S1 and Intern-S) Sreen-S1-mine by 7.09% and 2.22% on average, and notably both Thoth and Intern-S1-ming share Qwen3-8B their base model. As illustrated in Figure Thoth also
8

---

shows clear improvements its baseline models (slash shading), achieving average improve- ment of 10.87%, while Thoth-mini improves by 7.43%. The performance gap between Thoth and Thoth-mini further confirms the benefits of our approach. These results demonstrate that knowledge and reasoning skills acquired from scientific protocols generalize effectively to broader biomedical tasks. Additional experiments protocol comprehension are provided in Appendix I.3)
4.4 ANALYSES
SciRecipe To assess the impact of different data components, we conducted ablation studies on SciRecipe (Table 2). Models trained only on QA or generation QA showed weak results, with BLEU-AVG below 40%. Adding Protocol-Comppeeession tasks markedly improved executability, with Order-LCS reaching 74.50%. Joint training on both task types further enhanced alignment, with Step-M rising to 52.33%. The best overall performance was obtained when combining QA with scientific review, achieving 46.60% on Semantic-A and 49.45% on AVG. These findings high- light that diverse task coverage and systematic quality checks enhance the reliability of protocol generation.
Table 2: Ablation studies on training set. + trained only on SciRecipe Protocol-Cone Contrccl-Compeeeensiin tasks; trained both task types. QA and SeiCheck denote inclusion of basic protocol QA and scientific review of SciRecipe, respectively. *: trained only on protocol generation QA.
SciRecipe
QA
SciCheck
Semantie-A
Order-LCS
Order-S
Step-M
BLUE-AVG
ROUGE-L
METEOR
KW-F1
AVG
23.88
31.50
29.53
34.50
29.85
23.71
32.08
27.92
37.78
30.37
42.34
74.50
24,33
50.50
41.33
47.34
44.81
51.72
47.11
44,51
74.67
25.00
52.33
42.74
48.53
46.84
$2.30
48.37
44.54
75.14
23.50
47.00
41.79
48.09
47.01
51.89
47.37
75.34
$3.00
43.62
50.02
47.36
49.45
Reasoning Paradigm To align model outputs with experimental logic and generate evaluable pro- tocols, we propose the "Sketch-and-Fiil" reasoning paradigm. effect was systematically tested through ablation experiments on four representative models: DeepSeek-V3, GPT-5 Chat, Thoth, and its base model Qwen3-8B.. Since outputs are in natural language, exccutability metrics were not applicable, and Figure 4 reports results based on lexical similarity. The results show that "Sketch- and-Fill" improves performance for models except the base model, with notable gains on BLEU- AVG. DeepSeek- V3, GPT-5 Chat, and Thoth achieve average improvements of 3.89%, 2.92%, and 3.79% over counterparts without the paradigm. For the base model, the lack improvement is attributed to reasoning failure due to missing knowledge injection from scientific protocols.
Table 3: Ablation studies the SCORE mechanism. Evaluation covers step scale, semantic align-
ment, and overall reward design. K (.)) denotes the KI.-divergence penalty in the loss function, and "Vanilla" refers using standard semantic similarity metrics as the reward signal.
SCORE
Semantic-A
Order-I.CS
Order-S
Order-Tan
BLUEFAVG
ROUGE-
METFOR
KW-F1
AVG
w/o fid)
43:67
55.97
6.83
51.83
10.00
43.09
37.28
26.08
13.31
35.34
Step Scale
wio oi
38.89
74.37
23.67
71.00
51.00
43.18
46.14
44.92
52.80
49.55
(L)
wio may
38.68
73.70
23.17
67.67
50.17
42.84
48.91
45.58
$2.85
49.29
Semantic Aligament
w/o Ordenn
40.93
61.27
12.83
48.00
33.33
37.26
45.35
43.27
19,16
11.30
w/o Ordber())
+f(d++g(
44.15
58.22
7.67
42.83
43.53
38.45
45.78
43.52
48.74
41.41
w/o KL())
74.43
22.67
42.73
46.03
$3.36
48.89
Vanillaa
38.74
63.41
21.50
$2.36
44.50
45.52
50.12
43.90
50.54
45.62
Thoth
46.60
75.34
25,50
73.33
53.00
43.62
50.02
47.39
$4.13
$2.10
SCORE Mechanism In Section 3.3.2. we introduced the SCORE reward mechanism, consisting of step scale, order consistency, and semantic consistency. To evaluate its role in RL training, conducted ablation studies, including variants that removed individual components, dropped the KL divergence penalty, replaced SCORE with vanilla reward based only BLEU-AVG, ROUGE- L, and BERTScore. Results are summarized in Table 3 Omitting the step scale reward leads to sharp declines in Order-S and Step-M (6.83% and 3%/44.1 1%%, and 3), indicating failures

---

to generate executable protocols and producing either verbose or incomplete outputs. Excluding the positional decay factor (row 4) weakens penalties for misaligned steps and reduces semantic consistency, under which Thoth still achieves 7.92% gain in Semantic-A. The order consistency reward also critical because without enforcing execution order, steps become misarranged and semantic coherence breaks down. Thoth surpasses the variants in rows 5 and 6 by 6.36% and 5.17% on BLEU-AVG, respectively. Removing the KL penalty further degrades performance, with Table 3 showing a 3.21% 1% average drop compared to Thoth. Although the vanilla reward improves BLEU- AVG and ROUGE-L, the generated protocols exhibit poor executability with 10.65% average reduction. These results demonstrate the effectiveness of SCORE in guiding the model to generate coherent and executable protocols. For further analyses of reward, see Appendix J.1.2 & J.1.3
180
453
30
04.9
41.
$1.22
tmn
42.7
30.8
28.3
+130
13.0
130
Oeage
Eoperill
Deart
Degata)
Treph
Figure 5: Ablation studies on the training strategy. Left and middle: rightmost A marks extreme values across strategies. Right: Thoth's 's training trajectory across stages.
Training Strategy To progressively equip the model for solving complex scien- tific problems, we propose the three-stage Knowledge-to-Nction Learning strategy. We compared models trained different stages with Thoth to validate its effective- ness. As shown in Figure 5 (left and mid- dle), Thoth achieves SOTA performance after staged training and consistently out- performs models trained with only Stage 1+2 (pre-training and SFT) or without pre- training (Stage 2+3 or Stage 2 alone). Specifically, Thoth improves by an aver- age of 11.08% on executability and 4.2% on lexical similarity over Stage 1+2. It further achieves mean gains of 1.44% and 8.79% over Stage 2+3 and Stage 2 across both dimensions. Figure 5 (right) further demonstrates that the three-stage process progressively enhances performance, with SFT improving lexical similarity and RL strengthening rationality and executability.
XN-F1
Ingroveeent monk
DPT Out
Gwext Thatk
Figure 4: Ablation studies on reasoning paradigm. Performance differences with/without Sketch-and (SaF) are shown in the top panel, and visual compar- ison is provided in the bottom panel.
CONCLUSION
In this paper, we introduced Thoth, a the "Sketch-and-FFi reasoning paradigm, and the SCORE mechanism. By integrating structured reasoning with verifiable rewards, Thoth achieves SOTA performance on both ppococol-specific and broader scientific benchmarks. Equally important, the protocols it generates are concise, logically coherent, and experimentally executable. This work provides blueprint for building reliable sci- entific assistants and demonstrates that structured reasoning frameworks combined with targeted rewards are critical for advancing protocol generation and improving reproducibility.
model grounded in the SciRecipe dataset,
10

---

ETHICS STATEMENT
This work does not involve human subjects, animal experiments, or personally identifiable data. All datasets used in this paper will be publicly released upon publication to ensure transparency and reproducibility. We have carefully considered potential ethical issues related to data privacy, fairness, and misuse. We believe our findings pose no foreseeable risks of harm, and all contributions were made in compliance with the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our results. Details training data distributions, hyperparameter settings for pre-training. SFT, and RI,, as well as implementation frameworks, are provided in Appendix [ We will also release all code, models, and processed datasets to facilitate faithful reproduction of our findings.
ACKNOWLEDGEMENT
This work was supported by Shanghai Artificial Intelligence Laboratory and Intern Discovery.
REFERENCES
Hervé Abdi. The Kendall rank correlation coefficient. Encyclopedia of measurement and statisties, 2:508-510, 2007.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv: 087744 2023.
Sandhini Agarwal, Lama Ahmad, Jason Ai. Sam Altman, Andy Applebaum, Edwin Arbus, Rahul Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. PoTT-ss--100 & GPT-oss-200 model card. arXiv preprint 250iv:2508. 10925, 2025.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimiza- tion for learning from human feedback in LLMs. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 12248-12261 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 100 wwtp:///oooooo../222/2//// oot.:/oooooooooro02222..----o-..
URL
Anthropic. Claude (opus sonnet series) language models. https: www http://www.thhoicc.ooo 1ode 2025. Accessed: 2025-09-22.
Lei Bai, Zhongrui Cai, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, Yongkang Chen, et al. Intern-sl: scientific multimodal foun- dation model. arXiv preprint arXiv:2508. 15763, 2025.
Satanjcev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Jonas Becker, Jan Philip Wahle, Bela Gipp, and Terry Ruas. Text generation: A systematic literature review of tasks, evaluation, and challenges. arXiv preprint arXiv:2405. 15604, 2024.
Iz Beltagy. Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903. 2019.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48 2009.
11

---

Bio-protocol LLC. Bio-protocol. Ittps://bio-pozoocccc.oo 2025, Accessed: 2025-09-
17.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
ByteDance. Doubao 1.5 pro. https: http://wwwww.bbb..oooooooo 22.
2025. Accessed: 2025-09-
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626/ 2025.
OpenCompass Contributorss Opencompass: A universal evaluation platform for foundation models. 2023.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lueas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint 2xXx:2401.00221,, 2024.
Leonard P Freedman, Iain M Cockburn, and Timothy Simcoe. The economics of reproducibility in preclinical research. PLoS biology, 1006)160 2015.
Shanghua Gao, Richard Zhu, Zhenglun Kong, Ayush Noori, Xiaorui Su, Curtis Ginder, Theodoros Tsiligkaridis, and Marinka Zitnik. Txagent: An AI agent for therapeutic reasoning across universe of tools. arXiv preprint arXiv:2503. 2025.
Joshua T Goodman. A bit of progress in language modeling. Computer Speech & Language, 15(4): 403 434, 2001.
Steven N Goodman, Daniele Fanelli, and John PA loannidis. What does research reproducibility 2016.
mean? Science translational medicine, 8(341):341ps12-344
Aaron Grattafiori, Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788. 2024.
Maarten Grootendorst. Keybert: Minimal keyword extraction with bert., 2020. URL https:
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, et al. A survey llm-as-a-judge. arXiv preprint arXiv:2411.15500 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. DeepSeek-RI: Incentivizing reasoning capability in LLMs via reinforeement learning, arXiv preprint arXi::2501.12948, 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
12

---

Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025.
Kexin Huang, Serena Zhang, Hanchen Wang, Yuanhao Qu, Yingzhou Lu, Yusuf Roohani, Ryan Li, Lin Qiu, Gavin Li, Junze Zhang, et al. Biomni: A general purposse biomedical AI agent. biorxiv, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16722) 2024.
Shuo Jiang, Daniel Evans Yamamoto. Dennis Bersenev, Sucheendra K Palaniappan, and Ayako Protocode: Leveraging large language models (LL.Ms) for automated gener- ation of machine-readable per protocols from scientific publications. SLAS technology, 29(3): 100134, 2024.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. PubMedQA: A 2019.
dataset for biomedical research question answering. arXiv preprint
Ruofan Jin, Zaixi Zhang, Mengdi Wang, and Le Cong. Stella: Self-evolving Ilm agent for biomedical research. arXiv preprint wrXiv:557.00004. 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
J Richard Landis and Gary Koch. The measurement of observer agreement for categorical data. biometrics, pp. 159-174, 1977.
Jon M Laurent, Joseph Janizck, Michael Ruzo, Michaela Hinks, Michael J Hammer- ling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D White, and Samuel G Rodriques. Lab-bench: Measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10366 2024.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 1ee::-1444---400 1240, 2020.
Zongxia Li, Yapei Chang, Yuhang Zhou, Xiyang Wu, Zichao Liang, Yoo Yeon Sung, and Jordan Lee Goyd-Graber Semantically-awar rewards for open-ended rl training in free-form generation. arXiv preprint arXiv:2506. 15068, 2025.
Chin-Yeu Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pp. 74-81, 2004.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeck-V3 technical report. arXiv preprint 0rXiv:2412.19437 2024.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using GPT-4 with better human alignment. arXiv preprint arXiv:2303. 16634, 2023.
Yuyang Liu. Liuzhenghao Lv, Xiancheng Zhang, Li Yuan, and Yonghong Tian. Bioprobench: Com- prehensive dataset and benchmark in biological protocol understanding and reasoning. arXiv preprint w:Xiv:2505.07889 2025.
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie- Yan Liu. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief- ings in bioinformatics, bbbc)bbb 2022.
National Academies Sciences, Medicine, Policy, Global Affairs, Board on Research Data, Di- vision Engineering, Physical Sciences, Committee Applied, Theoretical Statistics, Board on Mathematical Sciences, et al. Reproducibility and replicability in science. National Academies Press, 2019.
13

---

OpenAl. Gpt-5 chat. https::/chat.
2025. Accessed: 2025-09-17.
Odhran O'Donoguue, Aleksandar Shtedritski, John Ginger, Ralph Abboud, Ali Ghareeb, and Samuel Rodriques. Bioplanner: automatic evaluation of Ilms on protocol planning in biology. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2676-2694, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's 's last exam. arXiv preprint arXiv:2501. 14249, 2025.
Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, et al. Hellobench: Evaluating long text generation capabilities large language models. arXiv preprint arXiv:2409. 16191, 2024.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL https: arxiv org/ abs/ 2412. 15115
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. DeepseekMath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Liangtai Sun, Danyu Luo, Da Ma, Zihan Zhao, Baocai Chen, Zhennan Shen, Su Zhu, Lu Chen, Xin Chen, and Kai Yu. Scidfm: A large language model with mixture-of-experts for science. arXiv preprint Xiv:224099. 1224499.1442,, 2024.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Kimi k2: Open agentic intelligence. arXiv preprint 2rXiv:2507.20534 2025.
Leonid Teytelman, Alexei Stoliartchouk, Lori Kindler, and Bonnie L Hurwitz. Protocols. io: virtual communities for protocol development and discussion. PLoS Biology, 14(8):e1002538, 2016.
Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409. 18839, 2024.
Hongru Wang, Cheng Qian, Wanjun Zhong. Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Acting less is reasoning more! teaching model to act efficcintll. arXiv preprint 250::25044.48700 2025.
Xin Wang. Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9):455544566 2021.
Wikipedia contributors.
Nature protocols Wikipedia, the free encyclopedia, 2025.
URL https://en. wtt.e/enn.wkkiieee www.::wia..ipioioo.c.....aaa wttp//////w///xxx/.////e/// php titteeNatteeee saa.Leooutuuree.pottoooo..sss Protocolsd o bldid=1296318228 [Online; accessed 22552).
14

---

XAL Grok 4. httt::://oc.....a/////8 http://///..ooo..//////////////. 2025, Accessed: 2025- 09-22.
Yifei Xu, Tusher Chakraborty, Srinagesh Sharma, Leonardo Nunes, Emre Kıcıman, Songwu Lu, and Ranveer Chandra. Direct reasoning optimization: LLMs can reward and refine their own reasoning for open-ended tasks. arXiv preprint arXiv:2506. 13351, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang. Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Seungjun Yi, Jaeyoung Lim, and Juyong Yoon. Protocollm: Automatic evaluation framework of Iims on domain-specific scientific protocol formulation tasks. arXiv e-prints, pp. arXiv-2410, 2024.
Sihang Zeng, Kai Tian, Kaiyan Zhang, Junqi Gao, Runze Liu, Sa Yang, Jingxuan Li, Xinwei Long, Jiaheng Ma. Biqing Qi, et al. Reviewrl: Towards automated scientific review with rl. arXiv preprint arXiv:2508. 10308, 2023.
Caiqi Zhang, Xiaochen Zhu, Chengzu Li, Nigel Collier, and Andreas Vlachos. Reinforcement learn- ing for better verbalized confidence in long-form generation. arXiv preprint arXiv:2505.23112, 2025a.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint (eXiv:994400967 2019.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05/ 2025b.
Zhongyue Zhang, Zijie Qiu, Yingcheng Wu, Shuya Li, Dingyan Wang, Zhuomin Zhou, Duo An, Yuhan Chen, Yu Li, Yongbo Wang, et al. Origene: A self-evolving virtual disease biologist automating therapeutic target discovery. bioRxiv, pp. 2025-06, 2025c.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595 46623,,66,,,,,3 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguis- tics. URL http: wwtp:/wwwww.wwaa.....o..oooooooe orq/ abs www.//03.22222 13372
15

---

APPENDIX CONTENTS
SciRecipe
17
A.1
Preprocessing of Original Scientific Protocols
17
A.2
Structured Integration of Scientific Protocols
17
A.3
Supplementary Details SciRecipe
17
Supplementary Details on SciRecipe Construction Pipeline
18
SciRecipe-Eval
20
SCORE Mechanism
20
Consistency Gate
20
Order Consistency
21
Action Anchor
22
D RL Algorithm
22
Baseline Models
23
Reproducibility Statement
23
Limitations
24
H Usage of LLMs on Paper Writing
24
Additional Experimental Results
25
Qualitative Study of Reasoning Models
25
Fine-grained Results on SciRecipe-Evvl
30
Other Results on Protoco) Benchmark
35
................................
Comparison with Chain-of-Thought and Sketch-and-Fiil
35 ................................
1.5
Human-Metris Consistency Analysis
35
1.6
Quantitative Validation of the Sketch-and-Fill
37
J
Futher Analyses
38
U.1
Reward Analyses & Results
38
J.2
Pre-training Analysis
42
Case Studies
43
Prompts
49
16

---

A SCIRECIPE
A.1 PREPROCESSING OF ORIGINAL SCIENTIFIC PROTOCOLS
In the preprocessing phase, we first employed MinerU (Wang et al. 2024) to identify and extract protocol texts, as the majority of online experimental protocols are distributed in non-editable PDF format. This step was essential because protocols published in PDF form often contain heteroge- neous content such as figures, tables, references, and lengthy narratives, which substantially increase the difficulty of accurate parsing and subsequent question-answer pair construction. To mitigate this issue, we analyzed both the page count and the number of figures/tables in each document, and ap- plied empirically determined thresholds to filter out overly long or structurally complex protocols. Another important consideration was the removal of redundancy, since many protocols share highly similar structures or represent slightly modified versions of the same experiment. Without careful filtering, these near-duplicate entries could cause information leakage across training and evaluation splits. To address this, we applied similarity detection and clustering. Specifically, we used Qwen3- Embedding-83 (Zhang et al., (025b) to vectorize protocol titles, built an efficient similarity search index using FAISS (Douze et al. (024), and conducted Top-K nearest-neighhor searches. Similar pairs were identified by thresholding cosine similarity scores, after which a union -ind algorithm was applied to partition protocols into connected components. This process produced both pair- wise and grouped similarity sets, effectively eliminating redundant samples while preserving data diversity and representativeness. In addition, some protocols were accompanied by electronic sup- plementary materials, which frequently contained fragmented or ambiguous instructions. Because these supplementary files risked introducing inconsistencies and annotation difficulties, they were uniformly removed from the dataset. To further guarantee the reliability of the corpus, we recruited three doctoral students with extensive experimental experience to manually review a subset of the data. Their verification ensured not only correctness but also balanced coverage across experimental domains, ultimately enhancing both the diversity and representativeness of the retained corpus.
A.2 STRUCTURED INTEGRATION OF SCIENTIFIC PROTOCOLS
Because experimental protocols vary widely in writing style and formatting, we extracted common elements to form a unified structured representation. The core fields included experiment name, re- search background, reagents and materials, experimental equipment, and experimental procedures. This integration was carried out in two complementary steps. First, applied a rule-based method that relied on regular expressions to match predefined keywords and extract the corresponding fields. During this stage, Markdown tables were removed to avoid parsing errors, while figure and table captions were preserved as valuable contextual information. The rule-based approach achieved re- liable performance for protocols written in highly standardized format, but its applicability was limited for documents with more complex or irregular structures. To address these limitations, we further adopted model-based strategy. Specifically, we leveraged Grok-4 (xAi (025) summarize and restructure the original protocols, ensuring accurate extraction of key fields without introducing fabricated content. The model also adjusted the ordering of experimental steps to maintain logical and procedural consistency. This hybrid two-step pipeline allowed us to combine the precision of rule-based parsing with the flexibility of model-based restructuring. Through this process, we ob- tained over 12K structured protocols. Quality control measures were subsequently applied to filter out entries with missing fields, excessively lengthy procedures, or dependencies on external online resources. The resulting corpus provides clean, logically consistent, and representative collection of protocols suitable for downstream training and evaluation.
A.3 SUPPLEMENTARY DETAILS ON SCIRECIPE
When designing the eight task types in SciRecipe, we defined explicit objectives and representative scenarios for each. Together, these tasks cover the entire lifecycle of experimental research, ensuring that the dataset trains and evaluates both general reasoning and practical execution.
Overview: Requires the model to generate a hicrarchically organized summary of the pro- tocol, enabling rapid comprchension of its global structure.
Specific: Focuses on step-by-step decomposition of procedures, ensuring the model cap- tures the micro-level operational logic.
17

---

Retrieval:
Targets precise extraction of experimental parameters (e.g., temperature, pH, concentration), emphasizing accuracy in reporting.
Planning:
Involves transforming high-level objectives into coherent, logically ordered ex- perimental steps.
Simulates diagnosing experimental failures, identifying error sources, and proposing corrective strategies.
Troubleshooting:
Constraint: Tests adaptability under resource limitations, such as restricted equipment or reagent availability.
Scaling: Requires numerical adjustments and unit conversions to adapt experiments across different scales.
Safety: Focuses on compliance with safety standards and identification of potential labo- ratory risks.
By encompassing comprehension, planning, error handling, adaptability, scalability, and safety,
SciRecipe ensures comprehensive and targeted task design. See Figure 21 detailed prompts.
A.4 SUPPLEMENTARY DETAILS ON SCIRECIPE CONSTRUCTION PIPELINE
During the construction of SciRecipe, we first generated QA pairs for multiple task types using the structured protocol corpus as the foundation. Particularly. for Protocol-Cone Caatoool-Compreeensionn tasks, we relied on hierarchical tree extraction (e.g.. "Step 1: xxx; Step 1.1: xxx; Step 1.1.1: xxx"), which preserved both the hierarchical structure and the stepwise dependencies of experimental procedures. This design ensured that the model could capture not only surface-level descriptions but also the nested logic inherent in complex experiments. Then, we introduced the "Sketch-and- paradigm. This approach first summarizes specific operations, thereby enhancing both interpretability and training effectiveness. The paradigm was particularly effective in decomposing high-level objectives into concrete experimental actions while maintaining logical consistency. Because LLMs may produce outputs with randomness and formatting inconsistencies, directly discarding such samples would risk losing high-quality data. To address this, we designed format validation and repair module. This module automatically detected non-compliant outputs and prompted the model regenerate them under low-temperature settings, ensuring structural consistency while retaining valid content.
reasoning elements in outline-like format and then fills in the
After passing the format checks, each QA pair underwent content review across six scientific di- mensions:
1. Scientific Accuracy: correctness of operations, parameters, and outcomes.
2. Safety & Compliance: adherence laboratory safety standards and ethical guidelines.
3. Logical Coherence & Actionability: stepwise consistency and rational flow of proce- dures.
4. Clarity & Ambiguity: precision and unambiguity of textual descriptions.
5. Generality & Specificity: balanced coverage of broad applicability and detailed context.
6. Efficiency & Resource Optimization: optimization of time, materials, and experimental resources.
For this step, we employed Gemini 2.5 Flash (Comanici et al. 2025) as the validation model, which performed dimension evaluations and produced review reports. QA pairs that failed to meet requirements were systematically discarded. Finally, to guarantee the reliability of the constructed dataset, we performed manual verification. Doctoral students from interdisciplinary backgrounds conducted thorough inspections, ensuring that the resulting data were not only structurally cor- rect but also scientifically feasible and practically reasonable. Through this multi-stage pipeline, which combines automated validation, targeted repair, ual checks, SciRecipe achieved both rigor and diversity, providing a robust foundation for protocol- oriented training and evaluation.
review, and expert-level man-
18

---

Table 4: Disciplinary distribution of the SciRecipe dataset.
Subdomain
Count
Percentage
Cell Biology
2366
19.82%
Biochemistry
1935
16.21% %
Molecular Biology
1788
14.98%
Microbiology
1163
9.74%
Plant Science
936
7.84%
Immunology
717
6.01%
Neuroscience
688
5.77%
General Laboratory Proccdure
521
4.37%
Bioinformatics
319
2.67%
Bioimaging Technologics
284
2.38%
Cancer Biology
275
2.30%
Model Organism-Speenn Techniques
246
2.06%
Others
127
1.06%
Genomies Technologies
111
0.93%
Structural Biology Techniques
0.66% 79
Biophysics
0.65%
Pharmacology & Drug Development
67
0,56%
Developmental Biology
64
0.54%
Genetics
0.46% 55
Histology Techniques
31
0.26%
Synthetic Biology & Bioengincering
0.26% 31
Stem Cells
0.15% 18
Bioengincering
0.09% 11
Toxicology & Safety Testing
0.09%
Systems Biology
0,07% 4
Medicine
0.03%
Drug Discovery
0.03%
Number actions
Mean-1.72
0.5
1.0
1.5
3.0
Number of parameters
Mean=4.40
2
4
Number of
Figure 6: Data distribution of experimental actions, parameters (each step), and step counts in SciRecipe after quality control.
19

---

B SCIRECIPE-EVAL
Although many benchmarks address scientific question answering, resources that directly evaluate experimental protocol generation are still very limited. Existing evaluations often require models to restate paraphrase the original protocol, which does not reflect the ability to generate new and executable procedural content in realistic laboratory contexts. To address this gap, we construct SciRecipe-Evall as benchmark designed to assess protocol generation under practical usage sce- narios. Following the design principles SciRecipe, we independently curate and process set of 400 protocols. The benchmark is organized into two main categories, and each category contains approximately 300 samples with a balanced distribution across subtasks.
While the names of these categories may resemble comprehension-oricne quire the model to generate structured protocol content based on the user query. The Protocol Comprehension tasks evaluate whether the model can understand the user's goal and produce pro- cedural steps that align with standard experimental workflows. The Problem Solving tasks focus on situations in which experimenters encounter practical issues that are not explicitly addressed in the original protocol, and the model must generate corrective or supplementary steps. In this sense, SeiRecipe-Evar serves as protocol generation benchmark rather than a traditional ques- tion answering benchmark. The understanding and reasoning involved in these tasks are integral components of the protocol generation process.
tasks, all subtasks
Protocol-Compreeensio::::
Overview: generating hierarchical summaries of experimental procedures.
Specific: decomposing steps to capture micro-level logic.
Problem-Solving:
Retrieval: extracting precise parameters such temperature, pH, concentration.
Planning: transforming objectives into coherent and logically ordered plans.
Troubleshooting: diagnosing potential causes of experimental failure.
Constraint: adapting protocols under resource limitations.
Scaling: performing numerical adjustments and conversions for different scales.
Safety: ensuring compliance and risk control.
Because the QAs were designed under specific reasoning paradigms, experimental actions (predi- cates) were treated as the core elements of the ground truth. The number of actions defines the size of the action space and thereby provides a direct measure of experimental complexity. Based on this criterion, tasks containing fewer than four actions were categorized as Level 1, whereas those with four or more actions were categorized as Level 2. For Level 2 tasks, we further inserted one to four randomly sampled distractor actions, enabling more comprehensive evaluation of model's ability to discriminate between correct and incorrect operations. standards established for SciRecipe, recruited three doctoral students in biology-related fields to perform manual sampling and review of the dataset. Their verification ensured that the final bench- mark not only structurally consistent but also scientifically accurate and representative of real laboratory practices.
line with the data quality control
SCORE MECHANISM C
C.I CONSISTENCY GATE
In protocol evaluation, consistency refers to the step-by-step correspondence between the structured outline (<key>) and its natural language expansion (<orc>), If an action, object, parameter de- clared <key> is missing from the corresponding <orc> description, the generated protocol is semantically incomplete and cannot serve reliable experimental guide. Therefore, the consis- tency gate is adopted as the second checkpoint in SCORE, and only protocols passing this constraint eligible for reward computation.
Concretely, we first require that the number steps in <key> matches exactly with <orc>, with consecutive numbering covering 1 through N. For each step i, we construct token set T, by
20

---

concatenating the declared action oN, object set O₁ and parameter set P,. After normalization (e.g., case folding, unit standardization, removal of subscripUssppesseritt variants), we compute the coverage rate of these tokens in the natural language step <orc>; as:
(1 E : C norm(<orc>)))
cov;
(11)
Ti
If all steps satisfy cov; 2 T (we set T = 0.95 in this work) and the step indices are strictly aligned, the consistency gate is considered passed:
= fL N = A (fe (fI.....,NNI. = (I....,,N NJ A min; cov,
Leons (v)
(12)
0, otherwise
where Ñ and N denote the step counts of the generated and reference protocols, respectively, and ñ, denotes the predicted step indices. If the gate fails, all subsequent rewards are set to zero. This hard gating design reflects a fundamental experimental principle: when the structured outline and its natural language realization are inconsistent, even partially correct fragments cannot make the protocol executable.
C.2 ORDER CONSISTENCY
Let á (a1,.--,,e â.)) denote the predicted action sequence extracted from <key>, and a" (aj...., am) the reference sequence. The goal of order consistency is to evaluate whether the pre- dicted exccution order preserves the logical progression of the reference protocol, since deviations in ordering can undermine reproducibility even when individual steps appear correct. Two comple- mentary scoring modes are provided.
The first mode, Strict Subsequence, enforces conservative executability constraint. The score equals the predicted sequence is identical the reference, or if one subsequence of the other; otherwise it is O:
(anistiiec(@aat Orderstrier (ā, a²) ( [1,
a" V subseq (a) a") V subseq(a", .))
(13)
otherwise
where subseq y) holds when x can be embedded i with order preserved. This criterion accepts insertions or omissions as long as the preserved actions remain order, but strictly rejects any reordering. It is also computationally efficient, requiring only O(n + m.) time.
The second mode, Longest Common Subsequence (LCS), provides a graded similarity that reflects how much of the reference order is retained. The score is defined as:
OrderLcs a²) LCS(@, a*)
(14)
m
where |a"| is the length of the reference sequence and LCS(..) is the length of the longest common subsequence between the predicted and reference sequences. This normalization yields a value in [o, 1] that represents the fraction of the reference order preserved. Unlike the strict mode, LCS assigns partial credit when the prediction maintains some correct subsequences even if other steps are misplaced. For example, given the ground truth [harvest, lyse. centrifuge, quantify], prediction [harvest, lyse, quantify] achieves a strict score of I and an LCS score of 3/4 = 0.75 (omission but order preserved), whereas [harvest, centrifuge, lyse, quantify] scores 0 under the strict rule but 3/4- = 0.75 under LCS, since most of the order is still preserved. By contrast, [lyse, harvest, quantify, centrifuge] more severely misordered, yielding an LCS score of 2/4 0.5.
For completeness, the LCS length is computed via dynamic programming. Let L(i,j) denote the LCS length between the prefix subsequences (ū1....,ū:) of the predicted sequence and (af... (ai..... az) of the reference sequence a". Then
0,
0 or j 0
L(i,))
L(i 1.j-1) +1,
à a
(15)
max{L(i - 1,5), L(i.j 1)}, à, + a
21

---

with LCS((, (n.m). The algorithm runs in (nm) time and requires ((n)) space.
Together, these two modes serve distinct but complementary purposes, with the strict subsequence mode being the primary criterion. Strict alignment best reflects the hard requirements of laboratory exccutability, where even minor order violations typically render a protocol unusable. The LCS mode is introduced only supplementary measure, offering smoother and more informative feed- back during training by granting partial credit when some procedural intent is preserved. In practice, strict mode is preferred for evaluation of executable protocols, while LCS can be selectively applied to provide graded optimization signals when softer supervision is required. All formal experimental results in this work are reported under the strict mode. See Appendix J.1.2 for further experiments.
C.3 ACTION ANCHOR
In semantic consistency evaluation, actions (i.e., step verbs) are treated as anchors to align predicted steps with reference steps before any further comparison. The aligmment is performed in a left-to- right, order-preserving manner. Formally, let the predicted action sequence be â (āi..... an) For each predicted action á, we select
and the reference action sequence be a* = (ai....,00 the earliest reference index j after the previous match such that aj = âz. If such a exists, we record the pair (i,j), otherwise â; remains unmatched and contributes no anchor. This procedure yields an ordered set of index pairs W {(11,11,..... (Kk] with I S i < .... < ik S n, < jK I m, and åيg a; for all (ix. ju) W. The construction is greedy first-match), which guarantees that (i) sequence order is preserved, (ii)
1 S J1 and monotone repeated actions are matched consistently, with each predicted occurrence aligning only to the next unused reference occurrence, and (iii) the overall alignment runs in linear time O(n + m).
Example For the ground truth sequence Iharvest, lyse, centrifuge, quantify) and the prediction [harvest, centrifuge, lyse, stain, quantify). the alignment proceeds as
i1 harvest
oj harvest,
å2 centrifuge
0g centrifuge,
03 = lyse
4 0 lyse (rejected: order violation),
å5 quantify - of = quantify.
Hence the retained alignment is = ((1, 1), 3), 4)), while the unmatched action stain is discarded. This anchor-based procedure ensures that semantic comparisons focus only on correctly aligned procedural units, avoiding misleading matches across unrelated operations.
D RL ALGORITHM
During the RL stage, we adopt Group Relative Policy Optimization (GRPO) (Shao et al. 2024), a variant of PPO (Schulman et al. 2017) designed to reduce variance and stabilize updates by exploit- ing multiple responses generated from the same query. Unlike standard PPO, which normalizes the reward across batch, GRPO performs group-wir normalization within each query group. This strategy ensures that the relative quality of responses to the same query is emphasized, thereby mitigating reward scale differences across queries and improving training stability. Formally, let denote query, and {3) be G responses sampled from the current policy Te(+1z). For each response yı, the normalized advantage defined as
r(x,, y:) mean (fr(z, 1;))
A
(16)
std((r(z, 91)1,
where r(z y) denotes the scalar reward assigned to yı By centering and scaling rewards within the group, the advantage highlights relative performance among candidate responses.
The GRPO objective combines this normalized advantage with the clipped surrogate function and KL-regularization term that constrains the updated policy toward the reference model:
G min (.)) clip(s:(0), 1 1 e, (++) (17)
(GR = EEyDD, 1
-BE (K((T((-12)
(+(x)),
22

---

where is clipping threshold controlling the size of policy updates and 3 determines the strength of KL regularization, with πe denoting the current policy being optimized and T the fixed refer- ence policy. The importance sampling ratio s;()) is computed the token level to account for the sequential nature of text generation. Specifically, is defined as the geometric mean the per-token probability ratios:
Te (31,
2, yb,sce)
s;( = exp
log
(18)
TT (3/t, I, Mi,
where Teaa denote the old policy models, respectively, and Vi, is the 1-th token of response yı- This per-token formulation maintains sensitivity fine-grained action probabilities and avoids domina- tion by sequence length. Overall, GRPO stabilizes training by normalizing advantages within query groups, constrains policy updates through clipping and KL regularization, and ensures robustness the token level via per-token importance ratios.
E BASELINE MODELS
Given the lack of models tailored for scientific protocol generation, we compare our approach against a broad set of large-scale language models. The closed-source baselines include the GPT-5
series [ 2025 (GPT-5, GPT-5 Chat), the Claude series (Anthropic) (2025) (Claude Sonnet 4,
Claude Opus 4. l). the Gemini 2.5 series (Comanici et al. 2025) (Flash and Pro), as well ChatGPT- 4o (Achiam et al. 2023), Doubao-1.55 1.5-Po (ByteDance 2025), and Qwen2.5-Max (Qwen et al.)
2025). For open-source baselines, we evaluate strong instruction-tuned models such as Qwen2.5-
72B (Qwen et al., 2025). Qwen3-235B (Yang et al. 2025), DeepSeck-V3 (Liu et al. (024). GPT-
OSS- 120B (Agarwal et al. 2025), Llama-3.1-405: (Grattafiori et al. 2024), J. and Kimi-K2 (Tam consider reasoning-oriente 2025). , Grok 3/4 (xAl 2025). and OpenAl's ol and o3 (Jaech et al., 2024). Finally, to assess sci- further evaluate lomain-speciff models including iatern-Sl-ming Intern-S1 [Bai et al. 2025), and SciDFM (Sun et al. 2024). Together, these baselines provide comprehensive landscape of frontier, open-source, reasoning, and domain-oriented models for comparison.
ct al. 2025 In addition,
models including DeepSeek-RI (Guo al.)
entific capabilities,
F REPRODUCIBILITY STATEMENT
We provide essential information to ensure the reproducibility of our experiments. Table reports the training data distribution across stages, covering both SciRecipe tasks and BioProBench sub- tasks. Key hyperparameters for pre-training, SFT, and are listed Tables 6 7 and experiments were conducted with the a. framework for pre-training and SFT, and VeRL for RL training. We release detailed settings, including LoRA configurations, learning rates, epochs, scheduling, and hardware, that independent researchers replicate our pipeline and validate the reported results. All data, code, and models will be released publicly.
Table 5: Training data distribution across different stages (one epoch). "PC" denotes Protocol- Comprehension tasks and "PS" denotes Wobllem-Solving tasks. Four task types from BioProBench (Lu et al. (022)) are also incorporated to enhance the model's ability interpret protocols.
Dataset
SFT stage
RL stage
SciRecipe PC
34461
6956
SeiRecipe_PS
16164
3263
BioProBench_PP PQA
52640
BioProBench.ER ERRR
103983
BioProBench.One
31039
BioProBench.GEr
59477
23

---

Table 6: Key hyperparameters used in LLaMA-Factory for pre-training.
Parameter
Value
finetuning.Le
lora
lora rank
8
lora_target
all
preprocessing. Bacrlngn_mmm____ norkers
16
dataloader aataaoaadernumm.nn num_workers
per srevvicettrin_batee
gradient accumulation steps
learning_rate num train epochs Ir schedulerr type warmup ratio b
le-4
6
cosine
0.1
true
Table 7: Key hyperparameters used in LLaMA--actory for SFT training.
Parameter
Value
finetuning
lora
lora rank
32
lora.targe
all
preprocessing wwt._nnmmmwseee
dataloader num workers
per_device_traie atriccttttin___e batchbaaciiiii
gradient_accummut awccmmllationnstteiit steps
learning-rad rate
3e-4
num. vrain_epochs
Ir.hceeuelrr.. .type
cosine
warmup_ratin
0.05
6
true
G LIMITATIONS
While our work presents structured framework for scientific protocol generation and shows con- sistent improvements across benchmarks, some limitations remain. First, although SciRecipe spans 27 biological subfields, it is still skewed toward widely used laboratory techniques, with rare or highly specialized protocols underrepresented. Second, the SCORE metrics mainly emphasize step completeness and order fidelity, but do not yet capture finer aspects such as stylistic variation, cross- step dependencies, or long-range experimental context. Third, our experiments were conducted in relatively controlled settings, which may not fully reflect the complexity and variability of real lab- oratory environments. We believe these issues can be progressively addressed in future work by expanding the dataset cover rarer domains, enriching evaluation dimensions beyond structural executability, and validating the framework under more diverse practical scenarios.
H USAGE OF LLMS ON PAPER WRITING
We used LLMs (e.g., GPT-5) to aid in polishing the writing of this paper. Specifically, LLMs were employed to refine grammar, improve fluency, and adjust the tone of academic writing (e.g., smoothing transitions between sections, shortening overly long sentences, and rephrasing repetitive expressions). The models were not used for generating technical content, conducting experiments, or formulating research ideas. All substantive contributions remain the work of the authors.
24

---

Table 8: Key hyperparameters used in VeRL for RL training.
Parameter
Value
algorithm.adee _estimator ao
grpo
data.train batch size
1024
1024
data.max
weaiseeesoos_eeeeeeee length actor.optim.Ir
1024
7
actor-ppo_min http_mmiiiiahhhhhteee _batch_sizee
256
actor.ppo mwwpoppppomioo.....mmmm annicrobbatci www.ssizeeeeeeeee per-gpu
32
actor.se eetoruuse_kl_ooe _l_soss
true
actor.kl loss coef actor.kl loorrklloossttppee actor.entropy -ccfff .ceeff rollout.tensor model parallel size
0.001
low_var kl
0
2
pellout.gpummmmoe utilization rollout.n
0.6
5
trainer.total_epoee
trainer.n, -aannnppppppppeeee Basl__ee__ooo_ _noe
trainer.nnodes
I ADDITIONAL EXPERIMENTAL RESULTS
1.1 QUALITATIVE STUDY OF REASONING MODELS
As discussed in the main text, general-purpose reasoning models often "overthink" proto- col generation, producing overly long procedures with redundant actions and no clear stop- ping criteria. In the permeability assay, the gold-standard protocol requires only four steps (wash-remove retue-- wash) to ensure structural preservation and experimental efficiency. Thoth closely followed this structure, preserving both the correct sequence and action fidelity. In contrast, Grok 4 expanded the task into 13-step immunostaining workflow, adding permeabilization, block- ing, antibody staining, and repeated washes far beyond intended fixation step. Although logically coherent within its reasoning chain, this output diverges from laboratory practice, where clarity and parsimony are essential. This case underscores a broader limitation of general-purposs reasoning models: their tendency to prioritize exhaustive reasoning practical executability, reinforcing our quantitative findings that domain-adand addate- models like Thoth achieve more concise, faithful, and actionable protocols.
25

---

Grok
Query:
"Generation Human Conditionally Imnortalized Cell-based Multinellular Spheroidal Blood-Brain Barriar Model for Permsability Evaluntion of Macromolacules' protocol, what are the exact sequential steps required for fixing and staining the spheroids during the permeability assay to ensure proper preservation and preparation for microscopy?
Ground Truth:
Step 1: Wash the spheroids tuice with 500 of ice-cold D-PBS|+).
Step Remove the supernstant with pipette.
Step incubate
spheroids with 500 Step 4: Wash the spheroids twice with 500 of ioe cold D-PBS[+).
temperature for
Model Output:
<thinks
The goal is outline the sequential steps for fixing and staining spheroids in the specified blood-brain barrier model protocol during the permesbility assay, ensuring preservation of structure and immunostaining procadure
preparation micreacopy. hypothesis will enable visualization macromalecule permeability without compramising spheroid integrity. The sequence starts with removing medium to prepare for fixation, followed by washing, fixing, per permesbiliznng blocking, antibody staining, techniques that sequentially preserve, permeabilize, and label cellular components while minimizing background and artifacts. </think>
standard
washes; this order justified by established
<key>
Step ("action": "iaiiin"""rrnaa "remove" /'ojects'' ("mediumm) 'parameters' :{}
('action's "waonn "object' ('spheroids" "perameters" pbs"])
Step 3: Faction"""ncubate "objects ! ['spheroids"] loroodds''""" pprrrrmttrrs ("'n 4% pfa","3 min", "room temperature B
Step 4: ('action"/uaahi "uash""""jja"" 'objects' ('spheraids") "parameters" 1x pbs"," times']:
Step ('action': "eation":'ineeeen diniuunnt"neuatte" "'ojectss" sphereids'' pegherids temperature'))
triton x-100',
Step 6: "action"/uash" "objects' ('spheraids") "ppramtters 1x pbs", 133 times')
Step ('action': "idioon""incae finciont'"nu "'bjects's spphrrriddl.ppprmtttrsss///// blocking temperature"]
h'.rcom
Step 8: "inciin" "objects X sphercids''] cidds "parameters' inin primary antibody'. "overnight"," "e"])
Step ('action' Twash", 'objects' I I'spheroids" "parameters" ("with phs", times']
Step 10: "aationn""nnuuate" "bojects'' '('sheroodss' p.aamamtterrr' ['In sacondary antibody"," h"o'romm temperature"]
Step 11: ("action) wwash", "okjects' i' "spheridds') "parameters' ('with pbs","3 times"]
Step 12: l'action"/iccuaatte" "bjjatss' ('spheroids')
parammttrr parameters' dapl", min' "room temperature")
Step 13: ("action'/uash "'sh''tt"'''""t''''' 'objects' sphercids"). "parameters' ("ith 1x pbs",'3 times"] </key>
Step Removs the medium..
Step the spheroids with pbs.
Step 3: Incubste the spheroids in 4% přa for 30 min at temperature.
Step Wash the spheroids with fx phs 3 times.
Step Incubate
spheroids
at room temperature.
Step 6: Wash the spheroids with 1x pbs 3 times.
Step 7: incubate the spheroids in blocking buffer for h at room temperature.
Step Incubate
spheroids primary antibody overnight Step Wash the spheroids with fx pbs 3 times.
Step 10: Incubate the spheroids in secondary antibody for h at room temperature.
Step 11: wlash spheroids with fx pbs times.
Step 12: incubate the spheroids dapi for min temperature.
Step 13: wash the spheroids with 1x pbs 3 times.
</ore>
<note>
Handle poraformalderyde and triton x-100 with care a fume hood using gloves and eye protection, as they hacardous chemicals that skin irritation, respirstory issues, toxicity; disposal of waste to avoid environmental contaminstion. </note>
Figure 7: Case study of the reasoning model Grok 4 on Protocol-Comprehension task.
26

---

OpenAI of
Query:
In the "Generation of Human Conditionaily Immortalized Cell-hased Multicellular Spheroidal Blood- Bran Barrier Model for Permeability Evaluation Macromolecules" protocal, what are the exact sequential steps required fixing and staining spheroids during permeability assay to ensure proper preservation and preparation for microscopy?
Ground Truth:
Step 1: Wash the spharoids taice with 500 pi of ice-cold D-PBS[+).
Step Remove supernstant 500 yl of 4% PFA at room temperature for 10 min.
pipette.
Step 3: Incubate the spheroids with
Step 4: wash the spheroids taice with 500 pl of ice-cold D-PBS[+).
Model Output:
<think>
We will sequentially remove old medium, wash, fix with paraformaldehyde, remove fixative, wash, permeabilize, wash, incubata with primary secondary antibodies, and removal before imaging. Each only "incubate" "wash", "remove" to maintain consistency with the allowed actions.
</think>
<key>
Step "remive"""reeve" ''ojject'' ("culture medium") ]. "parameters" ['from wells"]
('action" "action""wesh"", 'objects' ("spheroids'] "parameters" phs', "'nnce" ('t"")
Step 3: ("action"""rmouu" "objects' ("wash buffer"], "aaerr"]paaamttrrss]]l
Step "aaction""incubate"" "bojecsss"" ["spheroids] "oaidds""ppaammmn 1'paarameter's ('45% miin""1
"'action"rremove" "objects' ("Fixative "parameters':))
Step 6: ("action""uussh""d"" 'objett' "spherolds'] "parameters ["xx phs', "tuioe"(rt"))
Step ("action"'remowe "i'movee""objctt""" "objects' l'wash buffer"), "aaffer".prrrmmtterrss"
Step ('actiio""iincuaatee "ojjctss'' ['spheroids] p.ppaaattrsss triton x-100" min",'re')
Step ('action""remoxe" "raoorrreeeeeeeeee""""tt ''ojectt'' "dbjeccts((peermebblliiii peerrmabilization solution"),
Step 10: ("action": wesh"" "objects 'I "spherodds' [. parammters''""M ponc""one""""
Step 1f: ("acti"n"
'objects
buffer'], paramsttrrsst
Step 12: ('actiin""nnuuatt""" "objects' "ajjects"("spheroidd]]] paapiiddddppaammmterrrssss (epttttrrrr"""pppaaa ('primary antibodyy. h",'rt")
Step 13: Doction"""remove" 'objects antibody solution"] "paarmmterss"]]]
Step 14: ("action': "wssh, 'ohjects' ['spheroids ]. "parameters' ["1x pls"", pba",onne"" "oon""""rr'")
Step 15: ("actian":"rmows "objects' wash "ufferr'.pppaaneen paramsttrss'
Step 16: ("action' inncuhot"" objecct"""sspppiia"""""" (sspheroisos ".arraaaatttrrss ("sacondary antibody",' W,reD
Step 17: ("action" "objects"' 'f l'antibody solution'] (. p.
18: "ssction""wash"" "objects" of ['spheroids" 'parameters's pbs","pbs"""oo" pbs"""onee" "o"ee""""""""
Step 19: Daction":'rrmooe "objects' "wahh buffer'), "poramsters"[
</key>
Step Remove culture medium from wells.
spheroids pbs
Step 3:
Remove wash buffer.
Step 4: Incubate spheroids with 4% pfa for 15 min at rt.
Step
Step Wash spheroids with fx pbs twice
Step 7: Remove wash buffer.
Step Incubate spheroids
triton x-100 for min rt.
Step Remoue parmeabilization solution.
Step 10: Wash spheroids with 1x pbs ence at rt.
Step 11: Remove buffer.
12: Incubate spheroids with primary antihody for h
Step 13: Remove antibody solution.
Step 14: Wash spheroids with 1x pbs once at rt.
Step
buffer.
Step 16: Incubette spheroids with secondary antibody for
Step 17: Remoue antibady solution.
Step Wash spheroids pbs once
Step 19: Remove wash buffer.
</orc>
<note>
Wear personal protective equipment, handle paraformaldehyde in ventilated area, and dispose of hazardous solutions according to institutional safeeyy guidelines.
</hotes
Figure 8: Case study of the reasoning model ol on Protocol-Comprrhensiie tasks.
27

---

Grok 4
Query:
If the berlin [1800 ml vessel) normally hold distilled immersing basket unsvailable, how can the QuantiSlakeTest he adapted using smaller vessels while still ensuring accurate immersion and balance measurements?
Ground Truth:
Step 1: Measure 1800 ml of distilled water.
Step Transfer the distilled water into an alternative container.
Step Suspend the empty basket the distilled water without touching sides.
Step 4: Tara the balance with the backet immersed.
Model Output:
<thinks
The goal is to adapt the QuantiSlakeees use smaller vessels instead of the unavailable 1800 berlin vessel for holding distilled water and inmersing the basket, while ensuring balance meazurements, Hypothesiz/plan of attack: the test can be adapted by conducting it in smaller vessel with appropriate water volume, ensuring basket immersion through suspension and restriction, achieving
immersion and
by taring, measuring key points, recapturing slaked material, and sorubbing to recover all residue for complete quantification. Justification smaller vessels may limit water volume and affect immersion stability or residue recovery, so the sequence starts with transferring water. suspending restricting taring and measuring initially, then post-slaking recapture, sorubbing for residue, transferring residue, and final measuring to ensure all material is accounted for, minimizing errors in smallar-scale setup. </think>
cleaning
secure
<key>
Step ("action' "scrub", "objects" (aplecttss"'smaallle ['smaller vessel"), "parameters
Step 2: ("action" "transfer", "objects' ("distilled water') parameters' " ('to smaller vessel"))
Step 3: ("action""susparn"" "suspeedd;; "objects" (bbakke")) "aasket).ppaaameterr "parameters ['in smaller vessel']
Step 4: ("action" "reesrict"""ojjectt" "objects", basket")] 'parameters's to ensure immersion"). ']}
"restrict"
Step ("action""les "tare", 'objects" (rreii""""a ("balance") I'parameters']] (1)
Stap 6: ("action' "measure", "aaassur"""'bjjae""" 'objects' ( ('immersed basket') J. 'parameters' [ ('using balance']]
Step 7: ("action' /vecapturr "bojects'' | slakee material'), "parameters' ("rrmm smaller vessel") "]]
Step ('action' i'serb'"""" 'objects' I'smailer vessel"). pporameters' oo'rss''''t
residue
Step 9: ('action' "transfer", 'objects' { ['residue") porameters''("M slaked material"]
"obnect"""ooject''' "'bjects"' "eetin"ttm
Step 10: "mesurr"""
slaked material 'parameters' ('using balance"]]
</key>
<orG>
Stap
Step 2: Transfer distilled water the smaller vessel.
Step 3: Suspend the basket in the smaller vessel.
Stap Restrict
ensure immersion.
Step Tare the balance.
Step 6: Measure the immersed basket using the balance.
Step 7: Recapture the slaked material from the smaller vessel.
Step 8: Scrub the smaller wessel
residue.
Step 9: Transfer the residue the slaked material.
Step 10: Measure the slaked material using the balance.
</orc>
<note>
the balance is stable level before taring measuring prevent readings tip- overs, and wear protective gloves during scrubbing, transferring, and recapturing to avoid skin contact with potential residues or contaminants. </notes
Figure Case study of the reasoning model Grok 4 on Problem- -oolinng task.
28

---

OpenAI of
Query:
the berlin (1800 ml vessel] normally used to hold distilled water for immersing the basket is unavailable, ensuring accurate immersion
QuantišlakeTest adapted using smallar balance measurements?
Ground Truth:
Step
1800 mi of distilled
Step 2: Transfer the distilled water into altemative container.
Step 3: Suspend the empty basket in the distilled water without touching the sides.
Step the balance with basket immersed.
Model Output:
<thinks
We will employ smaller vessels and ensure precise immersion by sorubbing them clean, taring the scale with each empty vessel, transferring distilled water, verifying the fill and adjusting necessary, suspending the </thinks
measurement, finally recapturing water reuse.
<key>
Step ("action' ''sorub', "objeb'''bjects' :[smmaler vessels'] [.""]]]'paaaa ''parameters' 'ooo remove residue") "))
Step ("action' "taiiin ("ssall")] p''aarmmeters ('with empty vessel") }}
Step "raction//transeee 'objects' ('distilled water') ''porametar'' ['into vessel }}
Step ( 'objects' ('filled vessal"), parrameters'' ('to confirm total mass")) "reetrict', "objects' water level"), parameters' required volume']] B Step ("action's F "oojecsd"""ojeccts""" "objects" (baskke" ]. "parameters's :i'n vessel", for immersion") }
Step 5: ("acttoon/rrsttiitt""
Step ("action': "actionn:"mesurae 'objects' ("basket wsight") parameterr ("for
reading Step 8: ("action" "recapture"; "objects"' "objects' ('distilled water'), parrmaterrs' { for reuse") ")
</key>
<orc>
Step Scrub the smaller vessels to remove residue.
Step
empty vessel.
Step Transfer distilled water into vessel.
Step Measure the filled vessel confirm total mass.
Step
required
Step Suspend the basket vessel for immersion.
Step Measure the basket weight for accurate reading.
Step Recapture distillad
reuse.
</orc>
Always vessel's stablity before suspending objects, all surfaces remain dry pravent accidental spills or scale errors.
</note>
Figure Case study of the reasoning model ol on Problem-Solving task.
29

---

1.2 FINE-GRAINED RESULTS ON SCIRECIPE-EVAL
Table 9: Main results on SciRecipe-EE at level 1. Metrics left of the dashed line evaluate exc- cutability, those the right measure semantie similarity. Bold denotes the best score.
Methods
Semantic-A
Order-LC'S
Order-S
Trdee-Tan
Step-M
BLUE-AVG
ROUGEL
METEOR
KW-FI
AVG
Clov-Sourvv.
ChatGPT-0o
66.44
39.49
80.04
40.98
52.11
48.02
55.37
55
GPT-5
32.52
63.40
19.27
63.63
27.62
22.34
35.92
35.41
42.61
38.11
GPT-5
48.74
Clande Sunnet
44.35
33.70
76.90
43.61
51.01
Claude Opus 4.1
45.43
75.42
34,82
76.83
46.89
36.29
$6.90
42.14
52.36
50.79
Gemini 2.5 Flash
39.66
75.32
33.70
75,66
43.91
34.60
46.21
42.55
50.68
49.14
76.50
1.5pro Qwen2.5-Max
37.14
78.42
38:47
76,00
60.23
30.78
49.57
41.77
$1.50
52
45.77
78.43
37.78
79.28
61.93
31.96
51.43
47.16
55.29
$4.34
Open
Owea2.5:72B-Lastrun
40.13
77.15
37.10
77.93
59.21
31.90
49.57
46.38
52.54
52.49
Qwen3-238BB -a-.--255--A22----
39.68
78.19
35.08
78.28
54.82
35.68
48.10
47.11
$1.20
52.02
DeepSeck-V3
48.04
$4.55
GPT-OSS-120B:
36.56
Llama-3 3.1-aa--.--- 1.--065B--nstruu
39.50
74.12
28.34
73.06
49.50
37.28
47.38
45.11
$0.29
49.46
Kimi E2-istrrction
42.58
45.78
Reasoning Model'
DeepSeck-RR
37.86
77.08
35.43
76.35
45.97
40.8
48.06
42.45
$2.56
50.83
Grok
41.53
78.84
37.68
77.96
54.19
57.12
49.82
49.98
52.28
53.27
OpenAl
39.78
77.70
28.00
74,43
43.02
37.25
59.23
46.19
53.12
49.86
OpenAl o3
42.15
74.94
26.02
72.95
36.29
31.10
46.42
46.78
53.59
47.80
Qwen3-4B
$4.93
Theth-mini
Qwen3-SB
29.08
68.08
21.12
65.00
38.80
17.08
34.59
36.67
40.15
39.15
Thoth
52.21
80.12
41.19
83.20
62.37
46.41
$2.69
49.21
56.88
58.24
Table 10: Main results on SciRecipe-Eval at level 2. Metrics left of the dashed line evaluate exe- cutability, those on the right measure semantic similarity. Bold denotes the best score.
Step-M
Close Source SOTA ChatGPT-4o
35.52
67.82
9.12
61.01
32.00
37.00
64.58
41.44
48.57
42.00
GPI-5
22.96
43.87
20.33
29.81
35.87
Claude Somnet
36.55
67.68
8.47
63.58
24.48
32.64
41.13
38.44
45.08
39 64
Clande Opus
37.38
67.18
22.78
33.16
38.65
47.34
40.71
2.5
Gemini 2.5 Pro
32.94
68.04
8.14
64.04
24.16
3009
41.29
43.43
46.61
39.89
Dombao-1.5-pe 1.5pro Qwen2.5-Max
29.67
68.37
9.46
64.24
35.27
36.61
44.30
55.77
46.00
41 09
Open Source SOTA
Qwen2.5-72B Lnstructtttutt
32.82
64.74
5.54
60.76
25.80
27.30
42,40
40.43
47.04
38.54
Owen3-235B-A225 1AA-E--A22---sste---555-- 2507
DeepSeek-V3
CPT-OSS-1208
20.31
64.07
4.23
$4.56
12.06
27.46
30.73
39.71
45.5 50
35
Lluma-3.1-6 1omma-3.--005---mm
311-403T-Instruc
32.20
64.70
8.13
62.71
29.37
35.60
42.44
39.35
45.98
40.05
K2-Instruction
61.41
Recsoning Models
DeepSeck-R1
34.36
65.91
5.91
62.18
20.33
38.83
42,80
54.
47.31
59.08
Grok
31.17
6.16
35.58
56.78
OpenAl ol
29.90
69.27
9,64
60.75
28.06
34.18
43.47
41.55
47.26
40.43
OpenAl
Owen3-4B
Theth-mini
39.38
69.18
8.47
61.74
41.16
4037
46/66
44
50.40
44
Qwen3-SB
27.85
1.62
52.59
10.13
15.40
30.12
32.85
37.17
29.68
30

---

Table 11: Main results SciRecipe- Eval (Overview). Metries left of the dashed line evaluate executability, those on the right measure semantic similarity. Bold denotes the best score.
Methods
Semantic-A
Ordlrr-LCS
Order-S
Order-Tau
Step-M
BLUE-AVG
ROUGE-L
METEOR
KW-FI
AVG
Close Source SOTA
ChatGPT-4o
32.36
72:47
13.33
72.02
30.67
34.24
42.36
57.24
44.03
42.08
GPT-5 Chat
30.99
70.02
10.01
68.01
11:33
21.35
32.70
52.98
38.10
35.05
Clande Somnet
30.08
72.64
16,00
71.33
30.67
31.42
38.94
35.62
41.30
40
Opus
Gemini 2.5 Flash
28.87
70.62
12.67
73.67
25.00
31,85
39.04
33.00
41.57
39 39.70
Gemini 2.5 Pro
28.96
71.83
12.00
75.35
26.67
28.04
38.99
38.83
41.38
40.23
Domhno-1.55 1.5-pr
Qwen25-Max
34.73
10.00
76.67
28.11
42.12
36.27
42.17
Open- Sowrre SOTA
Bien225-72B-Lmstree
30.22
10.00
75.35
27.35
24.89
39.97
37.00
42.17
39.59
28.70
70.70
D3
35.24
73.23
15.54
71.01
57.16
35.33
43.43
39.58
43.80
43.82
GPT-OSS-120R
27.34
67.20
10.01
64.68
20.00
24.75
36.10
56.71
39.61
56.28
Llama-3.1- Gamma-31-405BBImme
Kimi- K2-Instructionn
30.92
72:21
12.00
74.87
36.67
30.90
39.20
37.88
41.38
41.74
Reasoning Models
DeepSeck-R1
40.45
52.09
12.42
Grok
32.19
72.43
12.52
67.45
20.30
36.06
42.33
56. 12
45:54
41.55
OpenAl ol.
21.78
72.50
12.67
70.70
31.33
31.80
40.20
37.84
60.75
40.65
OpenAl
72.00
Owen3-48
28.03
62.52
N.02
62.00
24.00
13.46
24.50
25.40)
27.38
30.39
Thath-mini
37.49
71.74
11:34
71.67
38,34
38.02
42.61
58.55
43.79
43.73
Qwen3-5B
Theth
40.15
71.01
37.75
39.09
Table 12: Main results SciRecipe-Evas (Specific). Metrics left of the dashed line evaluate cutability, those on the right measure semantic similarity. Bold denotes the best score.
Methods
Semantic-A
Order-LLCC Order-S
Order-Tau
Sup-M
BLUE-AVG
ROUGE-L
METEOR
KW-FI
Close-Source SOTA
ChatGPT-4
33.63
70.95
24.00
67.32
48.67
36.26
45.07
41.29
45.81
46
GPI-5 Chat
27.99
71.05
20.67
64.67
18.67
23.93
36.23
36.46
39.56
37.09
Clande Sonnet
36.39
66.46
20.67
66.67
37.33
33.11
40.55
35.55
42.58
42
Opus
33.60
Gemini 2.5 Flash
36.67
68:25
20.00
70.33
30.67
31.72
40.34
36.03
43.34
11.93
Gemini 2.5 Pro
35.48
71.68
23.33
69.33
30.00
27.89
80.09
41.81
42.08
42.41
Dombao-1.5-pre
Qwe.25-Max
64.67
40.97
Opea-Sourv S03A
Owee2,5-728-Insiieeee
36.45
68.15
23.33
67.33
43.33
28.58
43.82
41.51
64.89
44.19
Owen3-235B
Deg
37.54
70.37
17.99
69.34
39.33
33.72
43.54
40.02
05.49
44.29
GPI-0SS-1208
32.1
68.77
22.67
64.67
35.33
30.02
42.74
40.32
46.92
42.69
36.53
69.21
20.67
70.00
35.67
32.87
$1.68
38.99
45.07
43.74
Reasonnng Models
DeepSeek-R1
32.25
44.28
Grok
37.04
68.65
16.07
66.00
34.22
35.45
45.09
$7.99
48.54
43.27
OpenATol
31.77
70.59
18:12
63.10
34.33
33.31
43.55
40.49
45.53
42.33
OpenAl
Owen3-4B
23.34
59.81
19.33
58.00
41.33
16.21
28.38
26:77
29.94
33.68
Thoth-mini
45.21
75.14
28.68
69.67
54.34
42.41
48.91
45.34
51.32
51.22
15.02
35.41
Theth
31

---

Table 13: Main results SciRecipe- (Retrieval). Metrics left of the dashed line evaluate cutability, those on the right measure semantic similarity, Bold denotes the best score.
Methooss
Semantic-A
Ordlrr-LCS
Order-S
Order-Tau
Step-M
BLUE-AVG
ROUGE-L
METEOR
KW-FI
AVG
Close Source SOTA
ChatGPT-4o
51.78
80.87
42.00
73.99
60.00
46.27
58.29
53.51
61.29
56.67
GPT-5 Chat
54.81
81.86
40100
72.00
52.00
42.55
56.07
$4.05
61.58
57.21
Clande Somnet
53.10
80.66
40.00
78,00
48.00
44.13
55.60
50.88
60.91
$6.81
Opus
Gemini 2.5 Flash
48.78
36.00
71:00
33.37
50.58
47.87
56.35
52.52
Gemini 2.5 Pro
45.71
78.06
40.00
70.00
56.00
38.62
52.35
55.49
59.20
55.11
Domhno-1.55 1.5-pr
Qwen25-Max
$1.18
46.00
78.00
34.33
58.01
$1.88
99.62
57.65
Open- Sowrre SOTA
Owen2.5-72B-Lnsiien
50.37
70.00
70.00
35.23
56.68
53.80
60.13
58.08
70.00
D3
57.83
80.5
42.33
69.45
53.06
40.00
59.70
55.70
62.06
58.09
GPT-OSS-120R
47.62
80.48
38.00
78,09
24.00
41.26
55.77
52.12
59.97
55
Llama-3.1-6 3.mma-31-405BBInms
Kimi- K2-Instructionn
45.13
79.94
38.00
74,00
45.00
41.45
52.77
50.93
58.39
$4.31
Reasoning Models
DeqpSeck-R1
44.54
44.71
10.18
Grok
44.04
76.50
30.33
68.00
40.67
30.83
51.44
43.08
$7.84
50 19.19
OpenAl ol
50.51
80.18
32.00
78.00
42.00
41.74
55.37
51.96
60.02
$1.41
OpenAl
Owen3-48
21.50
43.11
18.00
38.00
22.00
12.44
21.30
20.96
24.30
24.70
Thath-mini
59.35
85.47
45.08
76.34
66.34
50.29
59.62
57.24
64.06
62.74
Qwen3-5B
Theth
$4.51
75.65
64.92
Table 14: Main results SciRecipe Evall (Planning). Metrics left of the dashed line evaluate cutability, those on the right measure semantic similarity. Bold denotes the best score.
Methods
Semantic-A Order-LCCS Order-S Order-Tau
Sup-M
BLUE-AVG
ROUGE-L
METEOR
KW-FI
Clase-Source SOTA
ChatGPT-4
44.30
76.84
34.01
79.99
60.00
42.65
53.16
49.96
56.70
55.28
GPI-5 Chat
33.90
75.02
22.00
62.00
26.00
32.32
43.75
47.28
52.21
44.13
Clande Sonnet
38.06
73.34
22.00
64.00
36.00
35.42
47.67
44.25
52.51
45.92
Opus
Gemini 2.5 Flash
80.29
76.34
25.00
65.00
25.00
33.95
44.78
42.62
50.99
45.33
Gemini 2.5 Pro
41.38
75.86
26.00
68.00
36.00
35.37
$0.58
51.43
$4.52
65.79
Dombao-1.5-pre
34.52
Qwe.25-Max
22.00
62.00
32.97
Opea-Sourv S03%
Oween,55728-IIsiteeee
39.57
73.26
24.01
64.00
50.00
32.45
51,16
48.38
$4.94
48 64
Owen3-235B
A228-Lastreet-2500
22.00
Deg
45.53
76.42
21.98
76.01
46.00
40.97
$2.75
49.20
$7.95
$1.87
GPI-0SS-1208
37.09
71.76
18.00
61.99
32.00
34.02
46.87
43.50
56.35
45
33.59
71.64
25.00
62.00
$2.00
35.83
18.26
47.39
51.41
47.57
Reasonnng Models
DeepSeek-R1
43.05
Grok
39.39
76.44
34.63
63.54
45.69
30.96
$1.50
44.76
56.70
50.27
OpenATol
35.64
75.73
20.00
64.00
42.00
37.93
49.47
48.84
$4.52
47.57
OpenAl
Owen3-4B
28.50
48.20
13:99
48.00
25.00
14.34
23.55
22.56
25.90
28:26
Thoth-mini
48.05
79.19
32.00
76.34
60.34
44.60
52.66
49.67
57.45
55.59
Qwen3-5B
Theth
32

---

Table 15: Main results SciRecipe-Ese (Troubleshooting). Metrics left of the dashed line evaluate executability, those on the right measure semantic similarity. Bold denotes the best score.
Methods
Semantic-A
Orderr-LCS
Order-S
Order-Tau
Step-M
BLUE-AVG
ROUGE-L
METEOR
KW-FI
AVG
Close Source SOTA
ChatGPT-4o
26.46
62.76
10.00
53.99
26.00
32.80
41.50
40.10
47.90
37.75
16.24
GPT-5 Chat
26.45
63.98
8.00
46,00
16.00
25.00
39.08
58.21
46.97
15.15
Clande Somnet
24.92
63.31
3.8
$2.00
9.99
23.83
37.34
58.26
45.77
33.25
Opus
Gemini 2.5 Flash
28.76
6,00
58.99
27.13
36.19
36.38
36
Gemini 2.5 Pro
22.05
61.00
10.00
32.00
16.00
25.36
37.48
40.39
45.48
34.42
Domhno-1.55 1.5-pr
Qwen25-Max
22,66
8.00
$5.97
29.08
39.97
47.33
40.27
Open- Sowrre SOTA
Bien225-72B-Lmstree
18.12
8,00
34.00
25.78
39.30
44.15
36.02
11.97
26.47
D3
30.52
65:57
3.97
47.38
32.00
33.12
45.59
42.50
49.35
39.00
GPT-OSS-120R
26.48
$7.03
11.00
46.00
0.00
22.57
35.45
56.83
46.00
50.05
Llama-3.1-- 3amma-31-405BBlmme
Kimi- K2-Instructionn
23.10
60.74
1.95
33.98
16.00
24.85
36.06
39.49
44.33
33.46
Reasoning Models
DeepSeck-R1
26.57
Grok
19.08
61.60
5.30
35.62
19.26
27.40
36.75
35.81
44.64
$1.72
OpenAl ol.
18.78
66.89
4,00
48.00
16.00
29.11
60,40
40.04
46.53
34.42
OpenAl
38.84
29.91
Owen3-48
17.53
5283
8.00
41.00
22.00
16.22
24.78
24.04
29.37
26.09
Thath-mini
35.15
67.81
11:98
58.51
50.31
40.65
44.95
41.82
52.04
44.78
Qwen3-5B
Theth
13.99
42.02
Table 16: Main results SciRecipe- Eval (Constraint). Metrics left of the dashed line evaluate executability, those on the right measure semantic similarity. Bold denotes the best score.
Methods
Semantic-A
Order-LCCS Order-S Order-Tan
SuppM
BLUE-AVG
ROUGE-L
METEOR
KW-FI
Close-Source SOTA
ChatGPT-4
40.34
74.56
30.00
73.99
53.08
30.94
50.87
48.34
55.33
$1.93
GPI-5 Chat
33.40
78.43
30.00
72.00
44.00
37.21
50..7
47.02
54.26
49.02
Clande Sonnet
48.30
73.30
17.97
$2.00
33.97
32.52
46.00
41 46
53.02
47
Opus
Gemini 2.5 Flash
36.61
68.91
11.99
68.97
31.97
33.88
44.40
39.63
50.71
43.01
Gemini 2.5 Pro
32.36
74.58
25.97
75,00
33.99
33.30
47.75
46.43
51.12
46.61
Dombao-1.5-pre
Qwe.25-Max
40.38
77.97
Opea-Sourv S03%
Owen2,5-728-Insiieeee
33:26
67.67
20.00
64.00
66.00
31.00
66.22
43.65
$2.61
44.93
Owen3-235B
A228-Lastreet22500
Deg
35.01
72.93
17.97
75.98
35.99
33.49
09.54
46.21
55.19
47.48
GPI-0SS-1208
18.39
70.31
12.00
62.00
23.97
20.09
43.66
44.23
51.98
39
35.50
74.28
21.97
71.97
31.98
30.80
45.14
44.00
51.35
45.22
Reasonnng Models
DeepSeek-R1
44.65
70.64
Grok
37.78
75.55
26.73
61.81
38.31
30.79
50.63
43.57
57.57
43.75
OpenATol
29.63
74.94
21.99
72.00
40.00
37.58
49.99
48.57
55.81
47.83
OpenAl
Owen3-4B
14.71
34.91
8.00
30.00
21.97
9.76
16.13
15.51
18:47
18.83
Thoth-mini
35.26
74.35
29.98
70.31
64.31
42222
50.53
48.19
54.34
52.28
Qwen3-5B
Theth
33

---

Table 17: Main results SciRecipe-Eval (Scaling). Metrics left of the dashed line evaluate cutability, those on the right measure semantic similarity. Bold denotes the best score.
Methooss
Semantic-A
Orderr-LCS
Order-S
Order-Tau
Step-M
BLUE-AVG
ROUGE-L
METEOR
KW-FI
AVG
Close Source SOTA
ChatGPT-4o
74.95
77.87
38,00
71:99
50.00
51.36
62.12
58.19
69.93
61 60
12:24
GPT-5 Chat
62.62
80.70
44.00
70.00
50.00
45.85
59.93
$7.32
70.79
60.13
Clande Somnet
63.92
78.50
32,00
72.00
46.00
44.63
56.48
$2.89
78
$7.36
Opus
Gemini 2.5 Flash
53.67
36.00
71.00
39.74
$4.01
49.62
64.62
$4.03
Gemini 2.5 Pro
36.34
78,30
32.00
75.00
34.00
38.05
54,83
59.07
64.73
55.09
Domhno-1.55 1.5-pr
Qwen25-Max
65.70
40.00
$2.00
61.83
$5.95
70.13
60.52
Open- Sowrre SOTA
Owen2.5-72B-Lnsiien
62.21
34.00
76.00
35.97
58.1
56.04
68.89
57.48
74.00
56.91
D3
65.90
83.35
43.97
75.36
50000
50.00
63.04
58.99
71.61
63.70
GPT-OSS-120R
47.37
79.96
34.00
70.00
50.00
44.23
56.22
$4.42
69:45
56.23
Llama-3.1-- 3amma-31-400BBInmss
Kimi- K2-Instractionn
62.16
70.45
38.00
70.00
52.00
44.02
39.76
53.93
68:27
53.35
Reasoning Models
DeepSeck-R1
72.64
Grok
48.13
70.11
37.75
78.98
30.74
46.50
57.48
50.66
67.51
56.23
OpenAl ol.
60.64
77.34
28.00
70.00
64.00
46.62
58.50
52.93
69.43
56.42
OpenAl
72.00
42.92
Owen3-48
24.38
20.08
18.00
41100
26.00
16.51
25.92
22.42
30.26
27.71
Thath-mini
68.87
85.46
47.98
88.51
68.34
$4.25
63.00
62.71
72:65
68.05
Qwen3-5B
Theth
93.65
63.61
74.15
Table 18: Main results SciRecipe-Evae (Safety). Metrics left of the dashed line evaluate cutability, those on the right measure semantic similarity. Bold denotes the best score.
Methods
Semantic-A
Order-LLCC Order-S Order-Taw
Sup-M
BLUE-AVG
ROUGE-L
METEOR
KW-FI
Close-Source SOTA
ChatGPT-4
46.78
76.08
22.00
71:99
40.00
42.88
$2.51
50.23
61.04
$1.50
33.00
GPI-5 Chat
47.42
72.72
18.00
68.00
22.00
32.27
43.09
51.20
55.03
45.58
Clande Sonnet
44.59
77.33
24.00
78.00
$2.00
36.76
49.68
50.35
57.72
$2.27
Opus
Gemini 2.5 Flash
33.47
72.38
26.00
77.00
35.00
34.50
46.52
47.71
53.76
47.70
Gemini 2.5 Pro
38.44
72.67
22.00
66.00
35.00
37.95
19.69
$2.26
$7.53
48 28
Dombao-1.5-pre
Qwen25-Max
46.16
14.00
76.00
Opea-Sourv S03A
Oween,55728-Insiieeee
32,56
72.62
22.00
72.00
48.00
33.76
69.18
46.46
$4.96
47.97
Owen3-235B
17.57
Deg
66.21
77.12
23.97
80.00
44.00
42.43
52.35
49.57
60.15
$2.87
GPI-0SS-1208
36.72
71.92
12.00
66.00
15.00
33.16
46.79
47.61
55.11
43.03
02.05
74.65
25.00
72.00
$4.00
36.86
18.65
49.53
55.30
$1.00
Reasonnng Models
DeepSeek-R1
40.83
46.94
Grok
42.15
74.32
22.49
73.08
41.60
33.51
48.18
41.98
55.22
48.62
OpenATol
43.03
74.45
24.00
76.00
44.00
39.85
50.20
48.51
56.41
$0.94
OpenAl
Owen3-4B
25.45
55.88
15.00
50.00
30.00
15.56
25.76
25.40
31.22
30.70
Thoth-mini
36.58
62:74
15.98
56.33
44.34
46.04
41.54
45.62
51.00
4437
35.11
Theth
34

---

1.3 OTHER RESULTS ON PROTOCOL BENCHMARK
To further assess protocol comprehension, we evaluated Thoth on BioProBench sub-tasks, with results summarized in Table Consistent with the findings protocol generation, the Thoth series achieved substantial gains over baseline models, with average improvements 28.02% and 20.41% % for Thoth and Thoth-mini, respectively. Scaling effects were also more evident in this setting, as Thoth outperformed Thoth-mini by 6.98%. a gap larger than in protocol generation. On individual tasks, Thoth showed clear advantages in error correction and protocol QA, exceeding GPT-5 by 17.05% and 12.09%, respectively, and consistently outperforming other strong baselines such as Claude Sonnet and Gemini 2.5 Flash. These results confirm that Thoth generalizes well across protocol-related tasks. Nevertheless, because BioProBench partially overlaps with SciRecipe and does not disclose the exact origins of its QA pairs, there remains a possibility of data leakage, and thus the reported results should be regarded reference only.
Table 19: Results on the protocol QA tasks.
ERR
ORD
PQA
Methods
ACC
F1
EM
K tau
ACC
ChatGPT-4o
59.33
36.95
41.89
68.05
60.75
GPT-5
67.72
59.65
48.86
73.44
70.58
GPT-5 Chat
60.33
41.38
45.66
71.71
60.75
Claude Sonnet
62.92
54.64
47.71
72.31
67.08
Gemini 2.5 Flash
62.17
51.91
43.79
73.12
64.47
Qwen2.5-Max
54.83
17.38
41.51
67.96
62.25
Doubao-1.5-pro
56.43
24.49
43.38
63.00
Qwen2.5-72B-Instrue
56.08
25.25
36.99
62.46
60.42
Qwen3-235B-A222
59.33
39.15
41.63
69.18
62.58
DeepSeek-V3
57.33
30.25
40.83
70.56
62.83
GPT-OSS-120E
64.33
53.17
39.88
68.08
64.83
3lama-33114055--nsttee
57.30
39.33
39.06
65.42
57.67
Kini-K2-Instrrettinn
59.50
37.05
42.62
71.70
63.00
Qwen3-4B
58.00
48.78
22.53
51.82
47.08
Thoth-mini
75.83
72.49
41.72
65.90
74.33
Qwen3-8B
56.83
32.90
28.63
56.12
50.58
Thoth
80.75
80.73
49.08
71.92
82.67
1.4 COMPARISON WITH CHAIN-OF OH-TUGHHTT AND SKETCH-AND-FiN
To clarify the advantages of the Sketch and Fill paradigm for structured protocol generation, we extended our evaluation include several commonly used reasoning baselines. The central idea of Sketch and Fill is to separate global planning from step level realization so that the model is guided produce explicit and verifiable procedural structure. This design differs from standard chain of thought prompting, which encourages free form reasoning but does not ensure that the final protocol follows coherent step wise format.
In addition to the direct Fill baseline reported in the Figure we introduced two additional baselines. The first is Standard COT, where the model is instructed think step by step and then immediately generate the protocol. The second is Advanced CoT, where the model first produces an extended reasoning trace and then outputs the protocol without structural constraints. The comparison results in Table 20 show that Sketch and Fill consistently outperforms both Standard and Advanced CoT across all evaluated models and metries. This demonstrates that conventional multi stage reasoning alone is not sufficient for generating well structured protocols and that introducing explicit sketch stage provides measurable benefits.
I.5 HUMAN-METRIC CONSISTENCY ANALYSIS
To assess how well the SCORE metric reflects real experimental executability, conducted human evaluation study using protocols generated by Thoth and Thoth-mini. Since SciRecipe is constructed from real laboratory protocols collected from Bio-protocol, Protocols.io, and related
35

---

Table 20: Comparison of reasoning paradigms across models.
Metric
Paradigm
DeepSeck-V3
GPT-5 Chat
Qwen3-8B
Thoth
AVG
Standard CoT
5.98
5.12
5.01
13.03
7.29
Advanced CoT
9.00
6.40
7.70
29.58
13.17
BLUE-AVG
Fill
33.36
31.37
20.69
38.91
31.08
Sketch and Fill
38.17
29.57
16.66
43.25
31.91
Standard CoT
19.82
17.41
15.06
27.54
19.96
Advanced CoT
28.11
22.14
19.91
41.34
27.88
ROUGE-L
Fill
44.21
41.10
33.38
46.44
41.28
Sketch and Fill
48.46
42.04
32.31
49.13
42.99
Standard CoT
24.92
22.73
18.61
25.78
23.01
Advanced CoT
26.90
23.53
17.65
36.51
26.15
METEOR
Fill
41.29
38.03
36,36
44.01
39.92
Sketch and Fill
45.03
41.95
34.72
46.73
42.11
Standard CoT
26.07
22.74
19.59
33.14
25.39
Advanced CoT
35.20
29.40
23.79
47.28
33.92
KW-F1
Fill
49.64
46.24
37.43
50.63
45.99
Sketch and Fill
54.28
49.87
38.63
53.57
49.09
Table 21: Cohen's kappa agreement between SCORE and expert ratings for Thoth.
Tasks
Step_Scale
Order Order,Consistents Consiitency
Semantic Semantic,Consistere Consistency
Overall
Overview
0.82
0.79
0.74
0.76
Specific
0.80
0.77
0.72
0.74
Retrieval
0.85
0.82
0.76
0.79
Plamning
0.84
0.80
0.75
0.78
Troubleshooting
0.81
0.78
0.73
0.75
Constraint
0.86
0.83
0.77
0.80
Scaling
0.83
0.79
0.74
0.76
Safety
0.84
0.81
0.75
0.78
Levell
0.79
0.82
Level2
0.79
0.76
0.71
0.73
Overall
0.83
0.80
0.74
0.77
platforms, the dataset reflects authentic experimental structure and operational logic. The SCORE metric was designed on the basis of these real procedures, incorporating three components that capture key dimensions of protocol quality: step granularity, action order consistency, and semantic fidelity.
To examine the alignment between SCORE and expert judgment, three researchers with extensive wet-lab experience manually evaluated a subset of protocols from SciRecipe-Fval. Each protocol was rated on step granularity, operational order, and semantic clarity using a five-point scale ranging from O to 4. The raters also provided an overall score for fluency and executability. For every criterion, we averaged the three expert report the Cohen's 's kappa coefficients between SCORE and expert ratings for Thoth and Thoth-mini. These coefficients quantify the agreement between human assessment and the metric along each evaluation dimension. According to established interpretative guidelines (Landis & Koch 1977), the obtained kappa values indicate a meaningful degree of correspondence between SCORE and domain-cxpert judgments across all task categories.
and normalized the resulting values. Tables 21
2 and 22
In addition to its alignment with expert ratings, we further examined whether SCORE is sensitive to the types of critical errors that directly influence experimental executability. Common failure modes in model-ggeeratee protocols include incorrect ordering of dependent steps, omission of essential operations, and mismatches between actions and their intended objects. SCORE evaluates these issues along three complementary dimensions, namely step scale, order consistency, and semantic consistency, which jointly capture both structural and procedural correctness. To assess the validity
36

---

Table 22: Cohen's kappa agreement between SCORE and expert ratings for Thoth-mini.
Tasks
Step Scale
Order Consistency
Semantic Consistency
Overall
Overview
0.80
0.77
0.72
0.74
Specifie
0.78
0.75
0.70
0.72
Retrieval
0.83
0.80
0.74
0.77
Planning
0.82
0.78
0.73
0.76
Troubleshooting
0.79
0.76
0.73
Constraint
0.84
0.81
0.75
0.78
Scaling
0.77
0.72
0.74
Safety
0.82
0.79
0.73
0.76
Levell
0.86
0.83
0.77
0.80
Level2
0.77
0.74
0.69
0.71
Overall
0.81
0.78
0.72
0.75
of this design, two-fifths of the SciRecipe-Eval samples were reviewed by three researchers with extensive wet-lab experience. The experts annotated whether a protocol contained any сπто that could compromise experimental success and also rated overall completeness and logical coherence. Protocols containing such errors exhibited substantially lower SCORE values, with an average re- duction of 34.3% across tasks, and the separation between valid and faulty protocols was statistically significant (p 0.001).
Addtionally, treating SCORE binary detector using threshold equal to the mean between the two distributions, we obtained AUC of 0.92, recall of approximately 0.90, and precision of approximately 0.87 when compared with expert annotations. These results indicate that SCORE reliably identifies the same categories of critical procedural mistakes that domain experts deem detrimental. To further assess its comparative sensitivity, we conducted a pairwise ranking study using protocol pairs generated by Thoth and Thoth-mini. Experts were asked select the protocol with better executability and more coherent procedural flow, and SCORE agreed with these rankings 0.72). Overall, these findings demonstrate that SCORE is not
in 81.2% of cases (Kendall's a heuristic scoring scheme but principled metric that detects consequential procedural errors and exhibits robust consistency with expert judgments across both diagnostic and comparative evaluation settings.
I.6 QUANTITATIVE VALIDATION OF THE SKETCH-AND-FILL
The SAF representation is designed to capture the minimal semantic structure necessary for execut- ing experimental step. Across diverse protocols, each step can be decomposed into operation, a target object, and set conditions that specify how the action is performed. Encoding these elements (Action, Object, Parameters) triplet yields compact abstraction that preserves procedural meaning while reducing stylistic variability in natural language deseriptions. Branching logic does not conflict with this formulation, SAF encodes atomic actions while conditional or alternative paths operate at a higher structural level. Conditions can be expressed directly in the parameter field (for example, "if the pellet not visible"), or represented as parallel SAF entries corresponding to distinct experimental scenarios.
To quantitatively assess how well SAF captures the semantics of real protocols, we conducted a se- ries of analyses. First, a coverage study was performed on fifty protocols comprising 423 steps. Each step was manually mapped into SAF format, after which two domain experts evaluated whether the SAF representation preserved the essential information. The evaluation used three categories: fully covered, partially covered, and not covered. We observed that 85.6% of steps were fully covered and 10.4% were partially covered. Only 4.0% of steps could not be represented, and these were predominantly narrative or mntaa-xxpeimental statements rather than actionable instructions. Inter- annotator consistency was measured by independently encoding 200 randomly sampled steps. The resulting Cohen's 's kappa coefficients were 0.86 for actions, 0.82 for objects, and 0.78 for parameters, indicating that the schema is well-defined and yields stable mappings across annotators.
37

---

Table 23: Token-level conditional entropy across tasks and model families when conditioning on
SAF.
Tasks
Qwen2,5-7B
Qwen3-8B
DeepSeck-R1 Rl-Lserk-
Lluma-3.1-88
Phi-4-mini
M.3-rr
Mistral-7B
Overview
1.0223
0.8956
1,1342
0,9298
1,0003
0.8205
0.7393
Specifie
0,9383
0.8144
1.0419
0.8425
0,9614
0.7548
0.6478
Retrieval
0,9000
0.7991
1,0411
0.8208
0,9300
0.6915
0.6106
Planning
0,9636
0.8942
10246
Truubleshoting
0,7925
0.7848
0.9673
0,7316
0.8580
0.6439
0.6158
Constraint
0,9649
0.8432
1.0700
0.8477
0.9388
0.7302
0.6830
Sealing
0.8974
0.7999
10037
0.7906
0.8990
0.7193
0.6143
Safety
0,9129
0.8073
1.0161
0.7996
0,9316
0,7133
0.6231
Levell
0,9803
0.8862
0,9599
0.7877
Level2
0,9052
0.8083
1.0320
0.8141
0,9303
0.7058
0.6340
Overall
0,9240
0,8200
1.0460
0.8321
0.9467
0,7263
0.6489
We further examined SAF from an information-thherrtie perspective. Let X denote the natural language step description and Y denote its SAF representation. The remaining uncertainty after conditioning on SAF given by the conditional entropy
H(X Y)= EE p(r y)-
(19)
To operationalize this quantity, each SAF triplet was prepended to the original step text, and token- level probabilities were obtained from a language model. The conditional log probability was com- puted using the standard chain rule
log qxx (1) = log logg(t; y, tes).
(20)
where x (t1...... tx) the tokenized step, and q(x y) represents the model's approximation of p( 3). Conditioning on SAF consistently reduced token-level uncertainty across all SciRecipe- Eval task categories. Prior work estimates that natural language typically exhibits 2.5-4.0 nats of entropy per token (Goodman] [001] Kaplan et al. 2020)) Hoffmann et al. [0022). In contrast, the conditional entropy of experimental steps under SAF is substantially lower, indicating that the triplet encodes the majority of semantic information required to reconstruct the procedural description. This trend persists across multiple language model families, as shown in Table 23
J FUTHER ANALYSES
J.1 REWARD ANALYSES & RESULTS
J.1.1 RL ALGORITHMS
We conducted ablation experiments to assess the robustness of the SCORE mechanism under differ- ent RL algorithms, replacing the third-stage optimization algorithm while keeping earlier training stages consistent. As shown Table 24, SCORE remained stable across variety of methods, con- firming its reliability as reward mechanism. Executability metrics revealed that GRPO offered the best overall balance, benefiting from group-wise normalization that stabilized training and strengthened action fidelity. This supports choice of GRPO as the primary optimization method in Thoth.
J.1.2 REWARD COMPUTATIONS
We further examined the impact of different reward computation strategies within the SCORE mech- anism, considering the choice of order consistency logic (LCS or Strict), the method of combining order and semantic rewards (Product or Sum), and the integration of step scale with step seman- tics (Product Sum). As summarized in Table 25 strict order scoring consistently outperformed LCS, product- based integration proved more effective than summation, and multiplicative incorpo- ration of step scale with semantics yielded stronger executability. These trends collectively support the adoption of the S-S-P configuration in Thoth, which achieved the best balance between surface similarity and procedural fidelity.
38

---

Table 24: Further analysis results on RL algorithms.
Algorithm
BERTScore-
BERTScore- BERTScore-
BLEU-1
BLEU-2
F1
P
R
GPG
97.81
98.01
97.62
47.30
GRPO Dr
97.81
97.99
97.63
68.10
47.31
GSPO
97.81
97.98
97.65
68.65
47.36
REINFORCE
97.82
98.08
97.56
68.83
48.18
++-baseline
ReMax
97.81
98.05
97.57
67.93
47.87
RLOO
97.81
98.04
97.59
67.35
47.64
Thoth (GRPO)
97.82
97.99
97.65
68.56
48.28
Algorithm
BLEU-3
BLEU-4
BLEU--
ROUGE-1
ROUGE-2
AVG
GPG
34.44
21.80
43.03
59.98
36.58
GRPO_Dr
34.44
21.85
42.93
60.31
36.89
GSPO
33.50
22.08
42.90
37.11
REINFORCE
34.71
22.69
43.60
59.92
37.08
++basseiie
ReMax
33.59
22.79
43.05
59.96
36.59
RLOO
33.60
22.61
42.80
60.41
37.15
Thoth (GRPO)
34.58
23.09
43.62
60.30
37.14
(c)
Algorithm
ROUGE-L
Meteor
KW-F1
KW-P
KW-R
GPG
49.01
45.81
53.47
58.52
50.84
GRPO_Dr
49.12
47.14
53.98
59.29
51.06
GSPO
49.32
46.91
54.20
58.59
51.98
REINFORCE
49.19
46.19
53.59
49.71
++baseelne
ReMax
49.52
45.95
53.47
60.27
49.56
RLOO
50.00
46.66
53.65
49.94
Thoth (GRPO)
50.02
47.39
54.13
58.82
51.70
(d)
Algorithm
Order-LCS
Order-S
Order-Tau
Semantic-A
Step-M
GPG
74.02
22.50
70.33
45.07
46.67
GRPO
75.34
26.67
71.50
44.11
50.50
GSPO
74.93
23.67
68.67
43.93
51.17
REINFORCE
++-baseline
75.69
25.50
70.67
44.14
50.67
ReMax
73.58
23.00
69.33
44.89
$1.11
RLOO
74.42
25.00
70.17
43.87
51,50
Thoth (GRPO)
75.34
25.50
73.33
46.60
53.00
J.1.3 REWARD MAGNITUDES
To better understand the sensitivity of SCORE to the absolute scale feedback, we examined different strategies for adjusting reward magnitude. Four configurations were compared: constant scaling, scaling up, symmetric shifting with equal positive and negative ranges, and the default Thoth setting. As shown in Table 26 model performance remained highly stable across all strategies, with only minimal variations observed on both text-level and executability-orieme metrics. This confirms that the effectiveness of SCORE arises mainly from its structured reward design rather
39

---

Table 25: Further analysis results on Reward Computations.
Combine
BERTScore-
BERTScore-
BERTScore-
BLEU-1
BLEU-2
F1
P
R
L-P-P
97.76
97.85
63.17
44.25
L-P-S
97.73
97.80
97.66
62.56
43.74
L-S-P
97.78
97.87
97.69
63.23
44.37
L-S-S
97.59
97.66
97.51
61,91
43.03
S-P-P
97.80
98.00
97.61
70,04
49.18
S-P-
97.80
97.91
97.70
64.18
45.14
S-S-$
97.80
97.92
97.68
67.28
47.15
Thoth (S-S-P)
97.82
97.99
97.65
68.56
48.28
(h)
Combine
BLEU-3
BLEU-4
BLEU-AVG
ROUGE-1
ROUGE-2
L-P-P
31.31
20.74
58.81
35.40
L-P-S
30.95
20.30
39.39
58.86
35.31
L-S-P
31,32
20.54
39.86
59.24
35.82
L-S-S
30.36
19.79
38.77
58,49
34.62
S-P-P
35.18
23.44
44.46
59.52
36.48
S-P
32,20
21.34
40.71
59.33
36.05
S-S-$
33.73
22.49
42.66
60.08
36.56
Thoth (S-S-P)
34.58
23.09
43.62
60.30
37.14
(c)
Combine
ROUGE-L
Meteor
KW-FI
KW-P
KW-R
L-P-P
48.17
45.93
53.21
$5.78
$2.72
L-P-S
47.23
52.67
54.75
52.47
L-S-P
48.09
46.08
53.50
56.30
52.78
L-S-S
46.85
45.63
52.83
$4.64
$3.05
S-P-P
49.40
45.57
53.35
59.23
50.02
S-P-S
48.47
46.37
53.62
56.23
53.10
S-S-S
48.47
46.54
53,56
$7.36
$1.65
Thoth (S-S-P)
50.02
47.39
54.13
58.82
51.70
(d)
Combine
Order-LCS
Order-S
Order-Tau
Semantic-A
Step-M
L-P-P
71.30
19.50
65.50
45.02
32.33
L-P-S
69.70
16.17
64.00
45.65
34,50
L-S-P
71.82
20.83
66.67
45.18
35.33
L-SSS
71.52
18.00
65.00
45.47
33.83
S-P-P
75.21
25.17
71.50
44.78
50.00
S-P-S
72.57
21.00
66,50
45.23
34.50
S-S-S
74.61
23.67
69.17
45.41
50.00
Thoth (S-S-P)
75.34
25.50
73.33
46.60
53.00
than the precise numerical range, and also suggests that it can applied robustly to more complex open-ended tasks where reward scales may differ.
40

---

Table 26: Further analysis results Reward Magnitudes.
(
Reward_Scate
BERTSSorre- BERTScore- BERTScore-
BLEU--
BLEU--
Constant 2.5]
97.80
97.62
67.79
47.50
Sealing.up [0, 5]
97.78
97.92
97.64
67.74
47.27
Shift [-1.25, 1.25]
97.79
97.98
97.60
67.76
47.45
Thoth [0,
97.82
97.99
97.65
68.56
48.28
Swward_Scale
BLEU-3
BLEU-4
BLEU- AVG
ROUGE-1
ROUGE-2
Constant [0, 2.5]
33.60
22.00
42.72
60.06
36.38
Scaling up 5]
33.63
22.31
42.74
59.97
36.24
Shift [-1.25, 1.25)
33.57
22.82
42.90
60.11
36.69
Thoth [0, 1]
34.58
23.09
43.62
60.30
37.14
(c)
Reward_Scall
ROUGE-L
Meteor
KW-F1
KW-P
KW-R
Constant 2.5]
49.06
46.12
53,92
58.80
$1.30
Sealing [0, 5]
49.07
46.01
53.02
56.72
51.28
Shift [-1.25, 1.25]
49.15
46.75
53.70
58.67
51.11
Thoth [0,
50.02
47.39
54.13
58.82
51.70
(d)
Reward_Scale
Order- 1.CS
Order-S
Order-Tant Tau
Semantic-
Step-M
Constant [0, 2.5]
74.71
25.17
71.50
45.22
51.50
Scaling.up [0,
73.92
26.17
72.33
45.14
51.67
Shift [-1.25, 1.25]
73.70
26.50
72.49
52.12
Thoth [0, 1]
75.34
25.50
73.33
46.60
53.00
41

---

J.2 PRE-TRAINING ANALYSIS
Table 27: Performance differences on the MMLU dataset (Hendrycks et al. 2021 before and after pretraining. To examine potential catastrophic forgetting of general scientific knowledge, eval- uate MMLU on the OpenCompass platform (Contributors) 2023). "-pt" denotes the Thoth model after the pretraining stage only, and indicates the performance gap between pretrained models and their base counterparts. The results clearly show that pretraining on large collection of high-quality protocols not only preserves general capability but even yields improvements in certain disciplines.
Dataset
Versãon
Qwen3-4B
Thoth-mini.pp
A
Q Qwen3-8B
Thoth.pt
mmlu abstaact algehra
2db3733
87.00
lukaemon mmnu anatomy
72183b
70.37
71.85
1.48
75.56
75.56
0.00
lukaemon wwmmmmmaaiitt astronony
d3cc01
84.87
89.47
4.60
90.
88.82
-1.97
lukaemon mmlu business ethies
1de08
77.00
72.00
-5.00
81.00
77.00
44.00
lukaemon. cmmlu_cliiien knowledgee
cb3218
80. 38
81.89
1.51
83.77
86.04
2.27
lukaemon, mmlu, mmmllpoollege biology
caec7d
90.28
88, 89.89
-1.39
95.
94.44
0,70
Iukaemon_ wwtmmmoommmmmmmammmmmm mmllusolllge pheeissryy
520mm6
66.00
63.00
-3.00
71.00
71.00
0.00
lukaemon minlu college computer science
99:216
lukaemon mmlu college mathematics
678751
90.00
86.00
4,00
77.00
79.00
2.00
Jukaemon mmluu college medicine
38703l
83.24
84.97
1.73
84.97
84.97
0.00
luknemon.nne aakaemoo.mmll..oom www.mmclolllgg....ommmmoo physies
4f382c
94.12
94.12
0.00
95.
95.10
0.00
lukacmon waemonnmmilu..oomm camnlu.comuutr.scce
007550
-3.00
84.00
Iukaemon_ mmlu, conceptual physics
63588c
88 94
88.09
-0.85
94, 04.04
93.62
-0.42
lukaemon rumlu_economeitee
lukaemon, cattrmoommmuuuemmmm
d1134d
71.93
71.93
0.00
80.
84.21
3.51
engi mngineering
770ce3
80.00
lukaemon minul elementary muthematics
269926
97.35
97.35
0.00
97.62
97.88
0.26
lukacmon mmlu formal logic
cfeb0c
92.86
88.8 39,89
-3.97
89. 68
90.48
0.80
lukacmon mmlu globall
lukaemon ummlu www.m.glooa......mmmmm
ab07b6
51.00
$2.00
1.00
44.00
57.00
13.00
schloo_boology
37b125
85.81
87.74
1.93
90.00
93.87
3.87
Iukaemon_mmmt mumlumuh himlu_highh, school, chemistry
ac8820
89.16
89.66
0.50
91.13
89.16
-1.97
lukaemon mmlu high. school compote science
9965a5
91.00
92.00
1.00
98. 00
96.00
-2.00
luknemon, mmlu, high school, europenn history
ecf:90
81.82
lukaemon mmlu
school geography
0780c6
86.87
91.92
5.05
93.94
91.92
-2.02
lukaemon. wew.emmoo.mm......mmmmmmmmmmm mww..high..seee scgh.shhollh.ccooo.o.oooooooocoe govemment. and polities
3:5219
91.19
90.67
-0,52
95.34
95.34
0.00
lukaemon. mulu high school. macroconomics
a01685
87.95
86.92
-1.03
92.82
91.54
-1.28
lakaemon_mme mmm_mmmm__aa fighhhhhh
ed4de0
95.19
95.19
0.00
67.41
64.44
-2.97
lukacmon .mml high school microcconomics
04d21a
93.70
93,70
0.00
96. 26
95,80
-0.42
Iukaemnnn_ maninmmmmuahhhhe high school_ moossisssiic
93278F
86.75
86.75
0.00
90.07
86.75
-3.32
lukaemon
high school, scheoll psychology
7Jb114
lukaemon, munlu high school statistics
813Ba
89.81
89.35
0.46
91.67
90.74
0.93
lukaemon munlu high school us history
8932df
85.78
87.75
1.97
90.20
88.24
-1.96
lakaemno.. .mnlu. ookeemon.mmmuuu.
high..chooll sccool.l wirld_history
048e7e
85.65
82.70
-2.55
88.61
89.03
0.42
lukacmon minuu human aging
82a410
71.75
-2.24
76.
luknemon_ Junman, sexuality
42407c
83.97
81.66
-2.29
89.31
85,50
-3.81
lokacmon ummlu international law
cf3179
76.03
78.51
2.48
77.69
80.99
3.30
lukaemon, matiooommmm_____ee mmmlujjuripppp jurispradence
79.63
lukaemon sumlu logical fallacies
9cebb0
85.28
84.66
-0622
86.50
85.89
-0.61
lukaemon. mmlu. machine learning
028
76.79
76.79
0.00
78.57
82,14
3.57
Jukaemon miluu management
808764
84.47
85.44
0.97
88.35
86.41
-1.4
lukaemoo. mmmoo mamlo_marketinn marketinn
7394c3
91.45
-0.43
94.87
-3.42
lukacmon_nd mmmu www.mm/lllgmmeeeec.cc
881e15
88.00
88.00
0.00
91.00
89.00
-2.00
lukaemon_mat m_mmm_ miscellaneous
935647
88. 12
87.48
-0.64
91.70
91 /44
-0.26
lukuemon, zumnu moral disputes
72.83
lokuemon mmlu moral scenarios
f66lhe2
62.46
60.67
-1.79
60.
62.46
2.24
lukaemon manlu nutrition
4543bd
76.14
76.80
0.66
85. $5.29
84.31
-0.98
lukaemon.nd monuu, philosophy
08042b
72.03
68.81
-3.22
76.53
79.10
2.57
bbb197
-0.92
luknemon. .mmummpppppoooooo camnllproofssionaa accounntng
444b7f
73.40
73,40
0.00
81.21
80.85
-0.36
lukaemoo_mme jammuumuuuaaae propessioonallaa....pppoooop law
517e6c
49.80
50.07
0.27
55.61
56.39
0.78
hukeemon_mne wwtpiimmmlluuuaamaeee professionnll meeeiciiammmmmmiiieeeeee. mediicine
857144
84.19
1.47
87.50
lukaemon, mumlu,pppppp professsoonl, psychology
221a16
77.61
75.65
-1.96
79.90
79.41
-0.99
lukaemon mmnuu public relations
c7d39h
68.18
70.00
1.82
73.64
76.36
2.72
lukaemon minlu security studies
9b1743
71.84
75.51
3.67
76.33
77.96
1.63
lukacmon mmlu sociology
c266a2
80.60
0.50
87.06
83.08
lukaemon_mmike mmmu.um www.muuu....oeigg..oo... policy
528cte
86.00
86,00
0.00
87.00
85.00
-2.00
lukaemon ooml_yvirloog. vvioology
12c270
55 42
51.81
-3.61
54, 82
57.23
2.41
lukaemon__mmlu world, relgiines
AVG
81.41
81.21
-0.20
83.88
83.79
-0.09
42

---

K
CASE STUDIES
Retrieval (Level2)
Query:
During the nuclei washing step the snPATHO seq workflow, researcher accidentally used wash solution for
resuspension instead solution before filtering through 40 μm strainer. To correct this mistake ensure nuclei are properly prepared counting, what sequence of corrective actions should be taken to replace wash solution with wash solution 2 prior to proceeding?
Action Library:
"centrifuge", "titer', "remove", "pour", "resuspend", 'filter', 'dry'.
Model Output:
<think>
The problem here is that the final resuspension was done with wash solution f instead of wash solution 2. Wash solution 2 is designed to remove residual contaminants and provide the correct buffer composition for downstream processing. Using wash solution 1 could compromise nuclei integrity interfere with downstream assays. The logical correction immediately replace the incorrect solution with wash solution 2. Since the nuclei just resuspended, the best approach resuspend
centrifuge the sample pellet nuclei, remove
incorrect wash solution resuspension, filtering through a 40 jm strainer is necessary ensure single-cell suspension suitable for counting. This plan assumes that wash solution is available and that the centrifuge and filter are functioning properly. The sequence is centrifuge + remove incorrect solution -> resuspend in correct solution - filter, which directly addresses the mistake and restores the workflow. < //think>
correct wash solution After
<key>
Step 1: ('action": "centrifuge", "objects": ('nuclei suspension") "parameters": ("500xg", "5
min", '4 "c']}
Step 2: ('action": 'remove", "objects": ["supernatant"], "parameters': ['carefully"]] "I)
Step 3: ('action": 'resuspend', "objects': ("nuclei pellet"), 'parameters": ("1 ml", 'wash
solution 2'I)
Step 4: ('action": 'filter', "objects": ["nuclei suspension'] "parameters': ['40 um strainer")
</key>
Step Centrifuge the nuclei suspension 500xg for min 4 "C
Step Remove the supernatant carefully.
Step Resuspend nuclei pellet wash solution 2.
Step 4: Filter the nuclei suspension through a 40 um strainer.
</orc>
<note>
Handle all wash solutions ice maintain nuclei integrity. Centrifugation at 4 "C prevents enzymatic activity that could damage nuclei. When filtering, avoid applying excessive pressure prevent rupturing nuclei. Dispose of waste solutions according biosafety protocols for handling biological samples. </note>
Figure 11: Case result of Thoth on the Retrieval task.
43

---

Planning (Level2)
Query:
How should prepare a freshly cleaved mica surface and functionalize it with APS working solution in order to stably immobilize DNA for AFM liquid imaging of Twinkle helicase dynamies?
Action Library:
"dry", "recover", "deposit", 'incubate,, "wash", "peel".
Model Output:
<think>
The goal here is generate stable DNA-immobilized surface for AFM liquid imaging of Twinkle helicase. Mica is used support substrate because of its atomically flat surface, but must be freshly cleaved to expose clean lattice suitable for binding. The protocol specifies that should be peeled with Scotch tape obtain fresh, uniform surface. Next, the functionalization stop is required APS [amino-polle polyeehyyltyteene glycol silane] is used introduce amine groups, allowing subsequent DNA immobilization To ensure effective functionalization, the APS solution must deposited evenly on the mica, incubated for a defined time (5 min), and then washed with Milli-Q water to remove unbound APS. Finally, the mica is dried briefly to prepare a stable substrate. This sequence ensures that the mica has the correct surface chemistry, the DNA will bind stably, and the substrate will be suitable for AFM imaging in liquid. </think>
covalent attachment.
<key>
Step ('action": "peel", "objects': ('mica sheet"), "parameters": ('with scotch tape",
expose fresh layer"])
Step 2: ('action": "deposit", "objects": ('aps working solution'). 'parameters", (*200 pl", "on
mica')) }}}
Step 3: ("action": 'incubate", "objects': ("mica with aps'), "parameters": ("5 min'))
Step 4: ("action": 'wash", 'objects": ["mica"], "parameters': ["milli-q water', "twice"]]
Step 5: ("action": "dry". "objects": ['mica sheet"), "parameters": ["2 min']]
</key>
<orc>
Scotch tape expose a layer. Step Deposit 200 pl of APS working solution on the mica.
Step Peel the mica
Step Incubate the mica with APS for 5 min.
Step 4: Wash the mica with Milli-Q water twice.
Step 5: Dry the mica sheet for 2 min.
</orc>
<note>
APS silane- based chemical and should handled in fume hood gloves eye protection avoid inhalation or contact. Ensure that mica freshly cleaved before use to prevent uneven surface binding. Dispose of APS-containing waste according to institutional bazardous chemical guidelines. </note>
Figure 12: Case result of Thoth on the Planning task.
44

---

Troubleshooting (Level1)
Query:
During fecal microbiota transplantation, observed that the supernatant collected after homogenization centrifugation 805 g for 10 was very cloudy, suggesting incomplete removal of solid material. This made oral gavage difficult because the gavage needle clogged repeatedly. How can troubleshoot this issue to obtain clearer bacterial suspension without lasing viable microbes?
Action Library:
"filter", 'collect", "centrifuge".
Model Output:
<think>
The problem is that the supernatant remains cloudy after the initial centrifugation, indicating that fine particulates are not being removed efficiently. This suggests either insufficient centrifugation force inadequate filtering. The goal is to obtain clear suspension suitable for oral gavage without damaging the microbes.
First, most logical step is to repeat centrifugation a higher speed to pellet finer solids. the pellet still unclear, the next best option is filtration through a sterile filter. The protocol mentions filtering with 0.45 um and 0.2 pm filters, so using 0.2 μm filter would ensure removal of remaining bacteria and debris while preserving viability.
This plan assumes that the centrifuge can be safely operated at higher speeds (e.g.. 1600 x without damaging the samples and that sterile filters available. By first clarifying the suspension with additional centrifugation and then filtering, maximize clarity and ensure the gawage needle does clog. < //think>
<key>
Step 1: f'action': "centrifuge", "objects': ("supernatant"). "parameters": ("1600xg', "10 min')
Step 2: ("action': "collect", "objects:: ["supernttant"), "parameters". ["after centrifugation '])
Step 3: ("action': "filter", "objects": ["supernatant"]. "parameters": ["0.2 ym filter"))
</key>
Step Centrifuge the supernatant 1600xg for 10
Step Collect the supernatant after centrifugation.
Step Filter the supernatant through 0.2 pm filter.
</orc>
<note>
Handle centrifuged supernatant carefully to avoid disturbing the pellet. Filtration under
sterile conditions is essential to prevent contamination. Use filters rated for microbial removal, and avoid applying excessive pressure during filtration to prevent filter rupture aerosol formation.
</note>
Figure 13: Case result of Thoth on the Troubleshooting task.
45

---

Constraint (Level1)
Query:
During the generation spheroids, we usually add to the bottom of the 10 cm dish to maintain humidity. However, PBS stocks are unavailable. How can we modify the hanging drop setup to maintain spheroid viability without PBS while still preventing evaporation and contamination?
Action Library:
"add", "sterilize', "place".
Model Output:
<think>
The key challenge here is that PBS is essential in the hanging drop setup to prevent evaporation and maintain spheroid viability. Without PBS, spheroids at risk of drying out, leading experimental failure. Since PBS is unavailable, reasonable substitution sterile distilled water, which provide hydration and prevent the lack buffering capacity, we sterilize water and keep at room temperature. Placing dishes humid chamber help minimize evaporation. This plan assumes that sterile distilled water is available and that the spheroids are not highly sensitive to changes in buffer composition, The trade-off is that distilled water lacks ionic balance, which may slightly affect spheroid growth, but it is acceptable for short-term maintenance. </think>
must find alternative. A
evaporation. compensate
<key>
Step ("action': "sterilizz",, "objects': ["distilled water"), "parameters': (*0.45 um filter')
Step ('action':: 'add', "objects": ['sterilized distilled water'), "), "parameters': ('10
"bottom of dish']}
Step 3: f"action': "place", "objects": ['dish"], "parameters': ['humid chamber"]
</key>
<orc>
Step 1: Sterilize distilled water through 0.45 um filter. Step 2: Add 10 mi of sterilized distilled water to the bottom of the 10 dish.
Step Place the dish humid chamber. </orc>
<note>
When substituting PBS with sterilized distilled water, ensure the water sterilized to prevent contamination. Avoid handling spheroids directly to prevent damage. Keep the setup in a humid chamber to minimize evaporation, but monitor humidity levels carefully to prevent excess moisture that could compromise spheroid viability. </note>
Figure 14: Case result of Thoth on the Constraint task.
46

---

Scaling (Level2)
Query:
You need to prepare only 100 of 4% formalin solution for perfusion instead of the standard 500 ml. The original recipe requires 300 ml isotonic saline and 200 ml 1:10 formalin solution to make 500 ml total. How would you scale the recipe to prepare exactly 100 ml while maintaining the correct proportions and final concentration?
Action Library:
"dilute", "measure', "setup", "mix", "blot", "combine", "chill".
Model Output:
<think>
The goal is to scale down the formalin solution recipe from 500 ml to 100 ml while maintaining the same ratio of isotonic saline to formalin solution. The original recipe has 300 ml saline and 200 ml formalin solution, totaling 500 ml. The ratio therefore 300:200, 3:2. To scale this down to 100 ml, need divide both components by This results 60 ml saline and 40 formalin solution. formalin solution
then be diluted with saline to reach 100 ml total. This plan assumes that the 1:10 formalin solution already prepared as per recipe and that isotonic saline is available. By maintaining the ratio, the final concentration will remain 4% formalin.
< </think>
<key>
Step 1: ("action": 'measure", 'objects:: ["isatonic saline'], "parameters": ['60 ml'])
Step ("action": 'measure", 'objects': [*1:10 formalin solution'), 'parameters', ['40 ml'])
Step ('action':
"combine", "objects": ['isotonic saline", "1:10 formalin solution"). "parameters': ('final volume 100 ml'))
Step ('action": 'mix", "objects": ["solution"], "parameters's ['thoroughly']]
Step ('action": "chill", "objects': ('4% formalin solution"), "parameters": ['4 "c'])
</key>
<orc>
Step 1: Measure 60 ml isotonic saline.
Step 2: Measure 40 ml 1:10 formalin solution.
Step Combine the isotonic saline and 1:10 formalin solution to final volume of 100 ml.
Step the solution thoroughly.
Step Chill the formalin solution to
</orc>
<note>
Formalin taxic and strong irritant; handle it in fume hood and wear gloves, lab coat, and eye protection. Isotonic saline is safe, but care should be taken to avoid spills. Ensure the solution is chilled properly to prevent degradation and maintain concentration accuracy. </note>
Figure 15: Case result of Thoth on the Scaling task.
47

---

Safety (Level2)
Query:
During the SeqFISH tissue fixation step, how should the lab safely handle and dispose of 4% paraformaldehyde (PFA) solution when preparing and washing tissues, given that PFA is highly toxic and generates harmful fumes?
Action Library:
"load", 'collect", "wash", decontaminate", 'prepare", 'wear".
Model Output:
<think>
The goal here ensure safe handling of PFA, toxic fixative that releases formaldehyde vapors. The major risks are inhalation of fumes and accidental skin eye contact. To mitigate these, scientists
work in certified chemical fume hood, PPE, collect liquid waste designated formaidehyde-compatible containers hazardous waste disposal. The plan should include preparing the fixative under hood, washing tissues in fresh buffer after fixation, and decontaminating the workspace to prevent residual exposure. By structuring these steps, we ensure both operator safety and compliance with institutional chemical safety regulations. This plan assumes that fume hood, PPE, and proper waste containers are available. </think>
appropriate
<key>
Step ('action': "wear", "objects': ['lab coat", "nitrile gloves", "safety goggles").
"parameters': ('before handling pfa"]]
Step ("action":
'prepare". "objects": ['4% přa solution'), 'parameters": ['in fume hood")
Step ('action": 'wash", 'objects": "aatrtts""""tssseeeii ("tissues"], "parameters"; ['1x phs", "in fume hood"))
Step 4: ("action": 'collect', "objects": ["pfa waste"]. "parameters': ["in labeled hazardous
waste container'])
Step 5: ('action": 'decontaminate", 'objects': ('work area"), 'parameters": ('with appropriate
cleaning solution'])
</key>
Step 1: Wear lab coat, nitrile gloves, and safety goggles before handling PFA.
Step Prepare 4% PFA solution a fume hood.
Step Wash the tissues 1x PBS the fume hood.
Step 4: Collect PFA waste in a labeled hazardous waste container.
Step 5: Decontaminate the work area with an appropriate cleaning solution.
</orc>
<note>
PFA toxic and releases formaldehyde vapors that harmful when inhaled. Always handle in certified chemical fume hood while wearing gloves, goggles, a coat. Collect all waste in properly labeled hazardous waste containers institutional disposal. Decontaminate surfaces immediately after use to prevent residual exposure. </note>
Figure 16: Case result of Thoth on the Safety task.
48

---

infer, exception of category as defined in L PROMPTS
You are highlly precise scientific information extraction assistant. Your sole purpose to parse unstructured laboratory protocols from the provided text and reformat them into a structured dSON array.
Core Directives:
quess, external knowledge, hallucinate any details, expliicitly present prouided
1. Extraction, Creation: must ONLY
schema. primary
organize given
2. Comprehensive Extraction: every field, you
shorten, paraphrase, simplify the original text. details such concantrations, quantities, catalog numbers, manufacturer they MUST included.
relevant informatioo comprehensively.
3, JSON Sanitization and Escaping [Crucial for Parsing): The source text may contain formulas and non-printable control choracters. Befors outputting, you MUST parform two sanitization steps:
Remove Invalid Characters: Remove all non printable ASCII control characters, especially the MULL character
[a00000), from the extracted text.
Escape Backslashes. Ensure all backslashes IN] within the JSON strings are properly escaped as double backslashes
PM). This is critical for file paths and LaTeX-style formulas (e.g., a literal Valpha must become Malpha in the final JSON stringl.
Missing Information: specific category information (e.g.. "notes" or "equipments"] mentioned protocol, you
any explanstory text, markdown formatting, code fances like ""json t" your response. The output must with and end with J.
Strict JSOM Output: Your final output MUST valid JSON array ( ).
JSON Schema for each protocol object:
Each object the array must contain the following six keys:
eep. hap_namme' A concise name for the experiment, extracted directly from headings or titles.
abstract'; A paragraph or section desoribing the background, purpose, or application scenario of the experiment.
"materials': list or text block of all reagents, chenicals, solutions, and consumables mentioned.
"qquipments:: A list or text block of laboratory instruments, tools, and hardware required.
"procedures": detailed, step-by-step experimental instructions. Preserve the order as detail as possible
original
"notes': Any safety precautions, warnings, troubleshooting tips, important side mentioned protocol.
"category": The scientific. discipline of the experiment. To determine this, follow strict two-step process:
Step (Direct Extraction): First, search the text for explicit kegwords, subject headings, category labels le.g..
"Field: Molecular Biology'. 'Kewwords: Biochemistry") found, use them directly to determine the category.
2. Step (Constrained Classification). If and ONLY no explicit category is found in Step f, Infer the most
appropriate category based on the name and abstract. You MUST choose from the following list: Molecular Biology, Cell Biology, Biochemistry, Genetics, Immunclogy, Microbiology, Chemistry, Bioinformatics, Plant Science, General Laboratory Procedare. the protocol is too generic doesn't fit, default to General Laboratory Procedure.
Figure 17: System prompt 1 for structured integration with model based.
49

---

You highly precise and intelligent scientific information extraction Al. Your purpose long, complex laboratory protocols containing multiple sub protocols, tables, and image references, and consolidate them into single, structured ISON object representing the main experimental workflow.
Care Directives:
1. Consolidation as the Primary Goal: Your main task is to synthesize the multiple related sub-protocols within the text into a single, cchasiue dSON object. Identify the overarching experimental goel and use that as the basis for your summany. Your role to intelligently aggregate, not simply concatenate, the information.
2. Extraction, Not Creation while consolidating, you must still base your output ONLY on information explicitty prr present in the provided text. Do existing information,
external knouledge or hallucinate details. synthesis process create new information. The exception is the category field, which follows own defined logic. 3. Comprahensive Aggr engregation: The final consolidated output
organize
comprehensive. when creating the single dSON object, ensure that all critical materials, equipment, and procedural steps from the various sub-protocals are included. Do not lose key experimental details during the consolidation process.
4. Handling Rich Content (Tables and Image Captions]:
TTables: You must parse information from markdown tables. For example, a table listing reagents and their concentrations must be fully extracted and integrated into the materials field.
Image Captions: You carnot see images, but you MUST read and interpret the text in image captions (e.g. FFiuure 1:
Results of the kinase assay.."). Use this contextual information to enrich the abstract or procedures fields where it
helps clarify the purpose or outcome of step.
5. Fallback for Poor Consolidatione If the sub-protocols are too distinct or unrelated to logicallyy combined into singla coharent workflow le.g., protocol DNA extraction followed by completely separate protocol for protain crystallization), NOT force summary. In case, identify
ONLY the most central first comprahensive protocol from the document single JSDN object.
6. Handle Missing Information: protocol, you MUST return 'None".
specific category information (e.g. "notes' "equipments"] 'I mentioned
7. Strict JSON Output: Your final output MUST be valid JSON array {... ). Do not include any explanstory text, markdown formatting, or code fences like ""json and "" in your response.
8. Strict Data Type: String ONLY: This is a critical rule. The value for every key in the output JSON object MUST be single string. Even if the source content is a list of items (like materials or procedural stops). you must format it as a single, multi-line string. Do NOT use JSON arrays (eg, [ ['item'', "item2"] II for any field's value.
9. JSON Sanitization and Escaping (Crucial for Parsing): The source text may contain formulas and non non- printable control char acters. Before outputting, you MUST perform two sanitization steps:
Ramove Invalid Characters: Ramove non-printable ASCII control characters, espacially the character
(N00000), from extracted
Escape Backslashes: Ensure all backslashes (N] within JSON strings are properly escaped double backslashes [NN) This oritical file paths and LaTex-style formulas (e.g. literal Valpha must become Valppes in the firal JSON string).
JSON Schema for the protocol object:
Each object in the array must contain the following seven keys:
"exp namemm The name of the main, over weraarhhing experiment, synthesized from the titles of the main protocol and its sub-protonols.
"absttrct": A consolidated paragraph describing the overall background, purpose, and application of the entire
exper imental workflow.
"materials": An aggregated and de de-duplicated list of all reagents, chemicals, solutions, kits, and consumables mentioned across all sub-protoccas and
"equipments': aggregated de dupduplicated entire workflow.
laboratory instruments, tcols, hardware required for
"pracaduras': The synthesized, step-by-step exparimental instructions represanting the complete, logicall workflow from finish. You should structure this clearly, for example. using headings like 'Part 1: Sample Preparation', "Part 2: Main Assay' etc., to indicatte which part of the procedure comes from which sub-protocol.
"notes': A consolidated list of all important safety pracautions, warnings, and troubleshooting tips from across the entire document.
"category": The scientific discipline of the overall expariment. To determine this, follow a strict two-step process:
1. Step (Direct Extraction): First, search the text for explicit keywords, subject headings, or category labels le.g..
Field: Molecular Biology', 'Keywords: Biochemistry") }) If found, use them directly to determine the category.
Step [Constrained Classification): If and ONLY no explicit category is found in Step 1, infer the most
appropriate category based on exp, name and abstract. You MUST choose from following list: Molecular Biology,
Call Biology, Biochenistry, Genetics, Immunclogy, Microbiology, Chemistry, Bioinformatics Plant Science, General Laberatory y Procedure. the protocol too generic doesn't fit, default to General Laboratory Procedure.
Figure 18: System prompt 2 for structured integration with based.
50

---

You scientific protscol refinement assistant. Your sole purpose receive pre-extracted, but potentially messy and disordered, fields of laboratory protocol and clean, rearder, and format them clean, logical, and structured ISON object.
Core Directives:
1. Preserue Content Integrity. This is your most important rule. You MUST NOT add new scientific information, delate existing steps, or change critical details like chemical names, concentrations, quantities, or durations. Your role is to reformat and rearder, NOT to rewrite or summarize the care content.
2. Logical Reordering Procedures: The procedures
may contain steps that are chronological order to logical sequance. explicit numbaring (e.g- workflow.
parsing arrors.
analyza
the steps to 'Noxt..., incubation...)
clues (e.g.
original numbering
but formatting broken, fix the formatting.
3. Format Markdown and Artifacts:
Tables: encounter Markdown tables within field, convert the information into readable, clean text formast
[eg., bulleted list).
You MUST completely remove these tags, If caption exists (e.g. "Figure 1: Western blot analysis.") "), you must retain this caption text and integrate it smoothly into the surrounding sentences to preserve the logical flow. The goal is to keep the descriptive information from the caption while discarding the non-textual image link.
Image References:
You may encounter Markdown image tags like
Parsing Artifacts: Remove any artifacts from PDF-to-MD conversion process. such as stray page numbers, repeated headers/Fuoters, or aukward line breaks that interruptt sentencees, while ensuring the scientific meaning is presarved.
JSON Output: final output
single, valid object. with ).
wrap
array use
markdown code fences like "json. The output
with
5. Strict Data Type: String ONLY: This critical rule. The value for every key the output JSON object MUST be single string. Even the source content is list of items (like materials procedural steps), you must format it single, multi-line string. Do NOT use dSON arreys [eg,, ['item"', , "item2"] for any field's value.
6. JSON Sanitization and Escaping (Crucial for Parsing): The source text may contain formulas and non non-printable control characters. Before outputting, you MUST perform two sanitization steps:
Remove Invalid Char actrrcters: Remove all non printable ASCII control characters, especially the NULL character
[u00000), from the extracted text.
Escape Backslashes: Ensure all backslashes (V) within the JSON strings are properly escaped as double backslashes (V). This is oritical file paths and LaTeX-style formuaas (e.g.. literal Yalphi must become Malphh in the final string).
JSON Schema for output
The output JSON object must contain these exact with their content claoned and reordered according
directives above:
"abstract"
'noterials'
"equipments"
"prosadrres"
'notes'
Figure 19: System prompt 3 for structured integration with model based.
51

---

solier, You expert scientific and problem solver. training world-class scientific research assistant Al. Your prinary task is to devise and articulate logical solutions to experimental challenges. You will express these solutions in high-quality, structured format basad on the context of provided scientific protocal.
You will be given a Scientific. Experiment Protocol in a structured 3SOM format. Your generated response must strictly adhere to the fallowing XML structure:
equestions
[A clear, relevant question you formulate that poses scientific challiage or query.
</ /question>
<thinks
[This the of your response. Articulate the scientific. strategy solue problem question. Start analyzing the goal, then formulate hypothesis clear plan of attack. Justify the proposed sequence of actions is the most logical and effective approach. This must reflect the oritical thinking of scientist.] </think>
<key>
[The structured, machine-readable plan derived from the reasoning in the <thinks tag. Each step represents single, atomic action.]
Step 1: ('action': 'detach', "okjects": : ['bndes"], "parameters": ['1x pbs-5 mm edta", '10 min", "37 "c"]
Step 2: ('action': 'wash", "objects": "' "parameters": [*fx pbs", "twice"])
Step 3: l'action::
"centrifuge", "objects": ["cells")] "parameters": ("367xg", min'l]
</key>
[A concise, natural-language summary of the plan <keyp. Each step must be imperative sentence.]
Step 1: Detach bmdos with fx pbs-5 edta for 10 at 37 "c.
Step 2. wash the cells with fx phs tuice.
Step 3: Centrifuge the cells at 367xg for 10 min.
</orc>
enotes
(A concise paragraph summarizing the most critical safety information relevant to the steps in the plan.)
</note>
Descriptions
<thinks (CRITICAL:: important part your output. explicit, step by stop scientific reasoning justifies plan presented ekeys. It bridge gap betueen problem the equestions and the solution dkeyp [Natural Longuage Summaryl: This section transiates each step from the <keys tag into simple, human-readable instruction. Each step must be concise, imperative (command) sentence that strictly reflects the action, objects, and parameters of the corrasponding step ckeys, adding no new information. The number of steps must match ckays exactly.
2. skeup Formatting Rules (CRITICAL)
1) One Action Per Step. Each Step must correspond to a single, atomic action and contain exactly one JSON dictionary. If a lagical operation requires multiple actions le.g. 'wass and then centrifuge'), ) you must break it down into sequential, separate steps.
'Stop
by adding
placing
Breakdown:
Step 1: ("action': 'add"; "objects" ('oold phs"), "parameters": I'to reaction") }]
Step 2: ("action': "place', "objects"; ["tubes"]. "parameters", ['on ice'))
2) Lowerease Content: All string values within the JSON object (action, elements in objects, and elements in parameters)
must be in lowerease, Numbers and standard units (e.g.. "C, mM] exempt.
3) Structure of Each Action:
"action": A string containing single, cancise verb in lowercase (eg.. "add", "incubate", "contrifuge", "wash").
"okjects"; A list of strings. These are the direct objects of the action, in lowercase, Use specific, meaningful nouns (a.g.. "bmdos", "call pellet", "Ix phs").
"parameters": : A list of strings that modify the action, lowercase. This includes quantities, durations, temperatures, speeds, key conditions.
4) Parameter-Object Relationship. Parameters "parameters" must papllyjointly to the action parformed on
objects 'objects' list.
Conciseness and Semantic Distillation: Your primary goal distill the scientific original text verbatim. Simplify verbose language into cancise keywords short phrases. Bad [Verbose]: "parameters.: ('in water bath set 37 "c")
transcribe
Good [Concise & Distilled); "paraneters': ['37 "e", "in water bath?]
Figure 20: System prompt used for constructing SciRecipe.
52

---

'Generate a data sample that demands precise extraction of factual, rabdy parameters from authoritative sources, emphasizing exact values, units, tolerances, environmental conditions, and contextual qualifiers (eg. temparature, pH. grade, catalog numbers] that can dirsctly transcribed protocal fialds, require cross- checking consistency across related parameters (stock working concentrations, volumes instrument ranges), explicit unit normalization, correct significant figures, and clear source attribution provensnce notes; the scenario should penalize guessuork and reward verifiability, internal coherence, and readiness for immediate without human reinterpretattion.
'Retrieval':
"Planning': 'Generate questions that demand the model transiate a high research objective into granular. step. by-step, executable experimental workflou. These prompts should challenge the model to logically sequence actions, identify necessary inputs and outputs for each stage, specify requiret ed equipment and resgents, and integrate quality control checkpoints. The alm is to assess the model's sbility to understand dependencies, allocate rescurces, and construct caherent experimental plan that is both feasible and reproducible. Fecus on scenarios where multiple steps are intercormeetedd requiring a holistic understanding of the scientific process, from sample pr preparation to final analysis. The generated data should reflect the thought process of designing robust, and-to-and expariment."
questions that simulate real-world experimental failurrs unexpected outcomes, requiring the model systematically diagnose problems and propose actionable, testable solutions. These prompts should present specific symptom devistion from espected results. The model should then identify potential causes, prioritize them, suggest spacific, singla-variable correctius actions, and outline how to verify the fix. The generated data should encourage iterative problem-solving, mimicking the annsrue-hgpothesiee-tte for resilience and efficiency in research. Think about common issues in various lab techniques (e.g.. PCR, Western blot. cell culture contamination) where methodical approach to error resolution is par papsmtt
"Trobblshootinng: "Your task
aycle crucial
"Constraint": "Generate questions that require the model to adapt experimental plans under non-ideal, constrained
conditions. These prongtts should present scenario where a critical resource (egg. specific reagent, equipment, budget, time, computational power) unavailable or limited. model needs to identify acceptable alternative methods or materials that can still achicve core experimental objective, while clearly articulating the trade offs involved (eg.. efficiency, sensitivity, comparabiltyy of results) The goal capable of navigating 'Constraint Satisfaction Problems in lab setting and preventing dead ends when ideal conditions not met. Consider real-worid limitations that often arise in resource-limited environments."
train the model be flexible and pragmatic,
"Sealing": "Your task is to create questions that test the model's essential practical skills in unit conversion, stoichiometric calculations, and scaling experimental protocols. These prompts should involve adjusting reaction vclumes, reagert concentrations, or sample sizes while maintaining critical ratios and ensuring technical fuasibility (e.g.. considering minimum pipetting volumes, solubility limits, or raquired excess for practical handling). The aim is to train the model to accurately transiate theoretical recipes into precise, acticnable laboratory instructions, preventing errors arising from incorrect calculations or impractical volumes, Foous on scenarios where a protocol needs to be scaled up or down, or whare units
be consistently conuarted for different componants."
'Generate questions that prompt the model proactively identify and address potential hezards associated with specific chemicals, biological naterrills, or procedures. The model should automatically append recessary safety precautions, personal protective equipment [PPE] requirements, if not explicitly asked. The gool is train the model to inherently integrate critical safety and regulatory information into advice it provides, minimicing risks in a laboratory setting. Think about scenarios involving hazardous chemicals, bichazordous materials, specialized equipment, or waste generation, where adharence to safety protocols and regulatory standards je.g. OSHA, institutional biosafety guidelines] is non-nege non-negatiable."
"Safety":
dispasal guidelinas, and compliance reminders,
"Overview": "Generate a question asking for high-level summary of an experimental workflow. The goal is to provide a concise sird'seyye view of the major stages while prr erserviing the logical sequerce. The generated <key> should be concise, containing approxinately
of the hicrarchical protocel to summarize: the top-level sections are few enough to length, summarize their second.
staps. To achieue this, intelligently selact the appropriate
there many top-level sections, choose single, major first-level section and summarize the level sub-sections instead."
"Specific": Genneaae question that asks for granular, step by step breakdown of specific, continuous segment of the experiment. The question must target exactly one major first-level section from the hierarchical protocol. Crusially, you must select a section of appropriate length such that its detailed procedural breakdoun results in concise skey> of spproximately 2 to 6 stomic steps. Avoid selecting sections that are either too simple [f stepl or overly complex (7+ steps). The <keys ansuer must be derived from the louest-level procedural text within that single section, not its titles, adhering to all distillation and formatting rules.'
Figure 21: Prompt used for defining each task.
53

---

Protocel]
Experiment Nome: (oxp_nane name)
Background: (shstract)
Matorials lals and Reagents: (natorials)
Equipments: (equipment)
Procedures: (procedure)
PPeecs/nettti (nates)
You are now tasked with generattnng (num_qu) highoquiity, single-turn QA pwir(s) for the (type category
Task Details for Category: (typenname)
Your taak guided by following definition: (typae instruction)
Gore Instructions Examples
f. Frinary Goal: Create Rich, Fecused, and Self-Contairent Soevariox
Your most importont goal to ensure every QA pair closed logical loop. The <question> not just a query, it is complitte prohlem
be constracted with enough detsil thus the skeys he logicallty deduced from
contents alone.
TTh Principle of the Complete Prohlam Statement: The equaations must uronbiiuuuall provide wil the necessoer y "given" information for the problen. Before finalizing, information presented within the question itself, without requiring external knowlddge beyond general al scientific principles. achieue this, your question include:
confirm that the dkeyp logical consequence of the
The Specific Goal or Symptom: Clearly state the objective jag. "scale down the reaction I the problem (eg.. 'chserved very lou call sishility').
Relevant Guantitative Data: Embed all neceesary rumbers, concentrations, and volumes (e.g. "scale from 3x10"6
0.5x10% calls," "the woch buffer contains 5 mM EDTA')
Key Reagents and Equipment: Mame the specific materials involued problem le.g. GM-CSF from Peprotech is urwosilablee" "using a FACSCwibur flow oytometerr)..
Experimental Contect: Sinpoint where the overall process the problen ocpars ln... Tinmediately after first centrifugation step," ddriig 'chasse' phuse').
The Principle of Fecused Scape and Connisenase: The sceeario you create must be solumble with a concise plan.. The non-nggttisbe requiremnnt. Actively design a The Principle of Varied Chrasing: Use diverse phrasing (procedural inquiry, goal-oriented request, direct command, scanaric-hasodd) swoid generating repetition quaatione.
resulting <kag> mast contain betusenn and stops, inclusive.
problen that is targeted and moragesble to fit this length.
2. Criticwl Gereration Rulex
You must strictly adhere to the following rules for every QA you generate:
Rule 1: Contect-Grounded Generation (Most Inportant Rule). The provided protocol is your contect and foundation, not problen posed in your equeations.
be oxpied. Your generated ckep This new plan must be scientificaly plausible and consistent with the techniques, resgents, and equipment nentioned in protocal contectt. For excemple, as trouhlechooting plan will not Rule 2; Strict Output Structure, Each pair must strictly follow the required structure: equestions, <thinks, <keys. cores, and <notes as five sepprrate, top-louel tago, All toxt valuse inaide the ckaye JSON mat be lovercsse. Each Step laturai-lanquage summary
nevel, logiosl solation
original text; you must invent it.
must contain only single action. The tag written concise imperation sentence.
step ckep tag.
Rule 3: Justify Your Mouel Plan ethinks. ethinks tag is where you must explain the scientific reasoning behind the skeys. Start analycing the scanario your equestions, proposs hypotheais stradagy. and justify why the sequenee of aection in your dkopp the moot logical way to solue the proisem.
plan you
Do (Scientific Reasoning:: Start by analyaing the scenario in your equestions. Then, propose hypothesis (e.g..
The Bieely cause is X Firelly justify why the sequencs of actions your ckege is the most logicsl and efficient aay
test that hypothesis or solue the probiam.
Do NOT Do This Matts-oomentaryj Do describe your generation process refer
protocol
directly le.g "I will invent troubleshooting step..). Your respense must be a standalone scientific explonation. Rule 4: Redicaal Distillation for ALL Ganerated Steps (Most Important Rulel, The goul of the chege tag is schieve maximam semoentic distillation. This rule applies to all steps you generate, euen if they are novel. You must aggressively simplify and decompase your planned actions into stonic keywords. A par ameter must be a concise modifier leg. '10 fragment [e.g. 'antil solution clear')
nin; presarmedd iow), descriptiun phease
Example of Radical Distillation:
Original Text: 'recusperd the cells in 100ul total volume preveemee conditioned complete medium"
Badd (Literal Transcription):: ('action': "resuspand", "objects': ("the calls"), 'paraneters': ('In 100u total volume preuerme conditioned complete mediun"])
Okay (Simple Simplifictiion; ('action": 'resuspend", "objects', ['colis"]. "parameters": ('100 vr, "preuaredd
conditioned complete nedian"))
Excellent (Radical Distilaation:: ('oction': "resuspend"; "objects": ['cells"], parametors": [1100 pl", 'conditioned
complicta mediun', "prewarnedd
Rule S: Question-Ansuer Aligmnent. The novdl plan you create in ckeup must directly and completely address the specific scenario you created <questions.
Ruia 6: Acknouwdgge Assumptions. When cr creating new plan, you will inevitably make assumptions. You must state these assumptions within the <think- tag, franing them as part of your export planning (e.g.. "This plan assures the avuilability standard spectrophotoneettr
check reagent concentration...
Rule 7: Keep the enetes Brief and Focused. The <note- tog must be single, concise paragraph. It should highlight only most oritical and spacific sefety hazards directly related to the plan doays. Avaid genaric obvious advice.
Figure 22: Input prompt used for constructing SeiRecipe.
54

---

Your previous output contained formatting errors and did not meet the required sehema. Please correct the output to conform strictly to the structure and rules outlined below.
<question>
[Your generated question text hara.]
</ //question>
thought process solving experimental problem.]
</think>
<key>
Step 1: ('actioon": 'verbf', "objects": ['objectf"] "parameters": ("param 1
Step 2: l'action': 'veb2"; "oljects": ['object2] ], 'parameters": ("param2
</kay>
<orc>
Step 1: Verbi the object1 with param1.
Step 2: Verb2 the object2 with param2.
</ore>
[Relevant safety and handling information.]
</notes
Mandstory Rules Follow:
1. Correct Mesting: The root must contain exactly five in this specific order: equestions, <thinks, <kop, and
<noteo. These are all top-leuel tags.
2. <key> Stop Format: The content inside the <key> must be sequence of stops. Each stop must begin with the
exact string Step X: (where X is number starting from 11 followed by newline.
3. JSON String Value ffor <keys)): The value for each Step X: In the <keys tag must be single, valid JSSN string.
This JSON string must parse into single Dictionary.
This dictionary must contain exactly three keys: "action", "objects", "parameters".
action, insida objects/parametant lists) Correct Example: Step ('action':
string
lowercase.
"objocts': ['pbs']. ]. "parameters": mf". "cold"]
Incorrect Example (not valid dSON string):
('action:: 'wdd', objects': [pbs']] [Uses single quotes)
Incorrect Example (is
dictionary): Step [[action':: "add", "objects": ['pbs'), "parameters": 1110 ml".
'cold"]]]
6. wores Summary Format: The corc> must be natural language summary. The number Step X: entries must exactly match the number of steps in ckeyp. Each step must he single, conciss imperative sentence that ancurately reflects the correspanding step in skey..
5. No Extra Commentary: Return only the corrected XML block starting with equestions and ending with </notes, Do not
include any apologies, explanations, or any text outside of these tags.
Figure 23: System prompt for the repair module.
55

---

You expert scientific protocel vaiidator and safety compliance officer Al. Your task to rigorously review given Question-Answe Answer pair, which generated from scientific experiment protscol, against set of strict criteria. You will focus solely the scientific content, safety implications, and practical utility.
Your primary responsibilities are:
1) Scientitic Accuracy: Verify that all scientific facts, parameters, prosedures, and conditions stated in the answer are consistent with typical laboratory practices and scientifically sound. Crucially, ensure they directly derive from or are logically inferable from the original protocol context provided in the question. Identify any hallucinations, inacouracies, or inconsistancies (a.g. incorrect calculations, mismatched concantrations, inappropriate reagent use).
section completeness, accuracy, and safety hazards (chemical, biological, physical), Personal Protective Equipment, engineering controls (eg,, fume hoods), disposal, and general lab safety best practices. Ensure that the safety advice is spacific, actionable, and aligns with standard regulatory guidelines (e.g OSHA, institutional biosafetyl. Flag any missing incorrect safety information. 3) Logical Caherence and Actionability: Evaluate the ethinks, splans, stools, and <onswer> sections for logical flow, feasibility, and clarity, Ensure the plan logically addresses the question, the tools appropriate and correctly parameterized for the plan steps, and the final answer is concise, accurate summary.
2) Safety Compliance: Critically assess
regarding
4) Clarity and Ambiguity Check: Identify question or answer that could lead to misinterpretation or error during experimental execution. Ensure all critical parameters (e.g., concentrations, temperatures, durations, volumes) are precisely stated with correct units and no missing information.
vague terms, ambiguous instructions, or unclear descriptions within the
5) Generality Specificity Check: Assess
recommended steps, reagents, or equipment appropriately context. Flag recommendation unnacessarily restrictive, overly vague when precision needed, alternatiee optimization option
general specific
mentioned broader applicabllity (e.g.. primer Tm
optimization rangel.
tool usage efficient and aptimizes resource utilization (time, reagents, equipment), Flag any unnecessarily complex, redundant, resource- intensive steps recommendations where simpler cost-effective alternatives could achieve the same scientific objective without compromising quality.
6) Efficiency and Resource Optimization Check: Evaluate the proposed experimental
You will be given question that includes the necessary protocol contextt, and an answer previously generated by another Al.
Your output structured as follows:
cvalidaation reports
beccuraoycchaks
[Detail any inaccuracies, inconsistencies, hallucinations none, "No significant scientific Inaccuracies found.")
</accuracy_dee pcccuracy_chhcco>>oooo
esafetyy_ooe wwwpliinccceeeeeeoe check>
[Detail missing, incorract, or insufficient safety information, OPE, waste disposal, or compliance issues. If none, state "Safety and compliance information is adequate and accurate."]
</ //saeety_comee compliance.eccccccccc checks
ological, wwwerencc_chheee
[Detail any issues with the logical flow, feasibility of the plan, appropriateness of tools, or clarity of the overall answer. none, state "Logical coherence and actionability are sound."]
</logical ccherence, -ccceke <clarity amblguittb checks
[Detail any issues with vagueness, ambiguity, missing precise parametors/units. none, state information is unanbiguous."]
sedarity_ambiiuee check.
<generality, specificity, check>
[Detail issues where recommendations Are insppropriately general spacific, or uhere useful
alternatives/optin /ppiiizzaiionss were overlcoked. If none, state 'Appropriate balance of generality and specificity.') </ s/generality, specificity, checks
sefficiency, respurce, optimization, checks
[Detail any inafficiencies, redundancies, or sub-optimal resource usage. If none, state "Plan demonstrates good efficiency and resource optinization."]
</ e/efficienoy, res resource wptprroattpiiiiiiiiin.oo check> </validation_ reports
section has issues, explicitly positive affirmation Je.g., "No found", "Adequate and accurate") specified for each section. Do not leave sections empty just say 'None".
Figure 24: System prompt for the scientific review module.
56

---

Please validate the following Question-Answer pair basedd on the protocol context provided in the question:
<questions
(question (ecsstt
</ s/question=
canswer to validates
{ansuer (instrr_extt)
</answer to validates
VALIDATION GUIDELINE: Apply Principle of Generous Interpretation
Your prinary goal to validste for scientific plausibility and safety, not for perfect, literal adherence the protocol. Your defoult stance should be to approus the ansuer unless find severe and unambiguous error. A severe error one that, if uncorrected, would almost cortainly lead to:
a) Guaranteed or highly pr probable experiments failure (e.g. concentration that is off by an order of magnitude, a fundamentally wrong sequence of critical steps). step represents a common or scientifically valid alternative practice, it should not be flagged as an error, even differs from the source text.
b] clear safety hazard (e.g, cmitting crucial warning for a highly taxic reactive chemical). el Fundamentally flawed logic that would invalidste entire exper imperimnntal approach. you
minor, debatabla, or represent a slightly and correspending mandatory affirmation Only report issues should threshold defined above.
still scientific approach, you
CRITICAL OUTPUT RULES:
1. Your output must he complete cualidation reports containing all six check sections.
2. If you find issues in section, provide concise explanation of the problems.
response for that section must consist only of the exact mandatory affirmation text listed below. There should be no additional words, explanations, or leading/trailing text.
3. If section has no issues,
Correct Example:
<accuracy,
vacourac__check-Noo check>No significant scientific inaccuracies hand.//accuraay.che checks
Incorrect Examples:
found. wttps///aayyyaheeeee accuracy_chee chackhhcc.. 'No significant scientific inacouracies faund.'
<accuracy Aftar review, significant
chacks
<accuracy check>No significant scientific
saccua//ccuurayy
Mandatory Affirmations
Material Errors Found':
safety, compliance check: 'Safety and compliance information is adequate and accurate."
loginal coherance, chack: 'Logioal coherince and actionability are sound."
nogheal_
dlarity ambiquity check: "All Information is clear and unanbiguous."
generality_speeieiie spscificityyyccccco check: 'Appropriate balance of ganerolity and specificity,"
efficiency, resource ce, optimization check: "Plan demonstrates good efficiency and resource optimization."
Failure to adhere strictly to these rules for all six sections will result in an invalid output. Begin your validation now.
Figure 25: Input prompt for the scientific review module.
57


Reviews:
[{"review_id": "GKyvVGMkHF", "reviewer": "Reviewer_WL6o", "summary": "This paper introduces a large scale dataset of protocols across many domains in biology. Authors also propose a “Sketch-and-Fill” reasoning paradigm, a structured approach to protocol generation to organize the output into reasoning, structuring and expression steps. This structured output allows them to define a multi gated reward function for formatting, consistency, order of steps etc.  Using these components, they present a trained model called Thoth which achieves improvement in performance compared to base LLMs.", "questions": "- Authors should compare the sketch and fill paradigm and the multi stage generation to performance of frameworks like chain of thought prompting to clarify the added value of sketch-and-fill for generating structured output. \n- Evaluating models ability to do multi step mathematical reasoning is also another idea for benchmarking LLM models for protocol generation,  for tasks like adjusting reagent concentrations or dilution rations which could influence quantitative changes in the downstream steps.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Authors presented comprehensive benchmarks across a wide range of LLMs. Authors also considered real-world context in experimental protocol creation in the way the framework and rewards are designed. The paper is also written well and the authors included a comprehensive appendix.\n- The trained Thoth model (including the mini) demonstrates strong performance across all evaluation metrics in Table 1, highlighting both the model’s abilities and the value of the dataset introduced by the authors.", "weaknesses": "See questions below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XHObecNrML", "reviewer": "Reviewer_btq2", "summary": "&nbsp;\n\nThe authors introduce Thoth, a model trained for biological protocol generation. To train Thoth, the authors introduce a dataset of 12K protocols termed SciRecipe. The main technical innovation of the paper is the SCORE mechanism that enables both RL training and evaluation of protocol generation. The authors rigorously evaluate the components of Thoth through multiple ablation studies and sensitivity analyses. Overall, the work is clearly impactful, the empirical analysis is extensive, and the paper's writing and presentation are excellent. My sole concern lies in assessing the reproducibility of the authors' results given that the codebase is not provided (though the authors have described their approach in meticulous detail). If the codebase is released during the review process to address this concern, I will upgrade my score and champion the paper to be considered as a spotlight/oral contribution.\n\n&nbsp;", "questions": "&nbsp;\n\n1. In Figure 2, Rollout 3, there is a tickmark in the order box despite the rollout having one fewer action relative to the ground truth? Does this still incur a positive reward (as indicated by the checkmark)? **Update**: This would appear to correspond to the strict subsequence order consistency. I would have thought that the addition of an extra step in the protocol would be damaging to the experiment?\n\n2. In Equation 2, there is an expectation over $y$, the protocol generated from the model $\\pi_{\\theta}$. Why is there a distribution over $y$. I would have assumed that a context $x$ denotes a specific experimental task and so this would imply that the distribution is over repeatedly sampled protocols for the same specific experimental task $x$?\n\n3. In Table 3, each row is the SCORE mechanism evaluated without a single component only?\n\n4. In Section D of the appendix, Equations 13 and 14, the authors do not appear to give a KL penalty for the GRPO definition. Was the KL penalty omitted for the authors' experiments?\n\n&nbsp;", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "&nbsp;\n\nThe work undertaken in this paper is highly impactful. The authors have contributed concrete artifacts such as a dataset of 12K structured scientific protocols, as well as technical innovations such as the SCORE mechanism which significantly advances RL training for protocol generation. Furthermore, the empirical results of the authors are extensive and highly informative for researchers in the broader field.\n\n&nbsp;", "weaknesses": "&nbsp;\n\nI describe major and minor points below that the authors may wish to consider.\n\n&nbsp;\n\n**__MAJOR POINTS__**\n\n&nbsp;\n\n1. For the SciRecipe dataset introduced by the authors, it would be useful to include a datasheet for datasets [2] to fully describe the curation process and detail the authors' plans for maintenance of the dataset.\n\n2. As mentioned in the summary, if the authors can release their codebase during the review process, I will be inclined to increase my score.\n\n&nbsp;\n\n**__MINOR POINTS__**\n\n&nbsp;\n\n1. There are missing capitalizations in the references e.g. \"PubMedQA\", \"Kendall\", \"GPT\", \"LLMs\", \"AI\" etc.\n\n2. There are missing conference references e.g. Ahmadian et al. 2024 was published at ACL.\n\n3. In the section on related work, it would be worth the authors discussing the relationship between their work and BioPlanner [1].\n\n4. There is a missing comma at the end of Equations 1, 2, 3, and 5.\n\n5. Reference [3] should be cited as the originating paper for GRPO in place of the DeepSeek-r1 paper in the main body of the text. The correct reference is given by the authors in Section D of the appendix.\n\n6. Line 1093, typo in $a^hat_{i}$.\n\n7. When mentioning PPO in Appendix D, the authors should cite [4].\n\n&nbsp;\n\n**__REFERENCES__**\n\n&nbsp;\n\n[1] O’Donoghue, O., Shtedritski, A., Ginger, J., Abboud, R., Ghareeb, A. and Rodriques, S., 2023, December. [BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology](https://aclanthology.org/2023.emnlp-main.162/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 2676-2694).\n\n[2] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wallach, H., Iii, H.D. and Crawford, K., 2021. [Datasheets for datasets](https://dl.acm.org/doi/fullHtml/10.1145/3458723). Communications of the ACM, 64(12), pp.86-92.\n\n[3] Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y.K., Wu, Y. and Guo, D., 2024. [DeepSeekMath: Pushing the limits of mathematical reasoning in open language models](https://arxiv.org/abs/2402.03300). arXiv preprint arXiv:2402.03300.\n\n[4] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. [Proximal policy optimization algorithms](https://arxiv.org/abs/1707.06347). arXiv preprint arXiv:1707.06347, 2017.\n\n&nbsp;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MtFFI5BAm9", "reviewer": "Reviewer_RwGZ", "summary": "This paper investigates the capability of modern large language models (LLMs) to perform scientific protocol design, focusing on the generation of correct and sense-making wet-lab protocols. The authors curate a large-scale benchmark of biological protocols and propose Thoth, an RL-finetuned variant of Qwen3-8B. Thoth is trained with the proposed Knowledge-to-Action process and a novel SCORE reward mechanism that encourages logical ordering, completeness, and fidelity of the generated protocol. Experimental results suggest that Thoth improves over strong LLM baselines regarding the tasks of interests.", "questions": "- Could the authors further clarify the motivation behind step-level alignment? Is reproducing the exact number of steps in the reference protocol necessary for practical success, or would multiple granularities (eg., merged or decomposed steps) be acceptable as long as the outcome is correct, similar to math problems?\n- The SCORE mechanism (Fig 2) seems to encourage near-exact replication of ground-truth protocols, which could risk overfitting like SFT. Yet in Fig 5, Thoth (Stage 2 + 3) shows stronger gains compared to its non-RL counterpart (ie. SFT only). Could the authors elaborate how SCORE avoids over-constraining the model and still enables genuine reasoning improvements, if this happens?\n- How well would the Thoth framework generalize to other scientific domains (eg. chemistry or material synthesis protocols)? Are there any domain-specific assumptions baked into the data preprocessing or reward computation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper offers an end-to-end pipeline for studying scientific reasoning in LLMs regarding the biological protocol design, including dataset construction, structured prompting, and RL-based finetuning. This integration is well thought out and can serve as a good example for future research in ai4sci.\n- The curated biological protocol dataset , SciRecipe fills an important gap between free-form scientific text and machine-executable protocols. It provides valuable supervision for procedural reasoning tasks.\n- The authors conduct comprehensive benchmarking against a wide variety of strong models (gpt-4, deepseek-R1, etc.) and perform multiple ablations, showing consistent improvements in structural accuracy and executability.", "weaknesses": "- The study of Thoth model relies primarily on the Qwen3-8B backbone. Including additional base models (larger parameters to 32B, or other pre-trained such as deepseek-v3) could strengthen the claim that Thoth’s improvements convincingly stem from the training framework.\n- While the SCORE metric captures fidelity, it remains unclear how well it correlates with real experimental executability (act like some heuristics). Incorporating partial human or domain-expert evaluation and examine the correlation between the SCORE reward and real \"fidelity\", even on a small subset, would better improve the practical credibility of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z1pgQspfkP", "reviewer": "Reviewer_ULSZ", "summary": "### Overview\nThis paper proposes a new dataset (SciRecipe) and metric (SCORE) to evaluate the quality of scientific experimental protocols generated by language models. They introduce a formalization of the protocol generation process (Sketch-and-Fill) and use it in conjunction with the SCORE reward to train Qwen3-8B via supervised fine-tuning and GRPO on the collected datasets and reward function. The main contribution is the development of a formal specification for protocol generation that enables systematic rule-based validation of the generated protocols. The effectiveness of this formalization is demonstrated by a suite of experiments in which a language model trained to generate protocols by following this spec and reward achieve higher performance than baselines on the SciRecipe benchmark and some public scientific QA benchmarks. \n\nWhile the motivation for the benchmark and formalization are well introduced, I am leaning towards rejection because (1) it is not clear to me that the tasks in SciRecipe-Eval are truly benchmarking protocol generation, (2) the choices made in the SAF formalization can limit the expressiveness of possible protocols, (3) the terms in the SCORE reward function are not sufficiently justified, nor are they shown to capture meaningful fallacies / weaknesses in existing protocols, (4) the metrics used to compare models are either too simple (ROUGE, BLEU) or novel metrics introduced by the authors (Step-M, Order-S, etc.) that are not introduced in the main text (hidden in appendix)\n\n### Main argument for reject\nMany of the choices made by the authors for both the formalization framework (SAF) and the reward function (SCORE) have weaknesses and lack justification. The SAF framework seems overly restrictive to me. For instance, how does one encode branching logic in this `(Action, Object, Parameters)` format? Given the importance of this formalization for the rest of the paper, it would be nice if the authors could provide some quantitative way of measuring how well this formalization captures the semantics of an existing diverse corpus of high quality protocols. On the other hand, the SCORE reward function can only be applied to models that adopt the SAF generation framework, limiting its generality. It is not evident to me that the combination of order and semantic consistency are sufficient to capture the critical fallacies of existing protocols. An important experiment here would be to show quantitatively that the proposed reward function is sensitive to critical issues in existing protocols that scientific professionals would be able to identify. Finally, the evaluation metrics and the benchmark (SciRecipe-Eval) used for protocol generation (Section 4.2) are not clearly measuring what it claims. As far as I can tell, the SciRecipe-Eval tasks are not actually protocol generation tasks, but rather a collection of adjacent tasks, more so related to protocol comprehension and manipulation. Can the authors clarify why they claim that these tasks (in appendix B) are benchmarking generation rather than understanding and reasoning? Moreover, the evaluation metrics used to compare different models in section 4.2 are either too simple (ROUGE, BLEU, etc.), which the authors themselves have acknowledged, or are novel metrics (Semantic-A, Order-LCS, Order-S, Order-Tau, Step-M) that are not introduced in the main text (Appendix F) and seem to correspond to terms that were actually used the in SCORE reward function to optimize the author's own method (Thoth). This does not seem like a fair comparison, nor does it convincingly capture the quality of the generated protocol as mentioned earlier.\n\n### Things to improve the paper that did not impact the score:\n- there should be citations for some of the claims made in the introduction such as:\n  - \"\"\"*As a result, researchers often find that models offer fragmented recommendations on experimental procedures but fall short of producing concise, logically ordered protocols that can be directly implemented in laboratory workflows*\"\"\"\n  - \"\"\"*the generated protocols often contain unordered steps, redundant operations, factual inconsistencies, or hallucinated actions, undermining both reproducibility and scientific credibility.*\"\"\"\n- the introduction of key evaluation metrics should be made in the main text", "questions": "See summary section above.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The motivation of the paper is well articulated\n- To my knowledge, the SAF formalization and the SCORE reward are original\n- The curation of the dataset is a meaningful contribution\n- The curation process is well documented in the appendix\n- I appreciate the thoroughness of the experiments and ablations", "weaknesses": "- The tasks in the benchmark are not actually benchmarking protocol generation\n- The design choices in the formalization and reward function lack justification and validation\n- The evaluation metrics used to compare models in the experimental section are either too simple or unfairly recapitulate the design choices in the reward function used to train the model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)