Expert-level protocol translation for self-driving labs
Authors: Yu-Zhe Shi
Fanxu Meng
Haofei Hou
Zhangqian Bi
Qiao Xu
Lecheng Ruan
Qining Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Self-Driving Laboratories
Domain-Specific Language
Structural Representation
Knowledge Externalization
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Recent development in Artificial Intelligence (AI) models has propelled their application in scientific discovery, but the validation and exploration of these discoveries require subsequent empirical experimentation. The concept of self-driving laboratories promises to automate and thus boost the experimental process following AI-driven discoveries. However, the transition of experimental protocols, originally crafted for human comprehension, into formats interpretable by machines presents significant challenges, which, within the context of specific expert domain, encompass the necessity for structured as opposed to natural language, the imperative for explicit rather than tacit knowledge, and the preservation of causality and consistency throughout protocol steps. Presently, the task of protocol translation predominantly requires the manual and labor-intensive involvement of domain experts and information technology specialists, rendering the process time-intensive. To address these issues, we propose a framework that automates the protocol translation process through a three-stage workflow, which incrementally constructs Protocol Dependence Graphs (PDGs) that approach structured on the syntax level, completed on the semantics level, and linked on the execution level. Quantitative and qualitative evaluations have demonstrated its performance at par with that of human experts, underscoring its potential to significantly expedite and democratize the process of scientific discovery by elevating the automation capabilities within self-driving laboratories.

Expert-level protocol translation for self-driving labs
Yu-Zhe Shi', Fanxu Meng', Haofei Hou', Zhangqian Bi, Qiao Xu,
Lecheng Ruan Qining Wang
Department of Advanced Manufacturing and Robotics,
College of Engineering, Peking University
"Equal contribution
X
ruan mranlechher echeng@ucla edu, giningwang@pku.ed
Abstract
Recent development Artificial Intelligence (AI) models has propelled their ap- plication in scientific discovery, but the validation and exploration of these discov- eries require subsequent empirical experimentation. The concept of self-driving laboratories promises automate and thus boost the experimental process fol- lowing Al-driven discoverics. However, the transition of experimental protocols, originally crafted for human comprehension, into formats interpretable by chines presents significant challenges, which, within the context of specific expert domain, encompass the necessity for structured opposed to natural language, the imperative for explicit rather than tacit knowledge, and the preservation of causality and consistency throughout protocol steps. Presently, the task of proto- col translation predominantly requires the manual and labor-intensive involvement of domain experts and information technology specialists, rendering the process time- intenniiesive. To address these issues, we propose framework that automates the protocol translation process through a three-stage workflow, which incremen- tally constructs Protocol Dependence Graphs (PDGs) that approach structured on the syntax level, completed on the semantics level, and linked on the execution level. Quantitative and qualitative evaluations have demonstrated its performance at par with that of human experts, underscoring its potential to significantly expe- dite and democratize the process of scientific discovery by elevating the automa- tion capabilities within self-driving laboratories.
Introduction
The evolution of AI techniques has significantly accelerated the processes inherent to scientific dis- covery, with a notable impact observed within the domain experimental sciences (Wang et al.) 2023b). This influence is manifested through variety of avenues: the generation of hypothesis spaces informed by extensive literature analysis (Jablonka et al. 2022, Kim al.] 2024), the inter- pretation of observational data via the identification of high-dimensional correlations (Jumper al.) 2021 Abramson et al., 2024), the engineering of novel structures that meet predefined specifica- tions Grisoni et al. 2021 Park et al. 2023), and the implementation of comprehensive simulations
to ascertain the characteristics of potential products (Hie et al.) 2021 Singh et al.) 2023).
However, the findings facilitated by Al-driven research require further validation and exploration via empirical experiments, and may even entail a cyclical process where Al-generated hypotheses are refined based on the outcomes of real-world experiments, which demands the assembly of a sizable cohort of experienced experimenters to carry out these investigations accordance with established protocols (McNutu 2014). Unfortunately, the formation and sustenance of such dedicated exper- imental cadre are fraught with considerable financial demands, and the collaborative engagement between people oriented towards AI methodologies and those grounded in experimental sciences is frequently encumbered by the communication gaps between distinct intellectual paradigms (Baker
2016 Freedman al. 2015 Munafò al. 2017 Baker 2021 Shi et al. 2023a
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

NL-based protocol
Structured protocol
Completed protocol
Linked protecol
[.J
Spllt the mitture aqually into 2
aqpanett murdl fectors flaiis for the
Iriaas Tuad]]
Tr rlaak cemeratin,
12 Fiast anperiiary
Bist fire
maintaia this ansperstare white
sttrringor hour Chrokt the ampeerscare eveyy 20 rinuttal Matetain she wecod flank at 751G.
aurecion a 11)
Foror (at see
Allow the mixture cool moore
1:
Inteteed,
Inperbbre 181
of loe water indo
(tank:
P (1aa))
4uc de plauonal Uii contaire (flane Nuad)
Syntax
Semantios
Execution
Figure I: Illustration of the protocol translation problem. An NI.-hased protocol is translated to struc- tured protocol, then to a completed protocol, and finally to a linked protocol that ready for self-driving laboratories along with a corresponding PDG, after being processed through the syntax, semantics, and execu- tion levels. The three colors of arrows and text/ code highlights indicate the three translation steps respectively.
To bridge the aforementioned gap, the paradigm of self-driving laboratories has garnrred attention, which automates experimental protocols via robotic systems, potentially revolutionizing the way ex- periments are conducted (Bédard et al., 2018 Steiner et al. 2019: Mehr al., 2020 Rohrbach al.) 2022 Burger et al. 2020] |Szymanski et al. 2023). Despite the promising outlook, designing such labs relies largely on the translation of protocols, primarily designed for human experimenters, into machine-readable instructions. This translation process necessitates extensive collaboration between domain experts. who possess the requisite scientific knowledge; and information technology special- ists, who encode this knowledge into software and hardware systems. The inherently labor-intensive nature of such translation significantly prolongs the development of self-driving laboratories. The primary challenges are rooted in the discrepancies across three critical aspects (see Dig.F.gg 1):
Syntax Human experimenters can effortlessly comprehend protocols articulated in Natural Lan- guage (NL), whereas automated systems frequently necessitate dedicated syntax parsers to con- vert these protocols into sequence of actionable steps. Consider the protocol: "Split the mixture equally into 2 separate 50 ml. round-boltom flasks for the next steps." This example highlights the meticulous control over experimental procedures, explicitly directing the "split" of the mixture into precisely measured volumes- - a crucial factor for achieving uniform outcomes in subsequent reac- tions. is imperative this level to uphold structured representation of the mapping of operation conditions and the control flows of operations.
the flexibil- ity and adaptability of human understanding. In contrast, machine instructions necessitate a level of precision and rigidity that human communication does not inherently require. For instance, consider the protocol: "Stir the mixture at room lemperature for 5 minutes." While a human expert might inherently understand that "room temperature" denotes temperature range of 20-25 CC drawing on their prior knowledge, an automation system necessitates explicit information regarding such implicit details, which therefore need be completed before execution.
Semantics Human experimenters can infer implicit knowledge and context
Execution Human experimenters can simulate possible intermediate states and outcomes by con- sidering the cumulative effects of sequence of actions. For instance, given the two instructions adjacently: "Add 35 mL water to the flask" and "Add 25 mL water to the flask", an experimenter can deduce that the flask's minimal capacity comes over 60 mL to prevent errors. For automated system to perform a similar function, the actions need to be linked along their execution order.
Great efforts have been made on such translation tasks, among which Chemputer is representa- tive (Mehr et al. (020). This algorithm parses the NI.-based protocol into XDL. a Domain- -Specfi Language (DSL.) specially designed to describe chemical synthesis reactions. The completeness and

---

linkages are constructed with set of caanuullly-wittter constraints, with which the correctness of protocols can be further checked. This methodology has gained widespread acceptance in automated chemical synthesis, as a festament to the intensive efforts by domain and IT experts in developing XDL. and the corresponding constraints. However, the application of a similar framework other domains of experimental sciences, such as Genetics, Medicine, Ecology, and Bioengineering, would necessitate repeating these sabor-intensive tasks on a case-by-case basis, thus underscoring the crit- ical need for a more generally applicable, human-free protocol translator.
In this work, we propose a novel framework of human-free translator, designed to potentially facil- itate applications across diverse experimental science domains without requiring extensive manual intervention. This framework decomposes the translation challenge into three hierarchical stages: structured on the syntax level, completed on the semantics level, and linked on the execution level, mirroring the cognitive steps undertaken by human experts in similar translation tasks. In the pro- posed work, the DSL, its constraints, and linkages are generated automatically, based on protocols tailored for human experimenters, thereby eliminating the need for labor-intensive manual processes.
Our contributions are threefold: (i) We conduct a systematic analysis of the existing discrepancies in protocol translation between human experimenters and automated systems in self-driving labo- ratories. From this analysis, we derive design principles that emulate human cognitive processes involved in protocol translation (Sec. 2). (ii) We devise an autonomous protocol translator through a tripartite framework that incrementally constructs PDGs, encapsulating the spatial- ies of protocol execution across syntax, semantics, and execution levels (Sec. (iii) Through both quantitative and qualitative evaluations in various experimental science domains, we demonstrate that our translator, when integrated as an auxiliary module for Large Language Models (LLMss,, approaches the efficacy of skilled human experimenters and substantially surpasses the performance of purely LLMs-based alternatives in protocol translation tasks Sec.. 4).
dynam-
2 Protocol translation for self-driving laboratories
In this section, we explore the translation of protocols for human experimenters to those suitable for self-driving laboratories. We analyze the task requirements across syntax (Sec. 2.1), semantics See.. 2.2 . and execution (Sec. 2.3) levels. We pinpoint challenges at each level for both humans and machines, delving into systematic methods for addressing these issues. Leveraging expert insights, we delineate fundamental design principles for achieving effective protocol translation (Sec. 2.4
2.1 Syntax level
Operation-condition mapping In NL-based protocols, operations and their corresponding pa- rameters such as imput reagents and conditions, are entangled with each other. For example, "Dis- solve 10 g of sodium chloride in 100 mL of distilled water at 80"C", the entanglements of actions and conditions highlight the complexity machines face parsing such protocols. Human experi- menters can recognize them without information loss thanks to the internalized language for parsing NI. (Chomsky 1956 2007). In contrast, protocols for machines must be represented precisely, with proper extraction of keys and values, and matching between them with appropriate data structures.
Operation control flows In NL-based protocols, both linear and non-linear control flows are im- plicitly embedded in the text. While linear control flows, i.e., workflows in sequential execution order, can be straightforward, non- non-linear control flows such as iterative loops and parallel operations can be hard to detect because the signal and the operational domain can be separated. Consider the protocol: "Repeat the titration until the endpoint reached, then record the volume of titrant used". These steps embody non-linear control flow, challenging machines to correctly interpret the iter- ative process involved. Even human experts have to read the protocols carefully to understand the local and global structures to match the signals with operational domains, let alone machines.
2.2 Semantics level
Latent semantics of known unknowns Some assigned values of parameters are regarded as com- mon sense knowledge of domain experts by default, thus the values are omitted for simplicity or referred to via a proxy name following the domains" conventions. For example, the protocol instruc- tion "Dry the purified product at room temperature" relies on the experimenter's understanding
3

---

Syntax level
Stuti
Dynanee
Contect-fres
Corleod sways
(Sructures protocsi
Spli the mistore epuaily Isto squrnato Aaris
program hypothesis space
qeil
sptred
Operation dependence graph
hest
cool
add
Vioupe
Presully apitte/
Popectifed spites
Chesk the temperatury Eaay
Sarr SEEEE
arrduratt fee, centalert: respurce
Nett reagent,
whaa arreyties), contatner: resounce
canditionc
monural;;
IaEn Optimizaiion
1:35t:
conditous:
conditions
Allow rebcturee
roum drrppendarr.
Bakk:
Preatarnal mnde will tel, rengert.a..
Istetornal ante/ steeh: resgent,
dicatoo:
Bunk?
Add 20 of ine water iito the flush.
potres
Simin:
Semantics level
Stutee
Dyname
Cormeet-free
Oprtect- awars
Completed proteco
reagent
Rengente iMecycles
nimure
Reagent graph
JOe Water
cook
For wake
Vol:
Ice water
ada
nixture
misturel
cool
nictua
spilt
misture
spit
spti
beatichack
cool
Exocution lovol
Stori Dynarric)
Connest-ioon
Conten- sward
(Lnoed protccol
Protocol dependence graph at step
component
Spo5ia tompooal dynamies of execution Protoool dependence graph at stop tr1
component
Vel 1i
Vil
Vil
voter
Vel
Ice suder
Vol ma
mabuue
3
Assi Gloid
Infeaune
Inlmuree
cool
Addiiare Iemportl coniralt
peacon
poazzn
27C
possa
conditooss
conditions
cot corditions
Add www.rreoopoaoeeeeteoooo
conditira
conditiee
conditeen
7XPPC
Figure 2: The design principles and the resulting pipeline of our translator. (Syntax level) Operation dependence synthesis on the syntax level, through the joint optimization of DSL program syntax space and the parsing tree of the NL-bbasdd protocols. This process is static and context-free. (Semantics level) Reagent flow analysis on the semantics level, through an automaton scheme maintaining the lifecycles of reagents and intermediate products. This process is static and context-free. (Execution level) Spatial-temporal dynamics analysis on the execution level, through the partial execution trace model based on the apatial-teppora dual constraint representation. This process is dynamic and context-aware.
of what constitutes "room temperature". However, machines substantially suffer from such latent semantics, implying that every value of parameters should be made explicit.
Latent semantics of unknown unknowns Sometimes, even required parameters for a specific operation are omitted from the protocols either or not intentionally, causing unknown unknowns that one may even be not aware of the absence of such information. For instance, the protocol instruction "Centrifuge the sample after adding the enzyme" " does not specify the key controlling parameter, speed or duration, for the "centrifuge" operation, before describing its specific value. Both human and machines require every parameter of operations to grounded.
2.3 Execution level
Capacity of resources Protocols often omit explicit specifications of resource capacities, lead- ing to potential execution errors like exceeding device's maximum capacity. This issue, inherent in the execution sequence, is undetectable by analyzing single operations alone. For example, the instruction "Transfer the mixture to a beaker" requires choosing a beaker with adequate capac- ity, a decision based on the cumulative volume from previous steps. Humans intuitively manage this through a mental simulation of the experimental process (Gallese and Goldman| 1998). Ma- chines, therefore, need a pre-execution verification mechanism to ensure resource capacities are not exceeded, highlighting the need for an integrated understanding of the experimental sequence.
Safety of operations In addition to managing resource capacities, another source of runtime errors stems from operations that, while semantically valid, may lead to adverse or dangerous outcomes in certain execution contexts. Such scenarios necessitate a duallconstrrint approach, where experi- menters are mindful not only of the actions required what I should do but also of potential missteps to avoid what must not do. For instance, the instruction "Heat the reaction mixture to 70"C" can be appropriate or hazardous, depending on the mixture's 's composition - safe with heat-stable cat- alyst, but risky with a heat-sensitive component due to potential decomposition. To navigate these complexities, human experts effectively run mental simulations, conducting "What if?" queries and counterfactual reasoning (Hoch) 1985) to anticipate the consequences of their actions. Similarly, machines need system to draw upon domain-specific knowledge and historical context to assess the safety of cach operation, ensuring that all actions are contextually appropriate and safe.

---

2.4 Design principles inspired by human experimenters
Human experts' cognitive capabilities on the translation of protocols serve two key roles: under- standing protocols for in-hand experiments and manually developing translators for self-driving laboratories. Inspired by these practices, we outlined design principles for our translator and as- sessed the strengths and weaknesses of current DSLs for NL.-based protocols, such as XDL. (Steiner et al. 2019). ULSA (Wang et al.] (002)) ORD (Kearnes et al. (021), Biocoder (Ananthanarayanan and Thies 2010), Autoprotocol (Strateos) 2023), and the family of DSLs (hereinafter called ADSL) which are automatically designed by the Aut tool driven by domain corpora (Shi et al. 2024a).
Operation dependence synthesis for the syntax level To precisely comprehend the complicated operation-condiig condition mappings and non-linear control flows, machines should equip with an exter- nalized language in parallel with humans' internalized language (Chomsky) 2007). A machine- recognizable language commonly possesses Context-Free Grammar (CFG) which externally de- fines the key-value structures on different hierarchies: (i) operation as key, reagents and conditions as values; (ii) condition as key, the corresponding parameters as values; and (iii) signal of control flow as key, the corresponding operational domains as values. If a protocol can be parsed into an Abstract Syntax Tree (AST) with the CFG, it is verified on the syntax level (Hopcroft et al., 1996). resulting in the dependency structures of the operation flow (see (see 2 Top). All DSLs mentioned before are context-free languages with CFGs (Fowler, 2010), echoing this design principle.
Reagent flow analysis for the semantics level Despite the merits of DSLs based on CFGs, the context-free nature hinders verification on the semantics level, which is pivotal in protocols es- sentially describing procedures, where the preconditions and postconditions between temporally adjacent operations can be end-to-end connected. To be specific, although CFG defines struc- tural space with hierarchies of operations, conditions, parameters, and control flows, ie., "There must be several parameters corresponding to condition", it does not constrain the mappings under the context of domain-specific knowledge, i.e., "There are parameters controlling the Temperature, Duration, and Acidity of the condition ". If the exact mapping between keys and values cannot be specified, the self-driving laboratories can hardly be aware the loss of completeness, i.e., being aware of the omitted conditions given an operation or the missing value given a parameter, due to the extremely large search space over all symbols given by the corresponding DSLs (Gulwani et al.) 2017). The design choices of DSLs diverge on this level, where Autoprotocol only supports ver- ification on the syntax level and does not possess any domain knowledge, ORD and ULSA offer the relations between operations and conditions without more fine-grained parameters, while XDL, Biocoder, and ADSL offer the find-grained key-value relation below the hierarchy of operations without more constraints about the values, e.g., suggested values of specific parameters. Hence, we require a mechanism for completing the structures of reagent flow (see Fig. 2] Middle), while the completeness of the fine-grained parameters is guaranteed by the DSLs
Spatial-tempores dynamics analysis for the execution level Completion on the semantics level is conducted statically because the semantics of operations are viewed individually rather than con- textualized in the execution scquence. Regrettably, such effort cannot guarantce that the protocols can be executed successfully without any errors in the run time, which is unacceptable by self- driving laboratories (Christensen et al. 2021 Seifrid et al. 2022]. One way is to have domain experts write down all of the potential bad cases as constraints and use them for verification. How- ever, run-time errors raised in the dynamic context of operations are heavily long-tail tail distributed. This makes it extremely hard to predict such errors from statistical hindsight (Pearl 2019), i.e., the set of collected post-hoc bad cases. Thus, we leverage the powerful foresight based on simulation, which spans the full probabilistic worlds of each operation by its semantic constraints, both on the spatial dimension, e.g., capacity of resources captured by the reagent dependency, and the tempo- ral dimension, e.g., safety of operations captured by the operation dependency. The simulation is conducted along the topological order of the corresponding execution flow graph. At each opera- tion unit, both historical operations in the same protocols and similar operations in other protocols are recalled dynamically, checking and refining the dual-constraint spaces accordingly (see Fig. 2 Bottom). Interestingly, none of the DSLs in our discussion take this feature as part of language de- sign and only XDL employs an external compiler with hand-crafted rules for error detection. Such consideration is reasonable because in mainstream DSL design, verification on the execution level is not guaranteed by the DSLs themselves for design simplicity and user convenience (Mernik et al.) 2005). Consequently, we require an environment to dynamically check the correctness of execution both spatially and temporally, through synthesizing operation and reagent dependencies.
5

---

3 The framework of protocol translation
In this section, we introduce the three-stage framework for human-free protocol translation, which gradually constructs a structural representation of protocols, called the Protocol Dependence Graph (PDG). The PDG makes explicit both the operation and reagent dependencies for protocol. Opera- tion dependence echoes the concept of program control flow, which derives the condition of sequen- tial, branch, or loop execution of protocol operations (Sec. 3.1). Reagent flow provides an explicit representation of the reagent instantiate-exploit relationships implicitly in the protocol (Sec. 3.2). Additionally, we simulate the protocol execution process using the PDG, checking and refining op- cration sequences under the spatial and temporal dynamics of execution (Sec. 3.3).
3.1 Operation dependence synthesis for the syntax level
The operation dependence models the topological order for executing operations in protocol. The procedure executed sequentially from the first operation the protocol to the last, unless the experimenter encounters structures that change the execution flow, such as branches and loops. In practice, we extract the operation dependence by compiling the protocol to DSL programs.
Input The compilation process is conducted based on the corresponding DSL L {S,A]. The CFG-based syntax S (S, V, Σ. R) includes (i) the start symbol S: (ii) the variable set V = (Verri, Vop: Voona, }pa= with placeholders for control flow signals Vetrl, operations Vop, conditions Veond, and parameters Vgaz: (iii) the set of terminals Σ, which are the grounded values of parameters; (iv) the set of production rules R defining the structural space between the four vari- able sets. The semantics A = (Totr Top: Toond, Tpar, TR) constrains the variables and production rules, assigning the placeholders with substantial meanings. To note, according to the design choice discussed in SScc. 2.4] the DSL used here should be one of XDL, Biocoder, and ADSL, i.e. the DSLs with the most fine-grained structural representation compared with their counterparts.
Pre-processing Given an input protocol e for translation, we first parse the NL sentences by an off-the-shelf tool and extract the actions accordingly. Then, the extracted actions are matched with the operations o Top of the DSL,, according to both exact match score and semantic similarity. Afterwards, we extract the arrays of entities related to the extracted action C₂ E & by an off-the-shed tool, where we regard the output labels to the entities and relations as pseudo- labels because they can possibly be noisy. Please refer to Appx. C.I for implementation details.
DSL program synthesis Synthesizing structural representation given unstructured signal is chal- lenging (Billard) 2000) 2006). Specifically, the one-to-many mappings of many DSL operations bring uncertainty into the matching. For example, the operation add possesses distinct patterns: two or three input slots. This further distorts the matching of reagents, conditions, and parameters because off-the-sellf tools can hardly detect the exact categories of these entities deeply rooted in domain-specific knowledge. Since the observation and hypothesis spaces are both noisy, we propose to jointly optimize the patterns of operations and the pseudo-labels We denote the set of all possible program patterns generated by operation o as o* = {plo =* p. Ta = P,P e T ctrll U T U Th O T A synthesized DSL program is defined as p(c) = (p(o1), p( (((pp)....p((pppe) )), where p(o)) is the program of an operation assembled with its corresponding conditions and param- eters under the selected pattern in o*. Let s(c) (e), e2,..., ,,,(e)) represent the sequence of operation-related entities, the objective of optimization can be
arg min p(o.).o,
((rcO)l(())..
(1)
where D(-II-) is a divergence function with three indicators: (i) the selected pattern examples should be close as possible to the text span; (ii) the selected pattern should be as similar as possible with the extracted subject-verboobject structure; and (iii) as many labeled entities as possible should be mapped to the parameter space (see Fig. 3B). Though |o*| is not a large value, the whole sequence of operations can yield an exponential complexity. Hence, to make the joint optimization tractable, we separate the search of solution into two steps the spirit of Expectation Maximization (EM); (i) Expectation: sampling programs from the legal space defined by the corresponding DSL, with program-size size-sensitive prior [p(ot)]_1; (ii) Maximization: randomly alternating symbols in s(c) by matching them with those in p(c) and greedily select the edits that decrease the objective function.

---

3.2 Reagent flow analysis for the semantic level
The DSL programs are further contextualized by associating operations with reagent flow. Reagent flow indicates the transfer of reagents among operations, reflecting how one opera- tion impacts the subsequent ones. We define the reagent flow following the reaching defi- nitions schema (Alfred et al., 2007), which is commonly used to capture the life cycle of a variable in compiler design. This schema de- termines set of reagents reachable at each point in a protocol, and subsequently tracks the kills and defines of an operation, i.e., whether a reachable reagent is consumed, or new reagent is yielded, in an operation.
Algorithm 1 Reagent flow analysis
procedure TRANSITION(M, o)
+- CCnntext Transition
(RASE(M(T)), (ILLS(M(()).. o))
(PP)ND(M(T), DEFINES(O))
P- SState Transition
M(1) - (M(1) u NEXTOPS(O)
procedure FLOW(p(c),
R = =1
+ Seet of reagent dependence
(()) {0}
Initial State
([) - ()
t Initial Memory
while (g) * Φ do TRANSITION(M, M(q))
Input
We denote the reagents consumed and the intermediate products yielded by each op- eration o as IN(o) and OUT(o) respectively. The objective is to find set of operation pairs { (o,, o,> |OU((o)) o IN(O,) * o} such that OuT(o:) is required as input by IN(o.).
The reaching definitions schema We determine the availability of a reagent at each step by locat- ing where it is defined in a protocol when execution reaches each operation. A reagent r reaches an operation if there is a path from the point following to o, such that r is not killed, i.e., consumed, along that path. Any reagent that reaches an operation o might be killed at that point, and o may yield new intermediate products r' that reach future operations. Notably, according to statistics on corpora of protocols /Vaucher et al., 2020p, for about 90% operations of a target DSL., if o, < 0g are two adjacent operations, OUT(o:) (N(o)) * & holds. This implies that the reagent generated by preceding operation is likely to be used and then be killed instantly by the following operation.
Reagent flow analysis via operation flow traversal We traverse the DSL program in execution order leverage the reagent locality revealed from statistical results, determining the reachability and life cycle of reagents. A Pushdown Automaton (PDA) with a random access memory is adopted to record reachable reagents as operation context, defining and killing reagents at each operation point along the " A PDA is formally defined as a 7-tuple M = (Q. Σ, r, 6, go. 2, F), where Q S Top indicates the set of states, Σ Top represents the domain of inputs, < Tpar de- notes possible memory elements, 8 = x
Σx r - Q r is the transition procedure (TRANSITION in Alg. Th, 90 01 defines the initial state, Z c the initial memory element, and F denotes the set of accepting states. The reagent dependence construction process (FL.W in Alg. 1) traverses the DSL program in execution order by leveraging the NEXTOPS utility, which evaluates to subsequent operations. In every transition step with input, the killed reagents are removed from the memory, and the defined reagents are added to the memory. After a reagent is killed, the pair of the operations that defined it and killed it will be added to the set of reagent flow constraints. The accepting state is reached if the memory is empty at the end of execution, i.e., all reagents defined in operations are killed by other operations. We employ -fate-of-the-art LLMs to extract reagent entities from NL- based protocol descriptions for the two utilities KILLS and DEFINES through cetrructionfoollwinnn in-context learning (Wei et al. 2021 Brown et al. 2020 (refer to Appx. C.2 for details).
3.3 Spatial-temporal dynamics for the execution level
While the pre-specified PDG analysis indicates the things should be done to follow operation and reagent flow, we still need to care about the things must not be done by describing the activities that may be performed and the constraints prohibiting undesired execution behavior. Therefore, we introduce a constrained- grounded in the theories of process modeling and execution (Dourish al. 1996 Pesic et al. 2007).
execution model to support dynamic protocol simulation, getting
constraint-based protocol execution model We extend the DSL program p(c) by a constraint set C = Cap Crey U C, U C. to construct constraint-based execution model S (p(c),
The execution of a program is represented by a trace = < c1), (02, C2), ((]p))) 9)())))
It is worth noting that this extended PDA with random access can be shown to be in the same computation class as Turing machines (Aho and Ullman| 1972. and employ this extended PDA due to its simplicity.
7

---

where the order of o, reflects the temporal sequence. The execution context defines the spatial environment in which each operation performed. Each constraint S(C) is a predicate that maps the execution trace to binary condition denoting satisfy or not. An execution trace is said to satisfy the program constraint if and only if |o| |p(c) and c(a) holds for all c C.
Leveraging partial execution trace for spatial and temporal constraints Explicit constraints, namely operation and reagent flow, are easy satisfy. Unfortunately, deriving implicit constraints, egg., the capacity of resources and safety of operations, is case-by-case for each protocol, requiring expert efforts. We propose profiling the context through execution derive implicit spatial and tem- poral constraints that meet domain-specifer requirements. An execution trace is defined as partially satisfying the constraints C' if it follows the operation and reagent flow; that is, for any (o,, 0z=1) in trace 0, there exists at least a pair of valid operation flow path and reagent flow path from to Oj.
4 Results
In this section, we compete our framework with human experts on the overall translation task, and assess the utility of each component of the framework by comparing with alternative approaches.
4.1 Experimental setting
Materials We select 75 complicated experiments with 1,166 steps in total as the testing set, from the domains of Chemical Synthesis (235 steps in 10 experiments; 235 in 10 for simplicity; "Synthe- sis" for abbreviation), Genetics (396 in 34), Medical and Clinical Research (307 in 23, "Medical"), Bioengineering (218 in 17), and Ecology (10 in 1). Please refer to Appx. for details.
Expert-created protocol translation We recruited five groups of experienced experimenters, each specializing in different domain. with seven participants in each group. Every participat- ing experimenter holds at least Master's degree related to the corresponding domain, has obtained least six years' experience in manually conducting pre-designed experiments of the domain, has acquired elementary programming skills, and has at least heard of self-driving laboratories. These human experts are asked to translate the original NL-based protocols for their domains into those suitable for self- self-driving laboratories. Their outputs are subjected to DSL-based representations and complete PDGs evaluate machines' behaviors, which are clearly demonstrated by running exam- ple and the examples in the DSL documentation. Outputs from experts are carefully cross-validated and the individual divergence between them is minimized through an expert-panel-drit
workshop discussion following the established workflow (Reilly et al. 2023). Translation results of human experts and machines are serialized and are compared through ROUGE and BLEU metrics (Lin 2004 Papineni et al. 2002). Please refer Ao B for ethics considerations.
Alternative methods We compare our translator with alternative methods on the first two levels, to investigate the effects of early stages on the overall translation result. On the syntax level, we compare the syntactic synthesis method (referred to as Ours-SY) with ConDec-SY (Wang al.] 2023a), which synthesizes DSL programs by LLMs with external DSL grammars as constraint, and a baseline DSL-LIM-SY leveraging the minimal realization used in (Sh et al. (0024)). On the semantics level, we compare the deductive verification method (referred to as Ours SE.) with NI.-RAG-LIIM- SF, which retrieves on the embedded vector database of original NI.-based proto- cols, and a baseline WL-LIM-SH SE implemented by pure prompt-engincering on LLMs. Since the 1/Oes of all stages are unified in the pipeline, we implement an overall baseline Best-Baselin that combines the strongest alternative methods within the evaluation of the first two stages.
4.2 Overall assessment on expert-created protocol translation
Result Comparing the overall output of our translator and ideal human experimenters, we find that our translator approaches the level of experts with average performance higher than 85% across all indicators. Our translator significantly outperforms the alternative pipeline Best-Baseling
the (t( ((118) = 17.771 aa < 0, < .0005; see Fig. 3C).
Discussion We find that our translator demonstrates similar performance to human experts in translating protocols with complete parameters and clear descriptions (see Fig. 4A). However, in cases where the linear description of the experimental protocol is lacking, our translator and human
8

---

A
Corpora
DSL
Objectivv funclion score
1.0
Synthesis
0.75
Medical
Genetics
0.6
0.50
Ecology
BloEng
0.25
0.2
BloEng
Ecology
Genetica
Medical
Synthesie
BioEng
Ecology
Genetiss
Medical
Synthesis
Iteration
0.00
C
parameter
structure
BLEU-1
ROUGE-LFF
ROUGE-LE ((PGEE-L(Prisiio)
ROUGE-L(Recall)
Expert
0.5
0.0
Best-Baseline
Ours
Best-Baseline
Ours
Best-Baseline
Ours
Best-Baseling
Ours
D
BLEU-1
ROUGE-L(F1)
ROUGE-L L(Precision)
ROUGE-L(Reca)
0.5
0.0
ConDec-SY
DSL-LLM-SS
Ours-SY
ConDec-SY SY
DSL-LLM-SSY
Ou
ConDec-SY
OSL-LLM-SST
Oura-SY
ConDec-SY
DSL-LLM-SY
Ours-SY
E
BLEU-1
ROUGE-L(F1)
ROUGE-LI
(UUGE-L(Precison))
ROUGE-L(Recad
1.0
0.5
0.0
ML-LLMMSE
-A-RAG-LLM-SE
Ours-SE
NL- -L-LLM-SE
AL-RRAGLLLM-SE
Ou
-L-LLM-SE
AL-RAG-LLM-SE
Ours-SE SE
WL-LLM-SE AL-RRG-LLM--SE
Ours-SE
domains regarding domain domain-speeific cor- pora and the corresponding DSLs. (B) Convergence of the three indicators in the objective function for program synthesis. (C) Our translator sigmificantly outperforms the best baseline and approaches human-level perfor- mance. (D) Our translator significantly outperforms alternative methods on the syntax level. (E) Our translator significantly outperforms alternative methods on the semantics level.
Figure 3: Results experiment. (A) Distinctions between
experts diverge. Specifically, our translator tends to translate based the information within the sentence or between adjacent sentences, while human experts tend to consider the overall experi- mental process comprehensively. Though there are minor gaps, these observations suggest that our translator is approaching the level performance of experienced human experimenters. Please refer to Appx. E.3 for case studies on the distinctions between the behaviors of experts and machines.
4.3 Comparison between alternative models
Result On the syntax level, our Ours SY significantly outperforms alternative approaches (t(148) - 11.007, μa 0, p .0005 for ConDec-Si SY; t(148) 15.47, μa 0,p .0005 for DSL-LIA SY: see Fig. 3D). On the semantics level, our Ours- SE significantly outperforms -2.2 p4 0,p .05 for MIL-RAG- RAG-LLM-- t( (148)
alternative approaches (4(148
-3.07, Pa < 0.p < .005 for NL-LIM-SE; see Fig. 38).
Discussion Compared with alternative methods on the syntax level, our translator excels in en- suring accuracy of translations across different protocols thanks to the PDG representation, as shown in Fig. 4B. On the semantics level, our translator performs better in translating incomplete protocols with missing information than other baselines, as shown in Fig. 4C. We explain these merits with the properties of structural representation, which defines representation space with

---

A
NL protocol
Tea
Agpare
Orignnal Test:
1. Incubatee sidlo EDTA for 50 antigrn anreasilng
Expert
incuiste
cool
incolate
wach
for 20 of room tompenation.
Jiddr
alider
abiee
siider
siido
incubota
cool
wash
inccubate
mash
3. Mlyuk to 820
Ours
4. Incullale siudo the Hi,
for 5 min ot room lompenature
6. Wash naker TRS.
Best-Baseline
incubate
sacubate
B
Syntax
Semantics
Orgginal Tos
Original Tixt
Tranger she romaining oi! yoote (400 pll a vonntilstion sial.
Stale the sids winh BAe kackts ard aute for scrind
2. Warh thrre timer with PAS slde beatom apphing pentie shobing GooLsEnuh
Darts
tramsfer
Velumm: 430
actrefttattinn vial
Lysete
velamas 450 14
Acfeetttarton viat
teaete
Peggete (PaS)
catte: cadtelanetes oelta
coneatier: acteerttacinn vist
1061C
[98]
Clesl)
Figure 4: Showcases of the results. (A) Examples the final PDGs generated by our translator, the alternative method, and human experts. (B) Examples of structured protocols output by our translator and alternative methods. (C) Examples of completed protocols output by our translator and alternative methods.
high expressive power (Felleisen, 1991 Lloyd 2012), capturing information along the spectrum of
granularity, from local details like tine-grained parameter-value relations to global-structure of con- trol flows. By contrast, the original NL representation and its embedding space may not possess such extent of expressive power (Zhang al.) (021). Consequently, mapping the protocols into structural latent space for query or retrieval should be more suitable than directly operating on the original space. Please refer to Appxs. E. and E.2 for details on the utilities of different approaches.
5 General discussions
In this work, we study the problem of translating protocols for human experimenters into those proper for self-driving laboratories. We design a human-free protocol translator under the inspiration of human experts' cognitive processes in protocol translation. Results suggest that our translator comparable with experienced human experimenters in protocol translation, further implying its potential to serve as plug-and-play module for fully automated scientific discovery.
Closing the loop of automatic scientific discovery The realization of automatic protocol trans- lation closes the last-mile of closed-loop scientific discovery. Integrating the translator with other cutting-edge techniques, we can expect such pipeline in the future: after AI models for scien- tific discovery have output insights from initial observations, there is a self-driving laboratory ready for producing and testing the designs. The creation of this laboratory is also automatic (Shi et al,) 2024c). In the conventional creation process, we must employ a cohort of experienced domain experts to hand-craft DSL tailored for representing protocols of the target domain, and then hand- craft translator with pre-defined production rules and constraints to translate the protocols into machine executables. Hence, the status quo has been changed: we can exploit the AutoDSL tool to obtain the DSL. based on corpus of the target domain (Shi et al., 2024a), and then set up our automatic translator for final exccution; we can also integrate the DSLs and our translator into LLM- based "scientist agents" for more reliable and flexible domain specification (Boiko et al. 2023 Bran et al. 2023). The integration of these approaches facilitates the grounding of Al for scientific discovery, which helps AI researchers test, produce and deploy their discoveries more seamlessly.
Valuing domain experts One might worry that the full realization of closed-loop scientific dis- covery would pose a severe impact on conventional experimental scientists. Indeed, Al for scientific discovery and self-driving laboratories do not compete with conventional scientific discovery. As interdisciplinary methodologies. they still demand the supervision of scientists for tacit knowledge, creativity, and integrity (Shi et al. (024b). Also, such techniques free domain experts from those and fabor-intensivy low-level workloads and focus them high-level thinking.
10

---

Acknowledgements
This work was partially supported by the National Natural Science Foundation of China under Grants 91948302 and 52475001. Part of the authors are visiting students at Peking University during this work. In particular, Z. Bi is visiting from Huazhong University of Science and Technology and Q. Xu is visiting from University of Science and Technology of China. The authors would also like thank Jiawen Liu for her assistance in figure drawings.
References
Abramson, J., Adler, J., Dunger, Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. (2024). Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3.
Aho, A. V. and Ullman, J. D. (1972). The theory of parsing, translation, and compiling, volume 1. Prentice-Ho Hall Englewood Cliffs, NJ.
Alfred, V. A., Monica, S. I.., and Jeffrey, D. U. (2007). Compilers Principles, Techniques & Tools. pearson Education.
Ananthanarayanan, and Thies, W. (2010). Biocoder: A programming language for standardizing and automating biology protocols. Journal of Biological Engineering, 4:1-13.
Baker, M. (2016). 1,500 scientists lift the lid reproducibility, Nature, 533(7604).
Baker, M. (2021). Five keys to writing reproducible lab protocol. Nature, 597(7875):93-294,
Bédard, A.-C., Adamo, A., Aroh, K. C., Russell, M. G., Bedermann, A. A., Torosian, J., Yue, B., Jensen, K. F,, and Jamison, T. F. (2018). Reconfigurable system for automated optimization of diverse chemical reactions. Science, 3616640)):220--222
Billard, L. (2000). Regression analysis for interval-valued data. Data Analysis, Classification, and Related Methods.
Billard, L. (2006). Symbolic data analysis: what is it? In Proceedings in Computational Statistics: 17th Symposium Held in Rome, Italy.
Boiko, A., MacKnight, R., Kline, B., and Gomes, G. (2023). Autonomous chemical research with large language models. Nature, 624(7992):570-578
Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A., and Schwaller, P. (2023). Augmenting large language models with chemistry tools. In NeurIPS 2023 Al for Science Workshop.
Brown, T. Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems.
Burger, B., Maffettone, P. M., Gusev, V. V., Aitchison, C. M., Bai, Y., Wang, X., Li, X., Alston, B. M., Li, B., Clowes, R., al. (2020). A mobile robotic chemist. Nature, 583(78155;237-241
Chomsky, N. (1956). Three models for the description of language. IRETaaanactions Information Theory, 133::113-
Chomsky, N. (2007). Approaching ug from below. Interfaces + recursion language, 89:1-30.
Christensen, M., Yunker, L. P., Shiri, P., Zepel, T., Pricto, P. L., Grunert, S., Bork, F,, and Hein, J. E. (2021). Automation isn't automatic. Chemical Science, 12(47):15473-1549)
Dourish, P., Holmes, J., MacLean, A., Marqvardsen, P., and Zbyslaw, A. (1996). Freeflow: medi- ating between representation and action in workflow systems. In Proceedings of the 1996 ACM conference on Computer supported cooperative work.
Felleisen, M. (1991). On the expressive power of programming languages. Science of computer programming, 17(1-3)355-75

---

Fowler, M. (2010). Domain-specific languages. Pearson Education.
Freedman, L. P., Cockburn, M., and Simcoe, T. S. (2015). The economics of reproducibility in preclinical research. PLoS Biology, 13(6)x1002116.
Gallese, V. and Goldman, A. (1998). Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Sciences, 2112):493-
Grisoni, F., Huisman, B. J., Button, A. L., Moret, M., Atz, K., Merk, D., and Schneider, G. (2021). Combining generative artificial intelligence and on-chip synthesis for de novo drug design. Sci- Advances, x(24)xeab33338.
Gulwani, S., Polozov, O., Singh, R., et al. (2017). Program synthesis. Foundations and Trends@ in Programming Languages, 4(1-2):1-119
Hie, B., Zhong, E. D., Berger, B., and Bryson, B. (2021). Learning the language of viral evolution and escape. Science, 371(6526):284 288.
Hoch, S. J. (1985). Counterfactual reasoning and accuracy in predicting personal events. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11(4):719.
Honnibal, M. and Johnson, M. (2015). An improved non-monotonic transition system for depen- dency parsing. In Annual Conference on Empirical Methods Natural Language Processing.
Hopcroft, J. E., Motwani, R., and Ullman, J. D. (1996). Introduction Automata Theory, Lan- guages, and Computation. Addison- Wesley Longman Publishing Co., Inc.
Jablonka, K. M., Patiny, I.., and Smit, B. (2022). Making the collective knowledge of chemistry open and machine actionable. Nature Chemistry, 14(4):365-376.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zídek, A., Potapenko, A., et al. (2021). Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588
Kearnes, S. M., Maser, M. R., Wleklinski, M., Kast, A., Doyle, A. G., Dreher, S. D., Hawkins, J. M., Jensen, K. F., and Coley, C. W. (2021). The open reaction database. Journal of the American Chemical Society, 143445):
Kim, C., Gadgil, S. U., DeGrave, A. J., Omiye, A., Cai, Z. R., Daneshjou, R., and Lee, S.-I. (2024). Transparent medical image ai via an image-text foundation model grounded in medical literature. Nature Medicine, pages 1-12.
Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summarics. In Text summarization branches out.
Lloyd, W. (2012). Foundations logic programming. Springer Science & Business Media.
McNutt, M. (2014). Reproducibility. Science, 343(6168):229-229
Mehr, S. H. M., Craven, M., Leonov, A. I., Keenan, G., and Cronin, L. (2020). A universal system for digitization and automatic execution of the chemical synthesis literature. Science, 370 108.
Mernik, M., Heering, J., and Sloane, A. M. (2005). When and how to develop domain-specine languages. ACM Computing Surveys (CSUR), 36-4)331--34.
Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, D., Percie Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., and Ioannidis, J. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1):1-9.
Papineni, K. Roukos, S., Ward, T., and Zhu, W.J. (2002). Bleu: method for automatic evalu- ation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics.
12

---

Park, N. H., Manica, M., Born, J., Hedrick, J. L., Erdmann, T., Zubarev, D. Y., Adell-Mill, N., and Arrechea, L. (2023). Artificial intelligence driven design of catalysts and materials for ring opening polymerization using domain-specifie language. Nature Communications, 14(1):6686.
Pearl, (2019). The seven tools of causal inference, with reflections on machine learning. Commu- nications of the ACM, 6203):54-60.
Pesic, M., Schonenberg, M. Sidorova, N., and van der Aalst, W. M. (2007), Constraint-based workflow models: Change made easy. OTM Confederated International Conferences" On the Move to Meaningfal Internet Systems".
Reilly, J., Shain, C., Borghesani, V., Kuhnke, P., Vigliocco, G., Peelle, J., Mahon, B., Buxbaum, L., Majid, A., Brysbaert, M., et al. (2023). What we mean when we say semantic: A consensus statement on the nomenclature of semantic memory. OSF preprint.
Rohrbach, S., Šiaučiulis, M., Chisholm, G., Pirvan, P.-A., Saleeb, M.. Mehr, S. H. M., Trushina, E., Leonov, A. I., Keenan, G., Khan, A., et al. (2022). Digitization and validation of a chemical synthesis literature database in the chempu. Science, (776660)
Seifrid, M., Pollice, R., Aguilar-Granda, A., Morgan Chan, Z., Hotta, K., Ser, C. Vestfrid, J., Wu, T. C., and Aspuru-Guzik, A. (2022). Autonomous chemical experiments: Challenges and perspectives on establishing a self-driving lab. Accounts of Chemical Research, 55(17):2454- 2466.
Shi, Y.-Z.. Hou, H., Bi, Z., Meng, F., Wei, X., Ruan, L., and Wang, Q. (2024a). AutoDSL: Au- tomated domain-specific language design for structural representation of procedures with con- straints. In Annual Meeting of the Association for Computational Linguistics.
Shi, Y.-Z., Li. H., Ruan, L., and Qu. H. (2024b). Constraint representation towards precise data- driven storytelling, In IEEE Visualization and Visual Analytics Gen4DS.
Shi, Y.-Z., Li, S., Niu, X., Xu, Q., Liu, J., Xu, Y. Gu, S., He, B., Li, X., Zhao, X., al. (2023a).
PersLEARN: Research training through the lens of perspective cultivation. In Annual Meeting of the Association for Computational Linguistics.
Shi, Y.Zz., Xu, M., Hopcroft, J. E., He, K., Tenenbaum, B., Zhu, S.-C., Wu, Y. N., Han, W., and Zhu, Y. (2023b). On the complexity of Bayesian generalization. In International Conference Machine Learning.
Shi, Y.-Z., Xu, Q. Meng, F., Ruan, L., and Wang, Q. (2024c). Abstract Hardware Grounding towards the Automated Design of Automation Systems. In International Conference on Intelligent Robotics and Applications.
Singh, S. H., van Breugel, F., Rao, R. P., and Brunton, B. W. (2023). Emergent behaviour and neural dynamics in artificial agents tracking odour plumes. Nature Machine Intelligence, 5(1):58-70.
Steiner, S., Wolf, J., Glatzel, S., Andreou, A., Granda, J. M., Keenan, G., Hinkley, T., Aragon- Camarasa, G., Kitson, J., Angelone, D., et al. (2019). Organic synthesis in a modular robotic system driven by a chemical programming language. Science, (63
Strateos (2023). Autoprotocol specification. httpss//huuoooooooooooggggggtitt
Szymanski, N. J., Rendy, B., Fei, Y., Kumar, E., He, T., Milsted, D., McDermott, M. J., Gallant, M., Cubuk, E. D., Merchant, A., et al. (2023), An autonomous laboratory for the accelerated synthesis of novel materials. Nature, 624-7990):86-91.
Vaucher, A. C., Zipoli, F., Geluykens, J., Nair, V. H., Schwaller, P., and Laino, T. (2020). Automated extraction of chemical synthesis actions from experimental procedures. Nature Communications, 11(1):3601.
Wang, B., Wang, Z.,, Wang. X., Cao, Y., A Saurous, R., and Kim, Y. (2023a). Grammar prompting for domain-specifie language generation with large language models. In Advances in Neural Information Processing Systems.
13

---

Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P. Deac, A., al. (2023b). Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47- 60.
Wang, Z., Cruse, K., Fei, Y., Chia, A., Zeng, Y., Huo, H., He. T., Deng, B., Kononova, O., and Ceder, G. (2022). ULSA: Unified language of synthesis actions for the representation of inorganic synthesis protocols. Digital Discovery, 33-3341-324.
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. (2021). Finetuned language models are zero-shot learners. arXiv preprint UrXiv:2109.01663
Xie, T., Li, Q.. Zhang, Y., Liu, Z., and Wang, H. (2024). Self-improving for zero-shot named entity recognition with large language models. arXiv preprint arXiv:2311.08921.
Zhang, J., Chen, B., Zhang, L.,, Ke, X., and Ding, H. (2021). Neural, symbolic and neural-symbolic reasoning on knowledge graphs. A/ Open, 2:14-35.
14

---

A Additional remarks
A.1 Rationale for the evaluation metrics
Direct comparisons across entire sentences under BLEU and ROUGE scores would indeed pose problem - it may be problematic considering instructions that look similar could have very different semantics. Therefore, circumvent this issue, we convert all results into standardized JSON-style format for data representation, and comparisons are made between key-value pairs rather than entire sentences, effectively resolving the metric concern.
Let us consider comparing similarity between the following two instructions "... pour hot water ..." and ... pour cold water ..... We transform them into the following ISON-style format:
1
2
3
functicn_nane "pour",
4
reagent: "water",
5
temperature: "hot",
6
7
8
9
function_name "pour",
reagent: "water", temperature: "cold",
14
15
The comparison between the sentences is then transformed into comparison between two JSON code blocks. We calculate the similarity score cumulatively based on the similarity between the values of matched pairs of keys. For instance, for the key "temperature", the values "hot" and "cold" yield a low similarity score under the ROUGE, BLEU, and even the Exact Match metrics. As "temperature" is one of the major keys within configuration parameters, high penalty this dimension significantly affects the cumulative similarity score. With this fine-graines comparison metric, can comprehensively track the distinctions and commonalities between results without losing expressivity regarding the quantities.
We also acknowledge that there advanced evaluation metrics, especially in the recent works where LLMs are leveraged as external judges and achieve considerable performance in general testing cases. Our choice of lexs advanced metrics is driven by the intention to focus specifically on domain-specific knowledge, which constitutes the primary scope of this paper and may be relatively sparse in general L.MSs. Nonetheless, the exploration of more sophisticated evaluation metrics represents a promising avenue for future research.
A.2 Insight behind the design of PDG
We have constructed the operation dependence graph on the syntax level and the reagent flow graph on the semantics level. Indeed, the two analytical results come with duality. the operation dependence graph, vertices represent operations and edges represent reagents passed between them. In contrast, the reagent flow graph uses vertices for reagent states and edges for operations causing state transitions. Interestingly, the vertices of one of the two graphs can be one-to-one mapped to the edges of another, echoing the duality. On a higher level, we say that the former provides an experimenter-centeen view while the latter offers reagent- centreee view. These two perspectives are complementary on encoding both the information of the interventions to the environment and the status of the environment itself. Consequently, by leveraging such duality, we are able to track spatial dynamics, egg., the variance of required resources, and temporal dynamics, e.g,, the context of sequential operations, simultaneously on the PDG.
A1

---

A.3 Computational complexity of the framework
Let us consider new coming protocol with & steps, with each step configured by constant number of parameters, denoted as On the syntax level, the primary computation bottleneck arises during DSL program synthesis, where the EM Algorithm exhibits a worst-case complexity Oce)). This is highly conservative estimate, as mainstream optimization approaches can solve the EM much more efficiently. On the semantics level, the bottleneck occurs during reagent flow analysis, which consumes O(k²) complexity. Notably, only approximately 10% of the steps included in the nested loop for reagent flow construction, as about 90% of the steps are linearly connected. On the execution level, the protocol execution model also exhibits O(A²) complexity, encompassing both forward and backward tracing. This can be optimized by replacing the full tracing strategy with a sliding window built upon the topological dependencies between steps. Although the complexities of the algorithms at these three levels are tractable, there substantial room for improving the efficiency of the framework. Investigating methods to speed up the translation process for protocols with extremely high complexity would be a valuable area research.
A.4 Generality of the framework
The general applicability of our proposed framework beyond experimental sciences indeed be a common concern. The core value of translating NL-based protocols into formats suitable for ma- chine execution substantially lies in facilitating experiments self-driving laboratories, thereby ac- celerating scientific discovery. Experimental protocols come with unique properties and challenges, such as the fine-grained incorporation of domain-specific knowledge, the non-trivial dependency topologies between operations, the long-horizon lifecycles of intermediate productions, and the ne- cessity for precise execution without run-time errors. These factors shape the scope of our research problem, emphasizing the need to handle protocols with stringent terminology and formatting.
Despite the specific scope of this paper, we are open to exploring the potential for generalizing our framework to other domains with similar properties and challenges as those found in scientific experiments such cooking. Imagine self-driving kitchen that automatically prepares all ingredients and executes all procedures for cooking a meal according to NL-based recipes. Such self-driving kitchens would also benefit significantly from translating numan-oriented recipes into formats suitable for machine execution. In the following, we present a running example of such translation, adapted from use case of the Core DSL.
The protocol after pre- pre-processing is as follows.
1
Pasta Bolognese
2
3
Yield: plates
4
5 Ingredients:
6
7 [ounces] white fresh (pasta)
9
[floz] clive (oil)
11
1/4 [ounce] (garlic); minced
12
13
4 [ounces] (onions); chopped
14
15
4 [ounces] shallow fried (beef); minced
16
17
1/2 [cunce] lean prepared (bacon)
18
19
1/3
|cup]
red [wine]
http://ss...uuuunttttttttettet wtw.t//eeetttteee...u...aaa///// for documentation.
A2

---

20
21
150 [gram] raw (carrots); thinly sliced
22
23
2/3
lounce] concentrated (tomato puree]
24
25
[ounces] red (sweet pepper); cut julienns
[ounce] [parmesan] cheese
29 Instructions:
30
31 Add the 0cil0 to a large saucepan, heat to <300 E>, and saute the ooniens@.
32
33 After 12 minutesl, add the agarlice. Keep medium um to high heat, and don
't stir.
34
35 After minutes| more, add the @beef0.
Fry the @baconé in separate pan, en high heat. Remove liquified fat when done.
39 Boil @pastal in a medium pan, until al dente (-18 minutes]] )), Drain when done.
40
41 Once the Obeefe is done, add the @carrotse, 8sweet pepperê and @tomato puree0.
42
43 Slowly add the @winel well, to not Tower the temperature. Let it simmer (but not boil) for 15-10 minutes
Given the protocol as the input of our framework, the resulting DSL program is as follows.
1 add (slot - "cil", target = "large saucepan", container = plate_l, enit mixture_1);
2
3 heat (target = mixture_l, temperature 300F, container = plate_l, postcon st)p
4
5 saute (target "onions", container plate duration 2mins)
6
7
add (slot "garlic", target mixture_l, container plate_l, emit mixture_2);
9
heat (target mixture_2, temperature 325r, container plate_l, duration 2mins);
10
11 add (slot = "beef", target mixture_2, container plate_l, emit mixture_3);
12
13 heat (target mixture_2, temperature 325F, container = plate_1, postcond check_dome (target "beef"));
A3

---

14
15 fry (target "bacon", temperature 350F, container pan_1, postcond renove (target = "liquified f
16
boil (target "pasta", temperature 212F, container pan_2, duration 8mi ns, postccond dr)in());
add (precond check done (target "beef"), slot pepper", "tomato puree"!, target mixture_3, container plate_l, emit mixture_4);
= ("carrots", "sweet
20
21 add (slot = "wine", target mixture_A, 4,, container plate_l, pace inL/s)
22
23 simmer (target mi mixture_4, temperature 211F, duration 7.5mins)
In this example, observe that the NL-based recipe possesses ambiguities and omissions. Our translation framework addresses these challenges by structuring the recipe on the syntax level, com- pleting the latent information the semantics level, and linking the programs with necessary re- sources, such as the usage of plates, on the execution level. Due to the modularity of DSLs, although Corel's 's distribution syntactic and semantic features differs significantly from those of DSLs used for representing experimental protocols, our translator can generalize to this new target domain through the structure of rules, namely rule-based generalization (Shi et al. 20236).
A.5 The motivations behind this work
In this work, we study the problem of translating experimental protocols designed for human ex- perimenters into formats suitable for machine execution. Our primary motivation to bridge the existing gap between machine learning algorithms in the field of AI for science, such as molecular design, and the grounded experimental verification facilitated by self-driving laboratories. Con- ventional workflows for setting self-driving laboratories and conducting physical experiments necessitate deep integration with domain experts, significantly impeding the progress of machine learning researchers verifying and iterating their findings. Consequently, our framework aims to provide an infrastructure that enables these researchers to advance their machine learning algorithms and seamlessly validate their findings, thereby closing the loop of automatic scientific discovery.
To meet the requirements of such infrastructure, we conduct a systematic study to identify existing gaps in protocol translation between human experimenters and automatic translators in self-driving laboratories. From the study, we derive design principles that emulate human cognitive processes involved in protocol translation. Under the guidance of these design principles, we develop the three-stage framework that integrates cognitive insights from human experts with approaches from program synthesis, automaton construction, and counterfactual analysis. On the syntax level, we synthesize the operation dependence graph to transform NL-based protocols into structured repre- sentations, thereby making explicit the operation-condition mappings and the control flows. On the semantics level, we analyze the reagent flow graph to reconstruct the complete lifecycles of intermediate products, addressing the latent, missing, omitted properties and values. On the ecution level. we contextualize both the operation dependence graph and the reagent flow graph within spatial and temporal dynamics, resulting in the protocol dependence graph. This graph con- ducts counterfactual reasoning to detect potential conflicts or shortages of execution resources and to identify inappropriate combinations of operations in execution sequences.
B
Ethics statement
B.1
Human participants
The meta-evaluation included in this work has been approved by the Institutional Review Board (IRB) of Peking University. We have been committed to upholding the highest ethical standards in
A4

---

conducting this study and ensuring the protection of the rights and welfare of all participants. We paid the domain experts a wage of $22.5/h, which is significantly higher than the standard wage.
We have obtained informed consent from all participants, including clear and comprehensive infor- mation about the purpose of the study, the procedures involved, the risks and benefits, and the right to withdraw at any time without penalty. Participants were also assured of the confidentiality of their information. Any personal data collected (including name, age, and gender) was handled in accordance with applicable laws and regulations.
B.2 Corpora collection
We carefully ensure that all experimental protocols incorporated into our corpora strictly adhere to open access policies, governed by the Creative Commons license. This approach guarantees full compliance with copyright and intellectual property laws, eliminating any potential infringement or unauthorized use of protected materials. By exclusively utilizing resources that are freely available and legally distributable, we uphold the highest standards of ethical conduct in research, fostering environment of transparency and respect for the intellectual property rights of others. This com- mitment ensures that our work not only advances the frontiers of knowledge but does so in manner that is both legally sound and ethically responsible.
C Implementation details
C.1 Details of pre-processing
We employ the SpaCy Dependency Parser to analyze the syntactic structure of protocol which allows for the extraction of verbs and the identification of associated objects and modifiers (Honnibal and Johnson 2015). After parsing, these verbs aligned with corresponding E Top in the DSLs by maximizing the cosine similarity between their word2vec representations utilize advanced few-shot Named Entity
g operational actions
and those of the DSL operations. Furthermore, Recognition (NER) algorithm, based on large language models, to accurately identify and classify entities e, within the text (Xie al. 2024). The rationale of integrating L.MSs with classical parsing techniques lies leveraging the advanced natural language processing capabilities of L.MMs while mitigating their inherent uncertainties.
The prompt for NER is as follows.
1 Given entity label set: (label_seth.
2 Please name the entities in the given text. Based en the given entity
1abe1 set, provide answer in the following JSON format: [["Eatity
Name"; "Entity Label";l. If there is no entity in the text, return the following empty list: [].
3 Please note that entities have already been annotated with [], need extract and analyze other entities.
4 (cases)
5 Text: (query)
6 Answer:
C.2 Details of reagent flow analysis
We extract reagents from the natural language descriptions of protocols using two utility functions, KILLS and DEFINES. KILLS identifies the reagent consumed in operation, while DEFINES iden- tifies the reagent introduced in an operation. Due to the potential for a single chemical substance to have multiple names among other factors, it is impractical to rely solely string matching to determine if a reagent killed in a given operation. Instead, we employ a method based on prompt engineering with LLMs for this analysis.
The following prompt is used to analyze whether the input of the current operation is the output of previous operation:
A5

---

1 This instruction descrises step in experimental process, which includes one action, multiple paramaters, and one output.
2 Please help analyze the output of this instruction. will provide list of potential outputs. You need to assist in determining which of these outputs is most suitable for this instruction.
3 Note that you must choose one output from the 1ist. Please output only string without any explanation.
5 [Examples]
6 Instruction: ("action"; "add", "reagent"; ["glycoblue"), "output": "")
7 Potential output 1ist: "RNA", "mRNA"
8 Output: "RNA"
9
10 Instric ("action"; "add", "ccncentration": [*1:10 volume 5 M Nac1"),
"output": "")
11 Potential output list: "a μMACS cclumn", "solution"
12 Output: "sclution"
Instruction: ("action"; "heat", "reagent"; ["limestone"), "output"; "*)
Potential output list: "water", "NacI" Output:
18 [Question]
19 Instruction: (Instruction]
20 Potential output list: {Input}
21 Output:
Additionally, this prompt is used to determine if reagents in the current memory M(T) are killed by the current operation:
This instruction describes a step in an experimental process, which and one output.
includes one action, multiple parametore-includee various reagents-
2 Please help analyze the missing reagents of this instruction. I will provide list of potential reagents. You need to help me analyze which of these reagents might be the ones omitted from the current instruction.
3 Please note how many reagent parameters missing from the current instruction. It is possible that some reagent paramoters cannot be completed with the list provided. Please output only separated list of strings without any explanation.
[Examples]
Instruction: ("action": "add", "reagent": [""], "output"; "")
7 Potential reagent list: "RNA", "glycoblue"
8 Reagents: "glycoblue"
9
10 ruction:: ("action": "add", "concentration": [*1:10 volume"), "reagent ["", ""], "output": "*]
11 Potential reagent list: "μMACS", "selution", "Nacl"
12 Reagents: "NaCI", "дMACS"
13
A6

---

14 Instruction: ("action"; "use", "reagent": ["BamHI", "XhoI", ""], "device
["PCR amplification"). "output": "")
15 Potential reagent list: "agar", "food"
16 Reagents:
17
[Question]
Instructien: (Instruction)
Potential reagent list: (Memory)
Reagents:
In the process of synthesizing operation dependencies on the syntax level, we implement the DE- FINES function. This involves pattern matching after pre-processing to accurately define reagents in each operation.
C.3 Cost of the implementation
The computational cost of our algorithm primarily arises from the expenses associated with API calls to LLMs. We selected OpenAl's 3 5-turbo-0125 model for experiments. Across 75 test protocols, we executed 1816 queries to achieve syntax-level translation, resulting in structured protocols. At the semantic level, conducted 4062 queries for completion tasks (in- cluding translating protocols retrieved from training dataset). During these experiments, the cost model charged US$0.50 per million tokens for inputs and US$1.50 per million tokens for outputs. Consequently, our expenditures were approximately US$10. Additionally, we utilized OpenAl's text embeddiig ng ada 002 model to embed the training dataset and build vector database, which incurred cost of about USS7.
D
The testing set
D.1 Collection
The real experiments for the testing set are retrieved from open-sourced websites run by top-tier pub- lishers, including Nature's 's Protocolexchange" Cell's Star Star-protocol Bio-protoco Wiley's 's Cur- rent Protocols and J
D.2 Showcases
1 [Protocol Biochemistry]
2 Preparation of lysates
3 1. Harvest approximately 10*7 cells by centri fugation at 2000 RPM for min. Aspirate media resuspend cell pellet with mL of ice- cold P2S and transfer to mL centrifuge tube. Microcentrifuge 2000 RPM for min 4 Pc.
4 2. Aspirate PBS, and then add Hypotenic Buffer (supplemented with 1$ Triton X-100, to disrupt membrane and cytoskeletonn-oour MEKK1 fractions).
5 3. Cell lysates are homogeni zed by passing through 22 22-gauge needles, and tubes are put on ice for 15 min to complete the lysis. Crude extracts are then centrifuged at 2500 REM for 5 min. Supernatants are transferred to fresh centrifuge tubes, and cold 5 M NaC1 is added to
htte:://ppccclleshhhee.errcccpep httess//ttar-ppooccool...llll com/
http:://iio-prootcol..ooo//
https://currennprottoooo
inel aibrarrrr... www.y.com
https://www.jove.com
A7

---

RT. each sample to make salt concentration of between 0.7-1.0 M to disrupt protein-protein interactions.
6 4. Spin the crude extracts by ultracentrifugatten properly pellet residual insoluble proteins from the extract. Transfer supernatants into fresh centrifuge tubes.
at 55000 RPM to
7 Immunoprecipitation
5. Rinse Protein A beads Hypotenic Buffer and place on ice until ready for use.
9 6. Take a volume of cell lysates (prepared as described above), and dilute with Hypotonic Buffer to 250-500 mM salt to enable protein- protein interactions.
10 7. Add μg of preclearing antibedy to the diluted lysate (e.g., anti-Mye or anti-p65), vortex, add 50 μl of Protein A beads, and rock for 45 min.
11 8. Touchspin samples, and transfer supernatant to fresh tube.
12 9. Add ng of polyclonal anti-MEkKI the lysates, and rock for 1 h. After this period, add 50 μL of Protein beads and rock tubes at for 1 h.
10. Touchspin beads, wash beads with hypotonic buffer (supplemented with Nac1 to concentration of 300 mX), vortex, and rock for min. In total, 3-5 washes of the beads are perforedd.
11. Pinally, wash once with Hypotonic Huffer, and resuspend Kinase Assay Buffer. Purified MEXK1 may be stored by snap-freezing in liquid nitrogen and long-term storage at -80 Cc Kinase assay Following preparation of MEKK1 immunoprecipitates (as above), incubate with of JNKK1 (K131M) along with 5 pci of ATF in Kinase Assay Buffer for 30 min at 30 "c."
15
16 [Protocol 2 Genetics]
17 1. Note that everything is in DEPC water. Inoculate W303a colls expressing different TORI-BR variants in 2 mL medium overnight.
2. Subculture the cells starting from 0D600=0.1 in 10 mL media, shake vigorously at 30 "c, 300 ReM for around 4-6 until 30500-0.4-0.5,
3. Collect the cells by spinning down without freezing on ice. Discard supernatant.
20 4. Re-suspend cells with ml, water and transfer to a 1.5 eppendorf tube, quickly spin down at 3, 000 x g for 15 sec.
21 5. Re-suspend cell pellet in 400 μL of AE buffer at roon temperature.
22 6. Add 40 μL 108 SDS (final around 18) and vortex briefly at room temperature (RT).
23 7. Inmediately add 500 μn hot phenel/AE (put in 65 "c for 10 min before
use), vortex vigorously for min. Incubate at 65 "c for 5 min. Briefly vortex every 30 sec.
24 8. Immediately freeze by dumping into liquid nitrogen. wait thaw at (put in 30 " to thaw may crack the tube).
25 9. Centrifuge for min
standard laboratory microfuge 20, 000 x g
10. Transfer around 400 μz supernatant to new eppendorf tube. Recyele the lower phenol fraction carefully following the chemical safety protocel your laboratory.
A8

---

27 11. Add equal volume (400 pL) phencl: CHC13/AB-Na. Vortex vigerously for
1 min at RT.
28 12. Spin down 20, 000 for 5 min in standard laboratory microfuge.
29 13. Transier supernatant (around 350 μL) to fresh 1.5 mL eppendort tube
Add cHc13: isoamyl alcohol (24:1). Vortex vigorously for min RT.
14. Transfer aqueous supernatant a fresh mL microfuge tube. white cloudy precipitate is observed between the aqueous phase and organic phase, repeat steps 17-18.
31 15. Add 1/10 volume of M NaOAc (рH 5) and vortex vigorously. Add 2.5 volumes of ethanol. Vortex again.
32 16. Place at -20 "C for at least 30 min.
33 17. Spin down in the microfuge at 200000 000
15 min at cc RNA pellet
is usually visible.
34 18. Add ice-cold 75% EROh, place 4 " for around 10 min. Vortex and spin down microfuge 20, 000 g, 15 min 4 "c. Discard supernatant. Suck out the liquid droplets in the tube. The white RNA pellet will turn clear when it dries out. Add 30-50 μl ddl20 (DEPC) immediately after it becomes clear. Do not 1et the RNA over-dry, whica will make disticult to dissolve.
RNA pellet is over-dry, dissolve RNA at "C for 30 min. Store RMAS at -80 " for more than months."
35
36 [Protocol Biomedical]
37 1. Passage through a 45 pm filter. Add 100 pl/well of 100 pg/ml salmon
sperm DNA to 96-weil Microtest assay plate.
38 2. Wrap the plate with plastic wrap and incubate at 4 "c evernight.
39 3. Discard the coating antibody sclution and wash the plate with ix PBS-
Tween 6 times.
40 4. Dry the plate and add 100 of blocking solution per well to the plate.
41 5. Incubate the plate room temperature (RT) for 1.5 h.
6. Discard the blocking solution and wash the plate with PBS-Tween 5 times.
7. Dry the plate and keep
C for later use.
8. Harvest the spleen and create single-cell suspension by gently
smashing spleen pieces with the frosted surface of a pair of microscope slides in 5 mL of DNEX.
45 9. Transfer the cells into 50 mL conical tubes and spin down the cells at
300 RCF for 5 min at 4 cc.
46 10. Discard the supernatant with aspiration without disturbing the pellet
47 11. Re-suspend the cells with ml. of 0.17 M ammon um chloride and keep the cells on ice for min.
48 12. Ada 15 mL DNEM to the cells and spin 300 RCE for min at Pc.
49 13. Discard the supernatant and ru-susdend the cells with 20 mL DMEM and count the cells.
14. Re-suspend x 10^7 cells serial dilution (a total of dilutions) with 108 DMEM.
mL of 10% DMEM and make a three-fold
15. Add 50 pl//well of the serial dilutions on the DNA-coated plate and centrifuge at 300 RCF for 5 min at c.
A9

---

52 16. Incubate the cells at 30 "c for 2 h in cell-culture incubator with 6 CO2.
53 17. Add 50 pL/well of biotin-conjugated anti-IgM anti-IgG (1:350 in
108 DMEM) to the cells.
54 18. Centrifuge the cells 300 RCE for min at "C and incubate the cell-culture incubator with 6$ 19. Discard the cells and wash the plates 10 times with 10x PBS-Tween
cells overnight
20. Dry the plates and add 50 μL streptavidin alkaline phosphatase BSA/PBS) to the plate.
(1:1, 00: 000
57 21. Incubate the plate at RT for h and wash the plate 10 times with 10x Pas-Tween 20.
58 22. Dry the plate and add 50 pL/well of 1 mg/mL BCIP in AMP buffer to develop the plate.
59 23. when the spots are clearly visible under dissecting microscope, stop the development by discarding the BCTP solution and rinsing the plate with tap water thoroughly.
60 24. Spots be counted using dissecting microscope using ELISpot reader."
D.3 Instruction for human experts
1 Instruction for Human Study Pretecol Translation and Parameter Completion
2 [objective]
3 The purpose of this study is evaluate the accuracy and completeness translating natural language laboratory protocols into a structured JSON representation and assess the manual completion of missing parameters within these protocols.
4 [Experimental Tasks]
5 Participants in this study will perform two main tasks:
6 1. Translation of Natural Language Protocols to JSON-structured Repzesentation
7 2. Manual Parameter Completion in 18ON-Structured Protocols
8 [Task 1: Translation of Natural Language Protocols to USON-Structured
Representation]
9 [Descripticn]
10 Participants will be provided with set of 1aboratory protocols trit in natural language. The task is to translate each protocol into JSON-structured format. This involves accurately mapping the operations, input reagents, and conditions specified in the natural language description to precise
schema.
[Proceaure]
1. Read the provided natural language protecol carefully.
2. Identify and extract the key elements of the protocol, including: Operations (e.g., dissolve, mix, /Input reagents (e.g., sodium chloride, distilled water) /Conditions (e.g., temperature, time, concentration)
14 3. Construct a JSON representation that clearly reflects the structure and content of the protocol. Ensure that each element is correctly mapped to its corresponding key and value pairs.
15 Example]
A10

---

16 Extract total RNA from at least 2 10^6 cells using TRIZCI reagent.
17 ("action": "extract", "output": "tctal RNA", "reagent"; ["TRI20L reagent
"I, "volume": [" least 2 10 cells"), "container"; [""]]
18 [Manual Parameter Completion in JSON-Structured Protocols]
19 [Descripticn]
20 Participants will receive
of suuu-strd protocels with certain parameters intentionally left incomplete. The task is to manually complete these parameters based on domain knowledge and logical inference.
21 [Procedure]
22 1. Review the provided JSON-structured protocol.
23 2. Identify any missing or incomplete parameters.
24 3. Use your expertise to infer the missing information. This may ude: Estimating reasonablc values for missing quantitics or conditions; 25 knsuring consistency and coherence within the protecol.
26 4. Complete the JSON structure with the inferred parameters, maintaining accuracy and logical consistency.
27 [Example]
("action;; "apply", "output"; "known DHB cluster signals", "device";
(aaction: "apply", "output": "known DEB cluster signals", "device":
[*<<aa pneumatic sprayer system>>>"]}
A11

---

E
Case studies
E.1
Contributions of the components in our translator
We provide series of case studies to illustrate the distinctions between the behaviors of the compo- nents within our proposed framework and those of the baselines qualitatively Tab. Al
Table AI: Distinctions between the behaviors three components within our proposed framework
Original Text
Stage 1
Feature
Stage 2
Feature
Stage 3
Feature
Kill most the
F:. RATMEEATE:
"K: taminating
[[con-
51. IMINATR::: [[ee:
Latent se- mantics unknown unknowns (forcc)
ELIMINATE:
Reg:
contaminating
[Reg:
[Reg:
spore
spores that have
contami amtinnin
spores], [spore mix- ture]]
contamiina www.antiiiiin.ommmmmmmm ing
aminnatinng
germinated. Centrifuge SPIN: [Regs
spores] J;
spores] Ji
aporea]] ]]
No specific
SPIN: [Reg: spore mixture),
space
volume
spore mixture <MASK> for 5 min.
spaze mixtura),
mixture;
provided
SPIN: [[Rag:
[Time:
[Time:
apore
Sminl, [Force: MASK] ]
Smin], [Ferce:
mixturel,
[Time:
1, 200g]
Sminl,
[Force:
1,200g11
Add pre-hybr soln directly to bybridiza- tion reaction get hyhrid molecule. Incubate for <MASK>.
ADD: [[ee:: pre hybz salnl, [Reg: hybridionn mation react oon
[[pre- hybe soin),
ADD: [:Reg:
Larent semanties unknowns (time)
[[Regs
Reg:
pre hybz [Reg: hybridi zat satioo react oo
pre hybr
pre hyb:
Ibybridization reaction]]
soinl, [Keg: aygridizatio
soln. by-
bridization
hybrid
reaction]]
reaction;
hybrid
molccule
hybrid ma eoc e;
hybrid
No specific
ma eeuu le INCUDATE:
molecule le;
volume
INCUDACE:
INCUBATE:
provided
[[Time:
[[Time:
[ [Reg:
MASK]
10min]]
hybrid
molecule),
[Timer
10min]]
prs-
CONPTRM:
[PCR]]
CONFTRM:
Reg:
iive colonies
[[Dvvie::
positive
[[Device:
mantics of unknown
[[Devioe:
RNAs;
by PCR. Take
[CR]] >>
colonics:
2CR]] >>
PCR], [Reg:
specific
fluorescence
poalt
[[positive
positivg
unknowns (dovice)
RMAa]]
volume
images under
colonies:
colonics]] >>
colonies:
positive
provided
<MASK>.
TAKE:
fluorescence
TAKE:
colenies;
[Cevine:
images
[Cevice:
TAKE;
MASK]]
crosoope]]
[aDevioss
Cluozescence
microscope]]
inages
Fluorescence
images
fluorescence
images
A12

---

E.2 Running cases of our translator handling specific challenges
Ensuring the safety and correctness of translated protocols an exceptionallyy challenging task. Sev- eral factors contribute to these challenges. including accurately mapping operations to their corre- sponding configuration parameters (Tab. A2], precisely parsing control flows from natural language (Tbb. A3), completing latent semantics with domain-specific knowledge (Tab. A4). inferring miss- ing or omitted key information (Tab. A5), tracking resource capacities Tab.. A6), and verifying the
safety of run-time execution of experiments (Tab. A7). Consequently, we have made specific efforts response to these challenges, resulting in our design of translator. Here we provide several running examples to demonstrate our translator's capability on handling the challenging factors respectively.
Table A2: Rumning cases the syntax level regarding operation-condition mapping
Original Text
Syntax Level
Action
Conditions
Spin media at x x for 10 min (optional), pre-x for 10 min, filter with 22 um PES membrane,
SPIN: [[Spoad: 500-1, 030 x s.r. [Time: min]] tiltered med'
Spin, Filter,
Speed: 500-1,000 Time: min, De- vice: 0.22 um PES membrane, Tempera- ture: SPCC
FTTTER:
[[Device: 0.22 um PES membrane]]
Freeze
at -80CC.
filtered media.
FREEZE: ([Temperature: -80%c]] -> frozen media
Thaw ml supernatant add ml XBP buffer.
[[volume: 4 mll, 'Reagent:
Thaw,
Volume: ml. Tem- perature: On ice
aupernatant shawed aupe aupe-natant, AnD: [Reagent: buffer], [Volume: m1]] >> sample/XBB XBP mix
Add
Add sample/XBP mix olasy masi spin column, cem- trifuge 1-3 min 500 dis-
ADD: [[Resgen:: semple/XBP mixl, [Container: spin column;] -> flox-through, CANTRTPUCR: [Speed:
Add,
Container: Spin col-
Cen-
umm, Speed: 500
500
trifuge,
Time: 1-3 min
card Dow low-through.
x gl, [rime: 1-3 min]] -> flox-through,
Discard
DISCARD: ([Reagent: flow-thrasgh]]
Add XWP spin col-
ADD: [[Resgent: XPP], [Volume: 10
Add.
Volume: 10 ml.
contrifuge 5 at 5,000 x transfer column to fresh collection tube.
mi]] gl, (Time: min), [Container: spin columa]1
CENTRIPUGE: [[3peed:
Speed:
5,000
uifuge.
Time: 5 min. Con-
TRANSFER: (icontainer: fresh collection subel]
Transfer
Spin
Fresh collection tube
Add 700 ul. Qiazal umn, contrifuge 5 min at 5,000 spin PLG tubes 16,000
col-
ADD: [[Reagents Diazol], (Volume, 700
Add, Cen- trifuge. Spin
Volume: 700 ul.,
ub]] -> CENTRIPUGE; [[Speed: 5,000 x gl, (Time: min]] 16,000 gl, [rime: 30 s], [Container: tubas]]
Speed:
5,000 x
[[Speed:
Time:
5 min, Speed:
16,000 Time: 30
Container: Spin
column, PLG tubes
Add flow-through PLG tube, vortex s. incubato 5 min at RT:
[[Reggent: Flox-through,, [Container: 21G sube]] >
Add,
Container: PLG rube,
VORTEX:
Vortex,
Time: Time:
INCOBATE: [[lime: min), (Temparature: RT];
[[lime: =11
Incubate
Temperature:
RT
Add 90 ul. chloroform.
ADD: [[volume: 90 ub], "Reagent:
Add
Volume: ul.
chloroform]] ->
Shake vigoruusly for 15 in- cubate 2-3 min at RT.
SHAME: [[Time: 15 8]1 -> TNCURATE:
Shake, Incubate
Time: 15 Time: 2- min, Temperature: RI
[[Time: 2.3 minl, (Tempersture: RT]
Centrifuge
CENTRIFUGE: [[Speed: 12,000 sl, [Time: 15 min]] -> upper aquaoua phase, TRANSFER:
Centrifuge, Tranfer
Speed: 12,000 g-
transfer upper aqueous phase tube.
Time: 15 min, Con-
[[Container: new tube]] upper aqueous phase
Add 2 volumes ethanol, mix.
ADD: [[Volume: volumes], [ReagenL: 100 ethanoll] -> ethanol mixture, mixture]]
Add, Mix
Volume: volumes
MIX:
[[eagent::
A13

---

Table A3: Running the syntax level regarding operation control flows
Original Text
Syntax Level
Action
Control Flows
Centrifuge the cell suspension
CENTRTFUGE: [[Force: 200 x g.r [Tamperatura:
Centriuue
Linear
200 g for 5 min.
temperature
room Lemperacure], (Times nin]] cell pellet
Remove the supernatant.
REMOVE: [[Ousput: supeznatant]]
Remove
Lincar
Suspend the cell pellet with ml ACK lysing bufler for I min deplete
SUSFFNN: [[Voluma: 2 mi;, [Reagent: ACX lysing beffer], (Time: min;] -> depleted suspension.
Suspend
Linear
If red blood cells are not com- pletely depleted, repeat the ACK lysing butter step until they
REPENT: [[Reggen:: ACX Lysing buffer), [Conditions
Repeat
Non-linear
blood celle not completely depleted]] ->
Filter the suspension through nylon siminer.
FILTER: [[Containe:: Itlteeed cell auapens'on
un nylea atrainex]]
Filter
Linear
Wash
WASH: [CContainez: strainerl, [Reagent:
DPBS for 5 min.
[PeS], [volume: 2 =l], "Time: 5 min]] >>
Wash the cell pellet with DPBS with ng/ml murine M-CSF 100 mm dish.
[[seagent: 1x with ng/ml murine X-CSF), [Container: 10C am Petri dish]]
Wash
Linear
Suspend 15 complete DMEM medium.
SUS2 END: [[Volume: 15 m_l, [Reagent: complete DMEN medium]] -> cell suspension in LMEM
Suspend
Lincar
Incubate at 37 "C, CO2.
INCURATE: [[Temporature: 37 sc], |nnrt ronment: [02]] -> incubsted
Incubate
Linear
After 3 days, replace half the medium with fresh medium], [Times plete DMEM medium.
REPLACE: [Reagent: freah complete DMEM days;]
Replace
Linear
Repeat this step every days.
REPENT: [[Reagen:: fresh complete DMEM madium], (Frequency: every 2 days]] +>
Repeat
Non-lincar
Table A4: Running cases on semantics level regarding known unknowns
Original Text
Semantic Level
Known Un- knowns
Transfer the sample (plasma, cell suspension) into a glass centrifuge vial.
TRANSFER: [Reageet: the sample Iplasma,
call suspension) [container: glass centri fuge visl]] >>
Adjust the volume to ml with PBS.
MORTPY: [ heparinized blood.1 ml
"I ml"
medium], IVolumer esl mi*>]] ->
50-200 plasma was taken from beparinized
TAKE: [Reagent:: heparinized blood.1 m1
blood.1 medium.
medium]
Plasma was directly taken from cell culture.
TAKE: [loupuu:: plasma sample]]
Add 10 of the internal standard (10 aM CI7-
ADD: [[Regeett: 18.5 HC1], [Volume: <<300
"300 ul"
McOIT). Add an example, extraction from plasma
s1>>]]
3H0w: [[Output: step A7], [Reagent: a
sample shown step A7.
plasma samplell ->
CHCI3-phase extracted by directly pipet- ting through the upper aqueous phase.
I[output: cHC13), [Censainer: the upper aqueous phase], [Saagent: stop AY]] ->
Add this CHClB-phase to the transferred CHCI3-phase step
ADD: [[Reagent: this CHC13-phase]] ->
Vacuum-dry CHCB in the vacuum rotator 60 "C' For 45 min.
DRY: [[Reagett: ecthe CHIC1=*)],
"the
[Jemperature: 60 sel, [lime: e<15 mins>]]
CHCI3".
min"
Alteratively, the samples can he dried under nitrogen gas flow.
DRY: [[Reagant: samples), [Tima: 1-20 min])
Re-oquilibrate with solution A.
TBUALIERATE: [Ooupptt: S1P], raancentration solution]]
analyzed m/s 264 miz. For quantitaive analysis. standard curve with SIP amounts pmol to 100 the internal standard genemted.
mass transition
EXAMINF: [[autput: quantitative analygis),
[Keagent: slP]] ->
A14

---

Table A5: Running the semantics level regarding unknown unknowns
Original Text
Semantic Level
Unknown
Un-
knowus
Harvest upproximately Ix10' cells centrifu- gation for min.
HARVEST:
[neviia: cantii ion)
"ec2000 RPMs>
[Time:
minl,
(Force: e<2000 RPMs>]]
Cell lysates are homogeniaed by passing through 22 gauge needles.
HOMOGENI28: [[esgent: cell lysates]]
Tubes pat on lysis. Crude extracts then centrifuged.
complete
INCUBATE: [CContaine:: tubes], [Time: 15 min], [Tamperatures en [ce]] >>
CENTRIFUGE: [[Force: #<2500 RP%>>], [Time: <<5 mins>]] ->
"ec5 mine>'
Supernatants are transferred to fresh centrifuge tubes.
TRANSFER: [[onnainer:: fresh contrifuge tubea]] ->
Cold M NaCl is added to each sample to make
Ann: [Container: each ааmplе], NaCl], [Cencentration:
concentration of between 0.7 disrupt protein- protein interactions.
[Reagen:: 0.7 - 1.0 M.] -> semple with NaCl
Spin the crude extracts by altracentifugation properly pellet residuml insoluble proteins from the extract.
SPIN: [Device: aitracentrifggaiion)
"e<55000 RPMs>
[Foree: #<55000 KPE>>], [Keagent:
reaidual insoluble prateina]] Hypotonic Buffer
Transfer supematants into fresh tubes.
centrifuge
TRANSFER: [[Reagenl: superastants),
[Centainer: fresh centråfuge tubes]]
Rinse Protein A beads in Hypotonic Bufter un-
AINSE: [[Reagent: Hypotonic Buffer]] >>
ready
use
Take a volume cell lysates (prepared as de- scrihed ahove).
TAKE: [[Volume: call lysates)] +>
Hypotonic Buffer
Hypotonic salt to enable protein-prosein interactions.
DITUTR: [[Reggent: Hypotontc Bu*fer]]
antibody
Add ug preclearing antibody the diluted lysate (e.g. anti), vortex, add 50 uL of Protein beads.
[Roagant: antibody, Bratain
beads]] -> polycional anti-MERK1
2 ug polyclonal ant-MEEKI to lysates, add 50 ul of Protein A beads at 'C for h.
ADD: [[Reaget:: polyclonal enti MERKI], [Centainer: the lysates], [Temperature: "01, [Time: 1 h]] >>
Touchspin beads, wash beads with hypotonic buffer (supplemented NaCI).
WASE: [[Reagent: hypatonic buffer), [Concentration one
"ec300 mMa>?
m[a2]] ->
In total, 3 5 washes of the heads are per- formed.
РERFORM: [[Reagent: hypotonto butter], 5]]
[Frequency:
Finally, wash once with Hypotonic Buffer. Purificd MEKKI may be stored by snap- freezing liquid nitrogen.
WAGE: [[Reagent: Hygotonic Buffer]]
STORE: [[Kethod: snap freezing),
[Reagent: liquid nitrogen]]
Following preparation MEKKI immune- precipitates (as above), incubate with ug INKKIKKI3IN along with uCi ly- PIATP for 30
INCUBATE: [[Reagent: JAKK (K131M),
"ecKinase Bufferso", "C'""
Assay
[y=""]ATT]; [container: esKinase Assay Sufferw>], [Terperature: <<30 *Cw>], [Time: min]]
"c<30
A15

---

Table A6: Running cases on the execution level regarding capacity of resources
Original Text
Execution Level
Reagent Flow Graph
Prepare annealing solution of 50 uM RNA/DNA oligos with 50 mM DNase/RNase water, aliquot 50 ul in PCR tube.
PREPARS: [[output: annezling solution (Concentrat lons 50 UM RMA/DNA ol oligos, NcCl), [Reagent: DNaae/ maase//aase-foo
DauseRNasee free water (50 uly, RNA/DNA oligos (50 uM), NaCI mM); out: annealing so- lution (50
water], [Volumes 50 u.J, [Containers PCR tube]]
Dissolve inhibitor compound mM. needed, pre-
DISSOLVE: [[output: inhibitor compound solutionl, [Reagent: inhibitor compound, [MSC]] ->
in: inhibitor compound. DMSO: inhibiter compound solution (volume depends on dilution)
pare serial dilutions in Milli Q wa-
Add water (20 ul in blanks, 10 controls) 96-well plate.
ADD: [1Output: water in we wells), [ReageaL: waterl, [Container: platel] ->
in: water (20 for blanks, 10 for controls); plate (20 ut in blanks. 10 ul in con- trols)
96-well
Add 80 ul RT reaction mix (1.25x).
ADD: [[outpat: S2 reaction mix in 1ells], [Volumes 80 J.
in: RT reaction mix (80 ul); out: RT reaction mix 96-well plate (80 ul)
Add 10 ul inhibitor dilution samples, each well.
ADD: [[outpat: samples with inhibitor], [Volume: 10 ull, [Reagent: inhibitor dilution]]
in: inhibitor dilution (10 uly: samples with inhibitor ul)
Stop (0.5 M. pH 8.0).
STOP: [[cusput: stopped reaction),
ul); out: stopped re- action with EDTA (50 ul)
[Reagent: 3DTAJ, [Volumes 50 ]i]] +>
Quantily reaction with Victor at 490/528 nm, report inhibitor val- wes as percentage control.
QuANtisY: [[ouppu:: quantified
reaction; quantilied reac- tion at 490/528 am
reaction], [Device: Victor 3]]
Subtract ples.
sam-
SUBTRACT: [[Output: corrected samplea],
blank value, samples, out: rected sample valaas
[Reagent: blank value]]
Calculate IC50 value the
CALCULATE: [[Output: IC50 valuel, [seagent: signal]] ->
signal, IC50 value
ventration reporting 50 reduction signal compared control.
Table A7: Running cases on execution level regarding safety of operations
Original Text
Exeeution Level
Reagent Flow Graph
Replace medium after 12 hours (Day 2).
REPLAUS: [[Output: medium replaced), [Contsiner: medium]
in: old medium: out: new medium
Digest mFSCs with 0.05 trypsin, prepare for FACS 96-well plates (Day 10).
DTGFST: [[Dutput: mRSCa], (Reagents trypsial, [Containess well plates]] ->
in: mFSCs. 0.05 trypsin: out: digested mESCs (ensure trypsin is neutralized to avoid over. digestion)
Remove single colonies from 96- well plates well plates.
REMOVE: [[Uatput: single colonies), [Containers 96-wall ates, 24-vell
in: single colonies; out: single colonies well plates
plates]] ->
Confirm positive colonies by tran- sient transfection sgRNAs anal- ysis (SPH primers) (Day (4-15), single plates by FACS.
CONF IRM: [[Output: positive colonies),
in: single colonies, SPH primers; out: positive colonies
[Reagent: SPIl primers.] ->
SOR1: [[Cusput: single celis), [Device:
single cells: sorted gle cells in well plates (en- sure proper calibration FACS avoid sorting errars)
FACS], (Containers 96-wel1 plates]] >>
Confirm insertion by PCR (Day 18).
[[Output: insertion
single cells; confirmed settion
confirmed]] >>
Confirm positive colonies PCR (Day 22).
CoNe [[output: positive cclonies]
single colonies; positive colonies
Measure fluorescent intensity of colonies by FACS, flaores- eence images under confocal mi-
TAKF: [[outpot: floorescence images),
in: colonies; out: flaorescence
[Device: confocal microscopel,
images (handle samples avoid photobleaching)
[Containes: colonies]: >>
croscoge (Day 27).
A16

---

E.3 Types of errors made by our translator
We present detailed analysis of the errors made by our proposed automatic translator compared to human experts. We discuss the potential improvements of the translator accordingly.
Distinctions on the syntax level Difference between the translation results of our system and those of experts subtle, with the biggest difference being in the analysis of long sentences in natural language. For human experts, natural and easy to analyze the parameters of events/actions multiple actions long sentences, while for our approach, there are sometimes problems with the correspondence between action and parameter, which need be improved in future work.
This series of examples Tab. A8 demonstrates the superior performance of our system at the syntax level when processing relatively short sentences.
Table A8: Distinctions the syntax level when processing relatively short sentences
Original Text
Syntax Level
Remarks
Transfer 200 ul of serum into 1.5 microfuge tube.
TRANSPER; :[Output: butferl, [Reagent: microfuge
sezum], [Container:
tubel, [volume: 200 will ->
Add 200 of buffer XBP.
ADD: [[outpat: the sample/XsP mixture], [Reagent: buffer], [Valumes 200 u1]]
sample/XBP the exoEasy spin column. Add ml of huffer XWP the
ADD: [[uuppt: 1, [seagent:
sample/XBF mixture]] >>
AnD: [1Outpat: }, [Reagents buffer
exoEasy spin column. Transfer the spin column to fresh collection tube.
XMP), [Volumes 10 ml]; ->
TRANSPER: [[Output: Qlasoll,
[Containes: fresh collestion tubel]
700 Qiazal
ADD: [[outpat: 1. [seagent: [iazol],
brane of the spin column.
ume:: 730 bl]] ->
Transfer the flow thr-ghh which is the lysate, to a PLG ube..
TRANSPTR: ploutput: foo-thhouuhh,,
[Containers PLG Lube]; ->
Incubate room temperature for minutes.
INCUBATE: [1Output: }, (Temperatures
room. temperaturel, [Tine: minutes]]
chloroform.
ADD:
[[ouppt:
1, [Volume:
ul]]
This series of examples in Tab. A9 illustrates the challenges faced with longer sentences due to the diversity of actions and the multiple parameters.
Table A9: Distinctions on the syntax level when processing longer sentences
Original Text
Syntax Level
Remarks
Rinse
RINSE: [Toutput:
Pas], [Seagent: 28S], (Volume: ml]] ->
Trypsinize with 0.5 ml trypsin.
TRYPSINIZE: ([output: [Rezgent:
trypsini, [Volume: 0.5 mi]] ->
Transfer to 15 ml tube.
TRANSFSR: [[Output: J, [Container: tubel]
Add 10 ml DMEM with 10 PBS.
ADD: [[Outpat: 1, [Reageat: DMEM,
rasl, [vclume: mil:
Incubate overnight at "C, CO2.
LNCUBATE: [[Uutput: 1, [Temperature:
Lack parameter
37 *c]]
Wash cells with PBS, add osteogenic differentia- tion medium.
[[Output: IVolume:
PBS1,
Lack action in single
[Reagent: osteogenic di difrerettiation
medium]]
Change medium every days for days.
CHANGE; [[Patput: Alizarin red 5), every days for days]]
day 10, stain Alizarin red
STAIN: pluutput: J, [Reagent: Alizarin
S for 5 min.
red 51, [Time: min]: >>
A17

---

Distinctions on the semantics level When supplementing known unknowns, human experts tend rely on contextual reasoning. Since experts are not familiar with protocols from all fields, they often infer parameters based on context for protocols outside their expertise. The primary source of their errors lack of understanding of protocols in specific domains, which is fundamentally consistent with the approach of our system. When supplementing unknown unknowns, human ex- perts tend to transfer their knowledge from familiar domains, such as instruments used or common parameters, protocols in various fields, using this basis for parameter supplementation Our system, however, completes parameters based on all collected protocols, which essentially the opposite of the transfer process used by human experts.
The example presents follows the completion of two types of parameters the semantic level is included: for instance, determining the configuration parameter for an operation, where human experts rely on personal experimental experience; and inferring the required reagents for one step, where human experts use contextual reasoning. When the context is not sufficiently clear, human experts cannot infer the known unknowns within single sentence.
Table A10: Machine results on the semanties level
Original Text
Machine Result
Remarks
Add 700 ul of buffer RWT to the
ADD: [[ouppa:: 1, [Reagent: esbuffer
known unknown
RNeasy MinElure spin column.
RMT=>], [Volumes 770 oll]
Discard the Ilow-the thw-thouugh.
DISCARD: [[output: the flox-through), [Volume: ]]
Discard the collection tube with the flow-through.
DISCARD: [[Output: the flow-through),
[Containers the collection tube],
[Volume: 1, [Reagent: the
flow-tarcuga!]
Trunsfer the RNeasy MinEllute spin column into ml col- lection tube (supplied).
TRANSFER: [[uutput: 1, [uevice: Rieass
MinFlute), [Sentainer:
collection tube (supplied) J, [Volume:
->
Open the of the spin column.
02EN: [[Output: 1, [Container: ssspin
known unknown
columno>]]
Centriluge at speed xg) to dry the membrane.
CONTRISUGE: [[output:
[speed:
full apeed (14,030 xg) [Container:
membrane], [Time: <e5 mins>]]
Discard the collection tube with flow-through.
DISCARD: [10utput: the flow-through), [Container: the collection tubel,
[volume: 1, [Reagent: the
flow-threugh!)
Trunsfer the RNeasy MinElute spin column 1.5 ml col- loction tube.
TRANSFER: [[uttutt: RNase-free
water], [Devices
MinElute],
[Containes: new 1.5 ml collection
(supplied) [volume:
Add RNase-free water di- rectly to the center of cnl. umn membrane.
ADD: [[outpat: 1, [Neagent:
known unknown
ecRNase-froe waters>), [Volume: 14 ul]]
A18

---

Table All: Human results on the semantics level
Original Text
Human Study Result
Remarks
Add 700 ul of bulfer RWT to the
ADD: [[outpat: 1, [Seagent:
e<NONEw>],
known unknown
RNeasy MinElute spin colurn.
[Volume: 730 ul]] ->
Discard the flow-throng. through.
DISCARD: [[Outpot: the flou-through),
[Volume: ]]
Discard the collection lube with flow-through.
DISCARD: [[Output: the flow-through),
[Container: the collection tubel,
[volume: 1, [Reagent: the
flow-through!) ->
Transfer the RNeasy MinElute ml
TRANSFER: [[Output: 1, [Device: Rleasy MinElutel, [Container:
spin column loction tube (supplied).
collecticn tube (supplied) 1, [Volume:
Open the of the spin colunn.
USEN: [[Cesput: I, [Container: esspin
known unknown
columnsx]]
Centrifuge speed kg) to dry the membranc.
CENTRIZUGE: [[Output:
[Speed:
unknown
rull speed 030 xg) [Container:
membrane), [Time: <<5 mins>]] >>
Discard the collection tube with the flow-through.
DISCARD: [[Outpot: the flow-through), [Container: the collection tube],
[Valume: }, [Reagent: the
flow-tarougal] ->
Trunsfer the RNeas MinElute spin column 1.5 ml lection tube.
TRANSFER: [[ttppt: Alase-free watesl, [Device: Blessy KinElute], [container: new 1.5 ml collection
tube (suppliad) ), [Volume:
Add RNase free water di. reatly the center of spin umn membranc.
ADD: [[Outpat: }, (Reagent:
known unknown
ecwaterwa], (Volune: ul]]
A19

---

Distinctions on the execution level Human experts track capacity primarily based on prior knowl- edge, subsequently using context to judge the appropriateness of the equipment used. In contrast, the machine extracts the entire flow process, enabling it to calculate each step and ensure that the capacity tracking scientifically sound and reasonable.
This series of examples in Tab. A12 demonstrates how our system tracks the required capacities at each step of the protocol by contextualizing the step into the spatial dimension.
Table A12: Distinctions on execution level (spatial dimenston)
Original Text
Execution Level
Resources
Add 4 ul of 160 mM KMa04 ng.
ADD: [[outpat: reaction mixture),
"radiolabeled DNA"
rndiolabeled
[Reagent: 160 MM KMn04, radiolabeled
10,000 opm) in 40 ul total volume.
DKA], [Volume: ul, 40 ul]
Precipitate with ethanol.
PRECIPITATE: [[Output: precipitate),
"reaction mixture"
[Seagent: ethsnoi]] ->
Dissolve 70 ul piperidine.
DISSOLVE: Houuppt: dissolved DNAI, [Reagent: 10 piperidinel, (Volume: 70 a1]]
"precipitate"
'C
TNCURATE: [loutput: incubated DNA], [eemerrature: 90 *c], [Time: 30 min]]
'dissolved
Precipitate with ethanol.
PRECIPITATE: [[Cutpu:: peliets], [seagent: ethanol]] ->
"incubated DNA
Wash pellets with ethanol, dry, dissolve in electrophoresis
RINSE: [[Output: нoл Labeled DNA), [Reagent: ethanol, ropotroph
"pellcts"
loading buffer.
loading bu[[er], (Volune: ul]]
This series of examples Tab. A13 illustrates how our system tracks the preconditions and post- conditions at each step of the protocol by contextualizing the step into the temporal dimension.
Table A13: Distinctions the execution level (temporal dimension)
Original Text
Execution Level
Resources
hour -80CC,
PREE28: [[output: DLFRED4], [Reagents
"DLFR004"
thaw at 37 for hour.
cells], [Time: housl, [Temperatures -30*c]1, THAN: [[output: 2L58004],
[Reagent: calla], [Time: heur),
[Terpersture: 37"C]] ->
If nor using DRO00 lyse cells with lysis butfer.
LYSE: [[Output: cell Lysate], [Reagent: lysis bufferl, [Conditions not using [LFRC04]] =>
"DL
Prepare serological pipette by ling at the 3 ml. mark, sealing but. tom parafilm.
PREPARE: [[Output: modified pipettel, [Device: serclogicsl pipette], [Modi tons cutt ing at the 3 mack, sealing bottom with para[ilm]]
"lysis buller"
Secure serological pipette to ver- surface.
S2CURE: [[Output: secured pipettel, [Device: serclogies] pipette]]
"moditied pipette"
Fill pipette with least 2.5 ml. cell lysate, 2 mL to mL mark.
FILL: [[Cetput: filled pipatte),
"secured pipetle"
distance from
7.5 mī.,, [Reagent:
cell lysate]]
Position cell phone camera record pipette, drop a glass bead insäde, repeal
POSITION: [[Output: zecorded experimentl, [Device: cell phone bead]]
"tilled pipette"
limes.
pipette), [Reagent: glass
Remove parafilm seal.
REMCUR: [[Datput: 1, [Containers
"recorded experiment"
parafilm seal]]
Rinse pipette.
RINGE: [[Output: cleaned pipette),
"next sample"
[Device: pipettel]
Repeat with next sample to obtain triplicttes.
REPEAT: [[output: triplicates]] ~>
"triplicates"
A20

---

E.4 Properties of the pre-processing pipeline
Significant differences exist between various stages of the pre-processing pipeline. We present sev- eral real-world examples to illustrate these distinctions in Tab. A14
Table A14: Showcases of action extraction, entity extraction, and classification in protocol steps
Original Text
Action Extruction
Entily Extraction
Classification
with
Preprocess Result
LLM-Pure
LLM
Stain with DAPI mucleic acid stain for 30 seconds.
stain
[Reagent:
[Reagent: DAPI nucleic acid asain], [Times 30 seconds]
STAIN: [ [Outpat:
STAIN:
DAP1 mucloic acid stain), [Time: 30 seconds]
[[uuration:
[Reagent:
seconda],
DAP1 nucleic stain), [Time: 30
[Reagent: DAPT mucleic stain]]
seconds]]
Purify CD4+
purity
[ReagenL: CD4+1, [Devica: the Auti MAC3 sorter (Miltenyi
[ReageaL:
PURIFY:
PURIPY: [[Device: Auto XACB sorter (Miltenyi Rictec),
magnetic isolation using
cD4+1, [Device:
[Device:
the Auto MACS
the Auto
sonter (M Ienyy
sorter
sorter
(Mikenyi
Biotect
(Milteny]
F0SSELD2],
Biotec) using
[Device:
Biotec),
[Method:
POSSELD2
POSSELD2
POSSE1D2
program.
Biotec) 1,
progcam]
progzam],
isclaticn],
[sevice:
[Output:
1
[Reagent: cu4+11
POSSELD2
[Seagent:
program]
CD4+]]
Mensure baseline oxidative every 20 for at min, add stimalating substances (c.g.- thapsigargin).
measure, [Output:
[Output:
NEASURE;
MEASURE:
baseline
baseline
[[Outpat:
[[Output:
exidative
exidative
baseline
beseline
status],
status), [Time:
exidative
oxidarive
[Times
every 20 s], [Tines 5 mini,
atatua],
statua],
evezy 20 (Times min),
[Time: evezy
[Reagent:
(Reagents
stimulation
s: innlating substances]
min]l: [[eegent:: stimulating ssbatances]
ADD:
substances],
[Reagen::
[Time: evezy
stimulating
5 min]]
substances]
Spin the crude by ultra- contrifugation at RPM properly pellet residual insolu- ble proteins from the extract.
spin
[Devices
(nevice:
SPTN:
SPTN: [[Reagent:
uliracentris
io
[[Device:
extracte],
[Force:
[Force: 55000
ultra
[Mehhod: ultra-
55000
RUM] [Reagent: resicual
centrifugstion]
centri fugaziin,. [Purpose:
RPM],
[Forca: 55000
[Reagent:
insoluble proteins),
Rех], [Output:
to properly
residusl
[Raagent:
pellet residual
inaoluble
[Container: the extract]
residual
insoluble
proteins],
inscluble
preteins
[Container:
proteins]
extract), [Speed: 55000 RPM]
extract]
Confimm pas- itive colonies transient transfection of sgRNAs analysis (SPH primers).
confirm [
[Dutpot:
CONF TRM:
CONFTRM:
positive
positive
[[Outpat:
[[Device: SPE
colonies],
colonies],
positive colcnies],
primers],
[Reagent:
[Seagent:
[Method:
sgRNAs
sgRMAs analysis
[Reageat:
transient
analysis
(SPH primers)]
aşRNAS
:ransfection of
(SPH
analysta (SPH primers)
SgRNAS analyaia), [output: positive
primers]
colonies]
A21

---

Reproducibility
The project page with supplementary files for reproducing the results of this paper will be available at htttss///eooss..ooooooooe///e//r html
G Limitations
As systematic study with proof-of-concept framework, the design and evaluation of the pipeline come with limitations, leading to further investigations:
We majorly exploit the approaches of empirical study observe the behavior of DSLs and human experts for extracting design principles. Can we draw theories from information theory to rigor- ously prove the expression capacity of DSLs and other structural knowledge representations, to advance our design choice?
We majorly consider the imperative programming DSL as the vehicle of PDGs in this work. This raises the question of whether incorporating alternative programming paradigms, such as functional and object-oriented models, could enhance the representation of complex entities within
protocols, particularly the properties of reagents.
Can we extend the protocol translator to larger set of experiments, especially those with hetero- geneous hardware devices such mobile robots?
Can we find similar mechanism other critical domains with the requirements on protocol exe- cution, such advanced manufacturing, and generalize our translator for such applications?
With many questions unanswered, we hope explore more on automated protocol translation for self-driving laboratories and beyond.
A22

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In this paper, we systematically study the problem of translating protocols for human experimenters into those suitable for self-driving laboratories, order standard- ize and automate the translation process. Further, we propose the initial proof-of-concept framework that fully frees domain experts from hand-crafting protocol translators.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the potential limitations at Appx. G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
A23

---

Answer: [NA]
Justification:
No theoretical result included in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided them with the implementation details at Appx. C also release our codes upon acceptance.
We will
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
A24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We have provided them with the implementation details Appx. also release our codes upon acceptance.
C We will
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (nttps: /ttp//ippccooo
http:/ic/ggi/e/eeeeeeeeeee Codeeee///ooSSuddomomm ss oon welooooPooilll for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
1/n 1nips..a cc /pbblpubppl 1 c/ gu i des CodeSubbn as onPol icy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All of them are carefully illustrated in implementation details Appx. C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Yes. Please refer to Fig.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
A25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please refer Appx. C.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////.aaaaa.eeoe//aiic//ttta
Answer: [Yes]
Justification: Yes. Please refer to the main text.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Please refer to the general discussions at Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
A26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes. Their licenses checked and their published works are properly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, |ppperswitte... www.swttthhh.cc.oc com/ datassts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
A27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Question:
Answer: [Yes]
Justification: Please refer to Appx. and Appx. D
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
We have obtained an approved IRB in advance. Please refer to Appx.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
A28


Reviews:
[{"review_id": "CsyMp1g0z5", "reviewer": "Reviewer_RqLD", "summary": "The work identifies the problem of translating from natural language instructions for scientific experiments to machine usable formats and frames it as a program synthesis problem. The proposed approach uses language models (along with other parsing techniques) to extract a structured sequence of instructions from natural language inputs which are then verified using an execution model. This approach is compared against expert translations as well as constraint decoding and prompt engineering baselines.", "questions": "Why is BLEU/ROUGE used as a metric? Can a more semantically aligned mode of comparison be used here.\n\nQualitative, how does the approach behave differently to the baselines? What does each component of the system contribute?", "limitations": "The evaluation presented in the paper considers datasets where the input is descriptions of experiments in standardised scientific terminology / format. While this demonstrates the system's usefulness in translating such inputs, it is unclear how well it may generalise to inputs following looser terminology / formatting. Since the tool aims to save time for domain experts, it should be demonstrated how much it is easier to write down instructions in this format as compared to the machine usable one directly.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper identifies an interesting problem in the AI for science Domain. It shows that the decomposition of the problem into syntax and semantics can be mapped to operations and reagent flow which is a useful insight. The paper further introduces useful formalism in the form of the PDG and algorithms to synthesise programs in DSLs. The paper evaluates on multiple datasets against reasonable benchmarks, showing the superiority of the proposed approach.", "weaknesses": "The use of BLEU and ROUGE scores as metrics to compare expert and machine generated instructions is not properly justified. It may infact be problematic considering that both metrics measure textual similarity however, instructions that looks similar could have very different semantics (for example: \"... pour hot water ...\" vs \"... pour cold water ...\")\n\nThe paper does not provide any insight into what each part of the system contributes to the final effectiveness. It would be especially valuable to understand (1) how much language model-based parsing makes a difference and (2) how the constraints imposed by the synthesis prevent the model from incorrect solutions (vs pure prompt engineering baselines) but also provide enough flexibility to do better than standard constraint decoding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Why is BLEU/ROUGE used as a metric? Can a more semantically aligned mode of comparison be used here. \n\nThis is a very good question. The same concern was considered during the development of our evaluation methodology. Direct comparisons across entire sentences under BLEU/ROUGE scores would indeed pose a problem as the reviewer mentioned. Therefore, to circumvent this issue, we convert all results into a standardized JSON-style format for data representation, and comparisons are made between key-value pairs rather than entire sentences, effectively resolving the metric concern.\n\nLet us consider the example mentioned by the reviewer: we represent the two sentences \"... pour hot water ...\" and \"... pour cold water ...\" in the following JSON-style format.\n\n``` \n{\n op: \"pour\",\n reg: \"water\",\n T: \"hot\",\n}\n\n{\n op: \"pour\",\n reg: \"water\",\n T: \"cold\",\n}\n```\n\nThe comparison between the two sentences is then transformed into a comparison between two JSON code blocks. We calculate the similarity score cumulatively based on the similarity between the values of matched pairs of keys. For instance, for the key \"temperature\", the values \"hot\" and \"cold\" yield a low similarity score under the ROUGH, BLEU, and even the Exact Match metrics. As \"temperature\" is one of the major keys within configuration parameters, a high penalty in this dimension significantly affects the cumulative similarity score. With this fine-grained comparison metric, we can comprehensively track the distinctions and commonalities between results without losing expressivity regarding the quantities.\n\nWe also acknowledge that there are advanced evaluation metrics, especially in the recent works where LLMs are leveraged as external judges and achieve considerable performance in general testing cases. Our choice of \"less advanced\" metrics is driven by the intention to focus specifically on domain-specific knowledge, which constitutes the primary scope of this paper and may be relatively sparse in general LLMs. Nonetheless, the exploration of more sophisticated evaluation metrics represents a promising avenue for future research, and we appreciate the reviewer's perceptive recommendation in this regard.\n\n> Since the tool aims to save time for domain experts, it should be demonstrated how much it is easier to write down instructions in this format as compared to the machine usable one directly.\n\nThanks for the comment. The scope of our proposed framework, in the current stage, is to automatically translate human-oriented protocols into formats suitable for machine execution, rather than helping domain experts creating new ones in an easier format. The goal is to transfer knowledge in a conventional lab into a format suitable for self-driving labs. Therefore, the human-oriented protocols used in this translation are existing protocols previously designed for human operators in the conventional labs, thus coming with no extra cost. \n \nIn contrast to conventional protocol translation processes, which require domain experts to manually develop rules and functions based on specialized knowledge, our proposed automatic translator attempts to eliminate the need for such expert intervention. Domain experts are not involved in either the development or the execution stages of our translator. Therefore, the evaluations in our paper mainly focus on translation performance rather than generation efficiency. We appreciate the reviewer's insightful suggestion and will make revisions accordingly for better clarification. \n\n> While this demonstrates the system's usefulness in translating such inputs, it is unclear how well it may generalise to inputs following looser terminology / formatting. \n\nThanks for the comment. The general applicability of our proposed framework beyond experimental sciences can indeed be a common concern. The core value of translating natural-language-based protocols into formats suitable for machine execution substantially lies in facilitating experiments in self-driving labs, thereby accelerating scientific discovery. Experimental protocols come with unique properties and challenges, such as the fine-grained incorporation of domain-specific knowledge, the non-trivial dependency topology between operations, the long-horizon lifecycles of intermediate productions, and the necessity for precise execution without run-time errors. These factors shape the scope of our research problem, emphasising the need to handle protocols with stringent terminology and formatting.\n\nDespite the specific scope of this paper, we are open to exploring the potential for generalizing our framework to other domains with similar challenges as those found in scientific experiments, such as cooking (see C.1-1).\n\n> Qualitative, how does the approach behave differently to the baselines? What does each component of the system contribute? \n\nThanks for the question. The rationales for the components within our proposed framework are grounded in both empirical and theoretical considerations. We develop the three-stage framework that integrates cognitive insights from human experts with approaches from program synthesis, automata construction, and counterfactual analysis. At the syntax level, we synthesize the operation dependence graph to transform natural-language-based protocols into structured representations, thereby making explicit the operation-condition mappings and the control flows. At the semantics level, we analyze the reagent flow graph to reconstruct the complete lifecycles of intermediate products, addressing the latent, missing, or omitted properties and values. At the execution level, we contextualize both the operation dependence graph and the reagent flow graph within spatial and temporal dynamics, resulting in the protocol dependence graph. This graph conducts counterfactual reasoning to detect potential conflicts or shortages of execution resources and to identify inappropriate combinations of operations in execution sequences (see C.1-2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aRCUSdYgSq", "reviewer": "Reviewer_AwgV", "summary": "The paper presents a framework for translating experimental protocols from natural language (NL) to machine-interpretable formats, specifically designed for self-driving laboratories. The proposed framework automates the protocol translation process through a three-stage workflow that constructs Protocol Dependence Graphs (PDGs) incrementally at the syntax, semantics, and execution levels. The approach is validated through quantitative and qualitative evaluations, demonstrating its performance on par with human experts.", "questions": "- Can the authors provide more details on how the system handles ambiguous or incomplete protocol instructions that may be common in real-world scenarios?\n- What specific optimizations could be applied to reduce the computational requirements of the proposed framework?\n- How does the system ensure the safety and correctness of translated protocols, especially in high-stakes domains such as medical and clinical research?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces a novel, automated approach to protocol translation for self-driving laboratories, addressing a critical gap in the transition from AI-driven discoveries to empirical experimentation.\n\n* The paper is well-structured and clearly written.", "weaknesses": "- The proposed method requires substantial computational resources for training and execution, which might limit its accessibility for some research teams.\n- The paper could benefit from a more detailed analysis of the types of errors made by the automated translator compared to human experts, which would help understand the limitations and areas for improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> What specific optimizations could be applied to reduce the computational requirements of the proposed framework? \n\nThanks for the question. Computational efficiency is always a topic of interest when evaluating new computational frameworks. Let us consider a new coming protocol with $k$ steps, with each step configured by a constant number of parameters, denoted as $\\epsilon$. At the syntax level, the primary computation bottleneck arises during DSL program synthesis, where the EM Algorithm exhibits a worst-case complexity of $O(\\epsilon^k)$. This is a highly conservative estimate, as mainstream optimization approaches can solve the EM much more efficiently. At the semantics level, the bottleneck occurs during reagent flow analysis, which consumes $O(k^2)$ complexity. Notably, only approximately 10\\% of the steps are included in the nested loop for reagent flow construction, as about 90\\% of the steps are linearly connected. At the execution level, the protocol execution model also exhibits $O(k^2)$ complexity, encompassing both forward and backward tracing. This can be optimized by replacing the full tracing strategy with a sliding window built upon the topological dependencies between steps. \n\nAlthough the complexities of the algorithms at these three levels are tractable, there is substantial room for improving the efficiency of the framework. Investigating methods to speed up the translation process for protocols with extremely high complexity would be a valuable area of research. We are committed to making the computational framework as accessible as possible for all research teams.\n\nIn addition, our proposed framework functions as an auxiliary module for LLMs, supporting the use of off-the-shelf LLMs such as GPT and Llama without the need for domain-specific fine-tuning. Costs associated with calling commercial LLM APIs are quite affordable. We selected OpenAI's gpt-3.5-turbo-0125 model for our experiments. Across 75 test protocols, we executed 1816 queries to achieve syntax-level translation, resulting in structured protocols. At the semantic level, we conducted 4062 queries for completion tasks (including translating protocols retrieved from training dataset). Consequently, our expenditures were approximately 17 USD in total.\n\n> How does the system ensure the safety and correctness of translated protocols, especially in high-stakes domains such as medical and clinical research?\n\nThanks for the question. In general, ensuring the safety and correctness of translated protocols in high-stakes domains is an exceptionally challenging task. Several factors contribute to these challenges, including accurately mapping operations to their corresponding configuration parameters, precisely parsing control flows from natural language, completing latent semantics with domain-specific knowledge, inferring missing or omitted key information, tracking resource capacities, and verifying the safety of run-time execution of experiments. Even minor errors in these areas can significantly compromise the safety and correctness of translated protocols. Consequently, we have made specific efforts in response to these challenges. \n\nAt the syntax level, we synthesize the operation dependence graph to transform natural-language-based protocols into structured representations. This approach explicitizes operation-condition mappings and control flows. At the semantics level, we analyze the reagent flow graph to reconstruct the complete lifecycles of intermediate products, thereby addressing latent, missing, or omitted properties and values. At the execution level, we contextualize both the operation dependence graph and the reagent flow graph within spatial and temporal dynamics, resulting in a protocol dependence graph. This graph facilitates counterfactual reasoning to identify potential conflicts or shortages of execution resources and inappropriate combinations of operations within execution sequences.\n\nWe provide several illustrative examples to demonstrate these concepts (see C.2-1).\n\n> The paper could benefit from a more detailed analysis of the types of errors made by the automated translator compared to human experts, which would help understand the limitations and areas for improvement.\n\nThanks for the suggestion. Here we present a detailed analysis of the errors made by our proposed automatic translator compared to human experts. We discuss the potential improvements of the translator accordingly.\n\nAt the syntax level, the major difference being in the analysis of long sentences in natural language. Human experts analyze the parameters of events/actions or multiple actions in long sentences with ease, while for our approach, there are sometimes problems with the correspondence between action and parameter (see C.2-2-1). \n\nAt the semantic level, when supplementing known unknowns, human experts tend to infer parameters based on established protocols outside their expertise; when supplementing unknown unknowns, human experts tend to transfer their knowledge from familiar domains to protocols in various fields. Our system, however, completes parameters based on all collected protocols, which is essentially the opposite of the transfer process used by human experts (see C.2-2-2).\n\nAt the execution level, human experts track capacity primarily based on prior knowledge, subsequently using context to judge the appropriateness of the equipment used. In contrast, the machine extracts the entire flow process, enabling it to calculate each step and ensure that the capacity tracking is scientifically sound and reasonable (see C.2-2-3).\n\n> Can the authors provide more details on how the system handles ambiguous or incomplete protocol instructions that may be common in real-world scenarios?\n\nThanks for the question. Here, we present a series of case studies to elucidate the the specific behaviors of components within the proposed three-stage framework at the syntax, semantics, and execution levels (see C.2-3).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M4tdKJJmR7", "reviewer": "Reviewer_uEvE", "summary": "This paper proposes an automated protocol translation framework, which takes natural language descriptions designed for human experimenters as input and outputs a structured representation that can be used for self-driving labs. The framework consists of a three-stage workflow. First, a domain-specific program is synthesized from the natural language description using a classic action/entity extraction and Expectation Maximization approach. Then, reagent flow analysis, essentially reaching definitions of the synthesized program is performed. Third, constraints prohibiting undesired execution behaviors are inferred and will be used to monitor the execution. Results show that the synthesized protocol translation matches manually written ones by human experimenters.", "questions": "What off-the-shelf tools are used in the pre-processing to extract actions and entities?  Are they LLMs? Are there important differences between the extraction of action/entities and the extraction of reagent entities (state-of-the-art LLMs are used in the latter)?", "limitations": "The authors briefly discussed limitations in Appendix E.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem is well-motivated and of great significance in advancing AI applications in scientific discovery. \n- The paper is easy to follow, although the required background knowledge is non-trivial. \n- The empirical evaluation shows that the proposed approach outperforms pure LLM-based synthesis and matches the manual translation by human experimenters.", "weaknesses": "- The proposed solution is a portfolio of standard applications of existing tools or well-known algorithms, which is less interesting and novel from a machine learning perspective.\n- The targeted DSL is relatively simple, and the proposed solution consists of ad-hoc design choices (particularly spatial-temporal dynamics), which may not generalize well to DSLs with richer features", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The proposed solution is a portfolio of standard applications of existing tools or well-known algorithms, which is less interesting and novel from a machine learning perspective.\n\nThanks for the comment. In this work, we study the problem of translating experimental protocols designed for human experimenters into formats suitable for machine execution. Our primary motivation is to bridge the existing gap between machine learning algorithms in the field of AI for science, such as molecular design, and the grounded experimental verification facilitated by self-driving laboratories. We appreciate the reviewer's recognition that \"the required background knowledge is non-trivial\". Indeed, conventional workflows for setting up self-driving laboratories and conducting physical experiments necessitate deep integration with domain experts, significantly impeding the progress of machine learning researchers in verifying and iterating their findings. Consequently, our framework aims to provide an infrastructure that enables these researchers to advance their machine learning algorithms and seamlessly validate their findings, thereby closing the loop of automatic scientific discovery.\n\nTo meet the requirements of such infrastructure, we conduct a systematic study to identify existing gaps in protocol translation between human experimenters and automatic translators in self-driving lab. Accordingly, we develop the three-stage framework that integrates cognitive insights from human experts with approaches from program synthesis, automata construction, and counterfactual analysis. At the syntax level, we synthesize the operation dependence graph to transform natural-language-based protocols into structured representations, thereby making explicit the operation-condition mappings and the control flows. At the semantics level, we analyze the reagent flow graph to reconstruct the complete lifecycles of intermediate products, addressing the latent, missing, or omitted properties and values. At the execution level, we contextualize both the operation dependence graph and the reagent flow graph within spatial and temporal dynamics, resulting in the protocol dependence graph. This graph conducts counterfactual reasoning to detect potential conflicts or shortages of execution resources and to identify inappropriate combinations of operations in execution sequences. \n\n> The targeted DSL is relatively simple, and the proposed solution consists of ad-hoc design choices (particularly spatial-temporal dynamics), which may not generalize well to DSLs with richer features\n\nThanks for the comment. In this study, we aim to investigate the development of automatic translators for executing experimental protocols in self-driving labs. To execute the instructions on Internet-of-Things-connected hardware devices, such as valves, pumps, and reactors, protocols must ultimately be formatted in JSON-style configuration files, which represent a mainstream format in hardware-software communication (see C.3-1).\n\nAlthough DSLs with syntactic and semantic features differing from those of JSON-style DSLs for self-driving labs are beyond the scope of this paper, generalizing the framework to DSLs with other language features represents a significant direction for future research. We appreciate the reviewer's suggestion in this regard.\n\nWe acknowledge that the core components of our framework are designed in an ad-hoc manner. The design guidelines are derived from both a systematic study of the required cognitive capabilities in protocol translation and established computer science theories. Rather than directly devising engineering solutions tailored to the specific problem, we decompose the problem to abstract subproblems, including symbolic regression, flow analysis, and counterfactual reasoning. We consider these subproblems as scientific challenges and develop solutions from the conceptual level to the implementation level in a top-down approach. \n\nSpecifically, the scientific challenge behind the design of spatial-temporal dynamics lies in the extremely long-tail distribution of historical run-time error cases. To address this issue, we propose leveraging foresight simulation by contextualizing individual operations within both the spatial dimension, i.e., the specific assignment of resources according to capacity requirements, and the temporal dimension, i.e., the specific precondition and postcondition of resources according to their properties. The division of spatial and temporal dimensions is mutually exclusive and echoes the two major aspects of computer programs --- computation resources and control logic. Thus, although ad-hoc, the design of spatial-temporal dynamics targets the underlying scientific problem behind the superficial challenges and holds the potential for generalization to other DSLs. Extending the application scope of our framework is a significant direction for future work.\n\n> What off-the-shelf tools are used in the pre-processing to extract actions and entities? Are they LLMs? Are there important differences between the extraction of action/entities and the extraction of reagent entities?\n\nThanks for the question. We employ the SpaCy Dependency Parser to analyze the syntactic structure of protocols, which allows for the extraction of verbs and the identification of associated objects and modifiers. After parsing, these verbs are aligned with corresponding operational actions in our DSL by maximizing the cosine similarity between their word2vec representations and those of the DSL operations. Furthermore, we utilize a few-shot model based on LLM to accurately identify and classify entities within the text. The rationale of integrating LLMs with classical parsing techniques lies in leveraging the advanced natural language processing capabilities of LLMs while mitigating their inherent uncertainties. \n\nThere are significant differences exist between various stages of this pipeline (see C.3-2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time and valuable comments. The feedback is both substantial and helpful for improving our paper. In this work, we systematically study the problem of translating experimental protocols for human to those suitable for self-driving laboratories. Accordingly, we propose a three-stage workflow that incrementally constructs Protocol Dependence Graphs at the syntax, semantics, and execution levels. Our qualitative and quantitative results underscore the framework's potential to accelerate and democratize the process of scientific discovery.

We would like to thank the reviewers for acknowledging our work to be:
1. The paper identifies "an interesting problem in the AI for science Domain" (reviewer #RqLD), "is well-motivated and of great significance in advancing AI applications in scientific discovery" (reviewer #uEvE), and addresses "a critical gap in the transition from AI-driven discoveries to empirical experimentation" (reviewer #AwgV).
2. The proposed method, which is a "novel, automated approach to protocol translation for self-driving laboratories" (reviewer #AwgV), provides "a useful insight" regarding "the decomposition of the problem into syntax and semantics" (reviewer #RqLD), and "further introduces useful formalism in the form of the PDG and algorithms to synthesise programs in DSLs" (reviewer #RqLD). 
3. The evaluations are conducted on "multiple datasets against reasonable benchmarks" (reviewer #RqLD), showing that "the proposed approach outperforms pure LLM-based synthesis and matches the manual translation by human experimenters" (reviewer #uEvE).
4. The paper is "well-structured and clearly written" (reviewer #AwgV), and "is easy to follow, although the required background knowledge is non-trivial" (reviewer #uEvE). 

Based on the reviewers' comments, we made revisions including:
1. Clarifying certain concepts to enhance the paper's accessibility for readers with a background outside experimental sciences.
2. Demonstrating running examples of the behaviors of different components within our proposed three-stage framework in detail to make the paper more comprehensive.
3. Conducting additional analyses and discussions regarding the computational complexity, safety, and theoretical foundation of our proposed framework to make the paper more rigorous and self-consistent.

In the following, we address specific questions for each reviewer.

Meta-review: The paper presents a method for integrating AI into the scientific discover process in the form of a framework that translates scientific protocols written in natural language into a structured form that could conceivably be executed by a self-driving laboratory. The approach is validated through quantitative and qualitative evaluations. 

The paper appears well written and the idea novel and well motivated. Results show that the synthesized protocol translation matches manually written ones by human experimenters. Identified weaknesses are that the proposed solution is a portfolio of standard applications of existing tools or well-known algorithms, and this may be less interesting to the ML community. The method also requires substantial computational resources which might limit its accessibility.
Decision: Accept (poster)