Title: Expert-level protocol translation for self-driving labs
Venue: NeurIPS 2024
Accepted: yes
Keywords: Self-Driving Laboratories

Domain-Specific Language

Structural Representation

Knowledge Externalization
Abstract: Recent development in Artificial Intelligence (AI) models has propelled their application in scientific discovery, but the validation and exploration of these discoveries require subsequent empirical experimentation. The concept of self-driving laboratories promises to automate and thus boost the experimental process following AI-driven discoveries. However, the transition of experimental protocols, originally crafted for human comprehension, into formats interpretable by machines presents significant challenges, which, within the context of specific expert domain, encompass the necessity for structured as opposed to natural language, the imperative for explicit rather than tacit knowledge, and the preservation of causality and consistency throughout protocol steps. Presently, the task of protocol translation predominantly requires the manual and labor-intensive involvement of domain experts and information technology specialists, rendering the process time-intensive. To address these issues, we propose a framework that automates the protocol translation process through a three-stage workflow, which incrementally constructs Protocol Dependence Graphs (PDGs) that approach structured on the syntax level, completed on the semantics level, and linked on the execution level. Quantitative and qualitative evaluations have demonstrated its performance at par with that of human experts, underscoring its potential to significantly expedite and democratize the process of scientific discovery by elevating the automation capabilities within self-driving laboratories.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an automated protocol translation framework, which takes natural language descriptions designed for human experimenters as input and outputs a structured representation that can be used for self-driving labs. The framework consists of a three-stage workflow. First, a domain-specific program is synthesized from the natural language description using a classic action/entity extraction and Expectation Maximization approach. Then, reagent flow analysis, essentially reaching definitions of the synthesized program is performed. Third, constraints prohibiting undesired execution behaviors are inferred and will be used to monitor the execution. Results show that the synthesized protocol translation matches manually written ones by human experimenters.\n\nSTRENGTHS:\n- The problem is well-motivated and of great significance in advancing AI applications in scientific discovery. \n- The paper is easy to follow, although the required background knowledge is non-trivial. \n- The empirical evaluation shows that the proposed approach outperforms pure LLM-based synthesis and matches the manual translation by human experimenters.\n\nWEAKNESSES:\n- The proposed solution is a portfolio of standard applications of existing tools or well-known algorithms, which is less interesting and novel from a machine learning perspective.\n- The targeted DSL is relatively simple, and the proposed solution consists of ad-hoc design choices (particularly spatial-temporal dynamics), which may not generalize well to DSLs with richer features\n\nQUESTIONS:\nWhat off-the-shelf tools are used in the pre-processing to extract actions and entities?  Are they LLMs? Are there important differences between the extraction of action/entities and the extraction of reagent entities (state-of-the-art LLMs are used in the latter)?\n\nLIMITATIONS:\nThe authors briefly discussed limitations in Appendix E.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper presents a framework for translating experimental protocols from natural language (NL) to machine-interpretable formats, specifically designed for self-driving laboratories. The proposed framework automates the protocol translation process through a three-stage workflow that constructs Protocol Dependence Graphs (PDGs) incrementally at the syntax, semantics, and execution levels. The approach is validated through quantitative and qualitative evaluations, demonstrating its performance on par with human experts.

STRENGTHS:
* The paper introduces a novel, automated approach to protocol translation for self-driving laboratories, addressing a critical gap in the transition from AI-driven discoveries to empirical experimentation.

* The paper is well-structured and clearly written.

WEAKNESSES:
- The proposed method requires substantial computational resources for training and execution, which might limit its accessibility for some research teams.
- The paper could benefit from a more detailed analysis of the types of errors made by the automated translator compared to human experts, which would help understand the limitations and areas for improvement.

QUESTIONS:
- Can the authors provide more details on how the system handles ambiguous or incomplete protocol instructions that may be common in real-world scenarios?
- What specific optimizations could be applied to reduce the computational requirements of the proposed framework?
- How does the system ensure the safety and correctness of translated protocols, especially in high-stakes domains such as medical and clinical research?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work identifies the problem of translating from natural language instructions for scientific experiments to machine usable formats and frames it as a program synthesis problem. The proposed approach uses language models (along with other parsing techniques) to extract a structured sequence of instructions from natural language inputs which are then verified using an execution model. This approach is compared against expert translations as well as constraint decoding and prompt engineering baselines.\n\nSTRENGTHS:\nThe paper identifies an interesting problem in the AI for science Domain. It shows that the decomposition of the problem into syntax and semantics can be mapped to operations and reagent flow which is a useful insight. The paper further introduces useful formalism in the form of the PDG and algorithms to synthesise programs in DSLs. The paper evaluates on multiple datasets against reasonable benchmarks, showing the superiority of the proposed approach.\n\nWEAKNESSES:\nThe use of BLEU and ROUGE scores as metrics to compare expert and machine generated instructions is not properly justified. It may infact be problematic considering that both metrics measure textual similarity however, instructions that looks similar could have very different semantics (for example: \"... pour hot water ...\" vs \"... pour cold water ...\")\n\nThe paper does not provide any insight into what each part of the system contributes to the final effectiveness. It would be especially valuable to understand (1) how much language model-based parsing makes a difference and (2) how the constraints imposed by the synthesis prevent the model from incorrect solutions (vs pure prompt engineering baselines) but also provide enough flexibility to do better than standard constraint decoding.\n\nQUESTIONS:\nWhy is BLEU/ROUGE used as a metric? Can a more semantically aligned mode of comparison be used here.\n\nQualitative, how does the approach behave differently to the baselines? What does each component of the system contribute?\n\nLIMITATIONS:\nThe evaluation presented in the paper considers datasets where the input is descriptions of experiments in standardised scientific terminology / format. While this demonstrates the system's usefulness in translating such inputs, it is unclear how well it may generalise to inputs following looser terminology / formatting. Since the tool aims to save time for domain experts, it should be demonstrated how much it is easier to write down instructions in this format as compared to the machine usable one directly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}