ProtocoLLM: Automatic Evaluation Framework of LLMs on Domain-Specific Scientific Protocol Formulation Tasks
Authors: Seungjun Yi, Jaeyoung Lim, Juyong Yoon


Reviewer paper summaries: This paper proposes an evaluation framework for the scientific protocol formulation task, which requires the LLMs to extract pseudocode from protocols using predefined lab actions. The evaluation framework is designed by prompting target LLM and GPT-4 as baseline, and then employing Llama-3 as evaluator. The evaluation framework is experimented with various LLMs to prove its effectiveness.
Summary of strengths: - The experiments in Table 3 and Table 4 are pretty solid.
Summary of weaknesses: - I am mostly concerned with the actual contribution that this paper brings. First of all, the paper is submitted to Resources and Evaluation track. The proposed dataset, BioPort2.0, only has 300 small-scale protocols.
- Secondly, the novelty in Llam-Eval is hard to distinguish from G-Eval. I would suggest the authors add discussions on how this work is different.
- The motivation is not very natural, especially lines 57-78 are not intuitive to understand. Since there is only one baseline, the authors could consider adding one table/figure to highlight the difference.
- The presentation of this paper also needs significant improvement. I suggest the authors add a working example throughout the paper since not all readers are familiar with this task.
Comments, suggestions and typos: See weaknesses above.
Overall assessment scores: 2