SeedBench: A Multi-task Benchmark for Evaluating Large Language Models in Seed Science
Authors: Jie Ying, Zihong Chen, Zhefan Wang, Wanli Jiang, Chenyang Wang, Zhonghang Yuan, Haoyang Su, Huanjun Kong, Fan Yang, Nanqing Dong
Accepted at: acl25

SeedBench: A Multi-task Benchmark for Evaluating Large Language Models in Seed Science

Jie Ying1*   Zihong Chen1*   Zhefan Wang1*   Wanli Jiang1
Chenyang Wang1   Zhonghang Yuan1   Haoyang Su1
Huanjun Kong1   Fan Yang2   Nanqing Dong1,3†
1Shanghai Artificial Intelligence Laboratory   2Yazhouwan National Laboratory   3Shanghai Innovation Institute

Abstract

Seed science is essential for modern agriculture, directly influencing crop yields and global food security. However, challenges such as interdisciplinary complexity and high costs with limited returns hinder progress, leading to a shortage of experts and insufficient technological support. While large language models (LLMs) have shown promise across various fields, their application in seed science remains limited due to the scarcity of digital resources, complex gene-trait relationships, and the lack of standardized benchmarks. To address this gap, we introduce SeedBench1—the first multi-task benchmark specifically designed for seed science. Developed in collaboration with domain experts, SeedBench focuses on seed breeding and simulates key aspects of modern breeding processes. We conduct a comprehensive evaluation of 26 leading LLMs, encompassing proprietary, open-source, and domain-specific fine-tuned models. Our findings not only highlight the substantial gaps between the power of LLMs and the real-world seed science problems, but also make a foundational step for research on LLMs for seed design.

1 Introduction

Food security is a fundamental global concern, with seeds serving as the foundation of agricultural production. However, the seed industry faces significant challenges, including its inherently interdisciplinary nature and low economic returns. These factors contribute to a persistent shortage of skilled breeding scientists, a trend expected to continue over the next decade (Egan et al., 2024). The critical shortage of agricultural scientists directly constrains productivity improvements and the sustainable growth of food production.

With the rise of artificial intelligence (AI), advanced AI techniques are transforming seed science. The integration of AI-driven solutions into seed breeding aligns with projections that the next agricultural revolution will be driven by smart, digital, and precision agricultural technologies (Iversen et al., 2021). Large language models (LLMs), in particular, offer the ability to process vast amounts of genetic, environmental, and agronomic data, optimizing crop development. However, effectively assessing and comparing LLM capabilities requires high-quality evaluation benchmarks.

Despite the availability of LLM benchmarks for general purpose, none have been specially developed for seed breeding, which is a field critical to agricultural production and food security. Progress in this domain has been slow due to a shortage of breeding experts and limited availability of online resources. While existing agricultural benchmarks, such as AgEval (plant stress phenotyping) (Arshad et al., 2024) and AgXQA (agricultural extension Q&A) (Kpodo et al., 2024), contribute to LLM evaluation in agriculture, they fail to address the complex decision-making and multi-step processes unique to seed breeding. The absence of a dedicated benchmark limits the ability to systematically measure LLM performance in this domain.

To bridge this gap, we introduce SeedBench, a multi-task benchmark designed to simulate expert decision-making across three essential seed breeding stages: (1) gene information retrieval; (2) gene function and regulation analysis; and (3) variety breeding with agronomic trait optimization. Each task category is carefully designed with information-rich, expert-validated question-answer pairs, ensuring the benchmark aligns with real-world seed breeding challenges. By providing a structured evaluation framework, SeedBench enables rigorous assessment of whether LLMs can assist human experts, accelerate breeding workflows, optimize outcomes, and advance towards
autonomous intelligent seed breeding.

As the first and most comprehensive benchmark in this field, SeedBench systematically connects the capabilities of LLMs with real-world breeding challenges. Developed by experts with interdisciplinary backgrounds and Ph.D.-level expertise, SeedBench ensures both domain relevance and scientific rigor. Each question undergoes two-tier validation and refinement through machine-based assessment and human expert review, which guarantees accuracy and reliability. To accommodate diverse evaluation scenarios, SeedBench supports both one-shot and zero-shot formats, enabling a comprehensive assessment of LLMs in seed breeding.

We evaluate the performance of 26 leading LLMs on SeedBench, including 7 proprietary LLMs, 16 open-source LLMs, and 3 domain-specific models. Recognizing that LLM performance is highly sensitive to prompt structures, we further analyze multiple prompt templates to enhance robustness. Our evaluation aims to address the following research questions.

RQ1. What is the relationship between the reasoning ability of LLMs and the performance on seed breeding tasks?
RQ2. Do domain-specific fine-tuned models outperform general models in seed breeding?
RQ3. What is the ideal model size for seed breeding tasks?

Answering these research questions can not only facilitate the research on LLMs on seed breeding, an emerging topic, but also the understanding of usage of LLMs in the field of AI for Science.

Our main contributions are as follows:
• SeedBench is the first benchmark designed to evaluate LLMs in seed science.
• SeedBench covers key seed breeding processes, ensuring reliability and accuracy through expert validation.
• Extensive evaluations of LLMs are conducted to identify their strengths and limitations, providing insights for future AI advances in breeding.

In the following sections, Section 2 reviews related work on LLMs and agricultural benchmarks. Section 3 describes the construction and methodology behind SeedBench, outlining its design, task types, and validation process. Section 4 presents the experimental setup, including evaluated models and performance comparisons. Finally, we discuss findings, limitations, and future research directions for LLMs in seed breeding.

2 Related Work

2.1 Domain-Specific LLMs
Based on the Transformer architecture (Vaswani et al., 2017), language models have rapidly advanced, achieving key milestones in their development. Starting with foundational models such as BERT (Devlin, 2018) and GPT (Radford, 2018), subsequent breakthroughs like GPT-4 (Achiam et al., 2023) and DeepSeek-R1 (Guo et al., 2025) have demonstrated exceptional text generation. By integrating specialized domain knowledge with continual pre-training and supervised fine-tuning, LLMs have shown potential in domains such as education (Gan et al., 2023), finance (Li et al., 2023b), and healthcare (Mumtaz et al., 2024). In the agricultural domain, LLMs are increasingly recognized for their potential to enhance food production and optimize agricultural management (De Clercq et al., 2024; Kuska et al., 2024). However, in contrast to the progress in these domains, leveraging LLMs for seed breeding remains an under-explored challenge in the field of AI for Science.

2.2 Domain-Specific Benchmarks
To effectively assess and compare LLM capabilities, high-quality evaluation benchmarks are essential. While general benchmarks now cover a wide range of areas, including causal inference (Wang, 2024), instruction following (Zhou et al., 2023), and safety (Lin et al., 2022), domain-specific benchmarks have also emerged. These specialized benchmarks are found in fields such as finance (Xie et al., 2024), geography (Li et al., 2023a), healthcare (Chen et al., 2024a), and law (Fei et al., 2024). These benchmarks help drive improvements by creating diverse datasets and tasks tailored to specific applications. However, many of these benchmarks rely heavily on website data and GPT-based re-annotation, which may limit the data diversity and depth of expertise, raising concerns about the reliability and accuracy of the results.

In addition to data limitation, another challenge is the difficulty of tasks, especially scientific tasks that require additional domain knowledge. Existing agricultural benchmarks does not cover the topic of seed breeding, a critical task in agriculture. For example, AgEval (Arshad et al., 2024) aims for plant stress phenotyping, AgXQA (Kpodo et al., 2024) focuses on agricultural extension, and CROP (Zhang et al., 2024) evaluates the crop knowledge. The disparity underscores the necessity
Figure 1: Breeding Expert Workflow Framework. We establish benchmark construction principles by consulting domain experts to replicate real-world seed breeding decision-making processes. (1) Gene Information Retrieval, utilizing established databases to obtain gene sequences and expression patterns; (2) Gene Function & Regulation, employing experimental approaches (e.g., gene knockout, overexpression) to investigate gene roles in plant development; and (3) Variety Breeding & Trait Optimization, implementing breeding techniques (e.g., hybridization, backcrossing) combined with agronomic trait selection for stable variety development.

for specialized benchmarks to evaluate the performance of LLMs in seed science.

3 SeedBench

This section outlines the design principles of SeedBench and details its 11 evaluation task types. SeedBench systematically evaluates LLMs in seed breeding by aligning with the typical workflow of breeding experts. It is structured into three primary categories: gene information retrieval, gene function and regulation, and variety breeding and agronomic traits. This taxonomy ensures comprehensive knowledge coverage and skill assessment.

3.1 Breeding Process Overview

Breeding experts typically follow three key steps in seed selection (Figure 1; further details in Appendix A.1). These steps form the basis for SeedBench’s task categorization.

3.2 Task Taxonomy

The tasks in SeedBench are categorized into three main areas, corresponding to the seed breeding workflow. These categories are further divided into ten subcategories in total, ensuring a systematic evaluation of LLM capabilities (Figure 2). The categorization follows the practice in seed science with the help from domain experts (Copeland and McDonald, 2012).

3.2.1 Gene Information Retrieval

LLMs retrieve essential gene information linked to specific traits, including key genes in biological processes, gene sequences, functional descriptions, and expression patterns across environments and developmental stages. They also determine the intracellular localization of gene products, mapping their distribution within the nucleus, cytoplasm, or membrane. These tasks corresponds to the initial step of the breeding process. Specific tasks include:
• Gene Basic Information Query
• Gene Expression Pattern Query
• Gene Product Cellular Localization Query

3.2.2 Gene Function and Regulation

LLMs describe gene functions under specific experimental conditions using available data, analyze gene product regulation of downstream genes and pathways, and predict functions of uncharacterized genes. These tasks align with the second step of the breeding process. Specific tasks include:
• Gene Function Experimental Observation
• Gene Product Regulation of Downstream Genes Analysis
• Gene Function Prediction

3.2.3 Variety Breeding and Agronomic Traits

LLMs gather information on breeding history, methods, objectives, and agronomic traits, including disease resistance, yield, and drought tolerance. They also propose suitable planting regions based on environmental factors. This aligns with the third step of the breeding process. Specific tasks include:
• Variety Breeding Process Query
• Variety Agronomic Trait Query
• Variety Cultivation and Technical Key Points Query
• Variety Suitable Planting Area Recommendation

3.3 Benchmark Construction

SeedBench was developed through a structured three-step process: data collection, automatic question generation, and two-tier quality validation.
Figure 2: Benchmark Taxonomy Distribution. Three core breeding steps are further divided into ten expert-curated subcategories within SeedBench, which comprises a total of 2,264 questions. The percentages shown in the diagram represent the proportion of questions in each category relative to 2,264.

This methodology ensures a relevant, diverse, and scientifically rigorous evaluation of LLMs.

3.3.1 Data Collection
SeedBench is built on a comprehensive breeding knowledge base. We extracted 308,727 breeding-related publications in English and Chinese from publicly available sources to minimize language bias2. These papers were converted to Markdown using MinerU (Wang et al., 2024a) for consistency, primarily consisting of open-access academic works for credibility and reusability.

Data cleaning involved three steps: (1) heuristic filtering to remove noise, including corrupted or irrelevant data; (2) deduplication using a local-sensitive hashing method; and (3) scoring corpus segments with the CCI3-HQ-Classifier (Wang et al., 2024b) to eliminate low-quality fragments. And we used the IndustryCorpus2 Classifier3 to exclude content unrelated to seed breeding. These steps filtered out 86% of low-quality or duplicate data, yielding a corpus of 1.1 billion tokens.

From this corpus, domain experts selected 279 high-quality text segments, sourced from 113 documents, each averaging 300 words. These segments cover 10 predefined subcategories, ensuring both depth and breadth. Each segment is highly relevant to a specific subcategory, containing multiple knowledge points essential for model evaluation. To aid in automatic question generation, domain experts manually designed 293 reference questions. For illustration purpose, we consistently use rice to demonstrate the benchmark construction, empirical evaluation, and case studies across the main text and Appendix, given its global importance and representativeness in seed science.4 The data collection and curation for maize, soy bean, wheat, and many other plants follow the same procedure.

3.3.2 Automatic Question Generation

<table>
  <tr>
    <th>Type ID</th>
    <th>Question Type</th>
    <th>Metric</th>
    <th>Count (n)</th>
  </tr>
  <tr>
    <th colspan="4">Q&amp;A</th>
  </tr>
  <tr>
    <td>QA-1</td>
    <td>Multiple Choice</td>
    <td>Accuracy</td>
    <td>199</td>
  </tr>
  <tr>
    <td>QA-2</td>
    <td>Multiple Answer</td>
    <td>Macro-F1</td>
    <td>186</td>
  </tr>
  <tr>
    <td>QA-3</td>
    <td>Fill-in-the-Blank</td>
    <td>ROUGE-L</td>
    <td>223</td>
  </tr>
  <tr>
    <td>QA-4</td>
    <td>Generation</td>
    <td>ROUGE-L</td>
    <td>241</td>
  </tr>
  <tr>
    <th colspan="4">Summarization</th>
  </tr>
  <tr>
    <td>SUM-1</td>
    <td>Simple Summarization</td>
    <td>ROUGE-L</td>
    <td>224</td>
  </tr>
  <tr>
    <td>SUM-2</td>
    <td>Key Information Extraction</td>
    <td>ROUGE-L</td>
    <td>224</td>
  </tr>
  <tr>
    <th colspan="4">Reading Comprehension</th>
  </tr>
  <tr>
    <td>RC-1</td>
    <td>Multiple Choice</td>
    <td>Accuracy</td>
    <td>112</td>
  </tr>
  <tr>
    <td>RC-2</td>
    <td>Multiple Answer</td>
    <td>Macro-F1</td>
    <td>107</td>
  </tr>
  <tr>
    <td>RC-3</td>
    <td>Fill-in-the-Blank</td>
    <td>ROUGE-L</td>
    <td>220</td>
  </tr>
  <tr>
    <td>RC-4</td>
    <td>Generation</td>
    <td>ROUGE-L</td>
    <td>239</td>
  </tr>
  <tr>
    <td>RC-5</td>
    <td>Subcategory Classification</td>
    <td>Accuracy</td>
    <td>278</td>
  </tr>
</table>

Table 1: Benchmark Task Types. Each high-quality text segment systematically incorporates these 11 distinct task types to ensure diversity. 'Count (n)' indicates the number of questions for this particular task type after quality validation, from the total of 2,264 questions in SeedBench. The complete distribution statistics are provided in Appendix C.2.

The question generation phase begins with

2See Appendix B.2 for detailed language composition. We also discuss the potential impact of linguistic differences on the model’s performance.

3https://huggingface.co/BAAI/IndustryCorpus2_Classifier

4Rice is the most widely consumed crop in the world, which feeds over 3.5 billion people in the world according to Food and Agriculture Organization of the United Nations and World Bank.
Figure 3: Benchmark Construction Pipeline. We developed SeedBench by extracting 308,727 breeding-related papers from English and Chinese sources and converting them into a unified Markdown format. The data underwent rigorous cleaning, ultimately yielding a 1.1-billion-token corpus. From this, experts curated 279 high-quality text segments, spanning 10 breeding subcategories, for generating LLM-based Q&A tasks. Validation included both automated and expert reviews, removing low-quality entries and ensuring relevance. SeedBench offers 2,264 refined questions across 11 task types, enabling fine-grained evaluation of LLMs in seed breeding.

expert-designed example questions across four Q&A types: multiple choice (QA-1), multiple answer (QA-2), fill-in-the-blank (QA-3), and generation (QA-4). This foundation extends to summarization (SUM-1, SUM-2) and reading comprehension (RC-1 to RC-5) tasks, detailed in Table 1.

Utilizing GPT-4’s natural language processing capabilities, unstructured text is converted into structured knowledge dictionaries through key information extraction (Appendix F.6), capturing genetic traits, phenotypic features, and practical applications. Based on these dictionaries, GPT-4 generates four Q&A task types:
• Multiple Choice Questions
• Multiple Answer Questions
• Fill-in-the-Blank Questions
• Generation Questions

Then, we generated 450 summarization questions (SUM-1, SUM-2), with two per text segment, to evaluate the model’s ability to summarize breeding literature without requiring prior domain knowledge. For reading comprehension tasks, the original text segments were provided as reference documents in the augmented context. Using GPT-4, we rephrased the questions to require retrieval, analysis, reasoning, and answering based on the given document, assessing the model’s capability in long-context breeding problems.

Additionally, we formulated 279 classification questions, where the answer corresponds to the category of each text segment, testing the model’s ability to distinguish breeding areas. All questions are available in both zero-shot and one-shot settings. SeedBench thus serves as a multidimensional benchmark, covering 10 thematic subcategories and 11 task types for fine-grained breeding assessment. All prompt templates that we used are provided in Appendix F.

3.3.3 Quality Validation

Since SeedBench relies on GPT-4 for annotation, we implemented a two-stage validation process to ensure accuracy and reliability: automated machine screening and manual expert review.

Automated Machine Screening. GPT-4 first assessed coherence, logical consistency, and task adherence for each question, filtering out those with errors or contradictions. About 0.01% of the questions were excluded at this stage.

Manual Expert Review. Domain experts reviewed the remaining questions for relevance and alignment with expert perspectives, removing irrelevant or weakly contextualized ones (e.g., “Is rice planted in Beijing or Shanghai?”) and eliminating about 20% of the initial set.

After validation, 2,264 high-quality questions were retained. Detailed case studies and question distribution are provided in Appendix C.2.
3.4 Evaluation
The evaluation framework consists of two stages: response standardization and task-specific scoring.
• Multiple Choice Tasks: Accuracy.
• Multiple Answer Tasks: Macro-F1.
• Fill-in-the-Blank: ROUGE-L F1 for segment-level comparison between model predictions and reference answers, averaged across segments.
• Generation Tasks: ROUGE-L F1 for sentence-level comparison of full generated responses against references. BERTScore included as an additional evaluation metric (Appendix H.7).
Formal mathematical definitions and post-processing methods are detailed in Appendix D.2. To account for prompt sensitivity in LLMs, Appendix G compares different prompt templates.

4 Experiment

4.1 Experimental Setup
Models. We compared the performance of 26 LLMs on the SeedBench, including 7 proprietary models, such as the GPT series (Achiam et al., 2023), Gemini series (Team et al., 2024), Claude-3.5-Sonnet5, and GLM-4-Plus6, as well as 16 open-source models, including the Qwen series (Yang et al., 2024a), DeepSeek-V3 (Liu et al., 2024), Llama series (Dubey et al., 2024), InternLM series (Cai et al., 2024), GLM-4 (GLM et al., 2024), and Mistral (Jiang et al., 2023). Furthermore, we evaluated 3 domain-specific models, represented by the PLLaMa series (Yang et al., 2024b) and Ak-sara7. This comprehensive comparison not only highlights the relative strengths and limitations of each model group but also provides key insights for future research and application development.
Implementation Details. We evaluated the performance of all models in both zero-shot and one-shot settings. In zero-shot inference, the model input includes only the task instructions and the query. In one-shot inference, the model input consists of the task instructions, an example query with its answer, followed by the actual query. The experiments were conducted using the OpenCompass8 framework. For proprietary LLMs, we performed inference through their APIs. The evaluation of each model took approximately 1 hour. For open-source LLMs, the evaluation was conducted on 8 NVIDIA A100 40GB GPUs, with an average completion time of 0.5 hours. The inference hyper-parameters are detailed in Appendix D.1.

4.2 Performance Evaluation
Here we assess overall performance across breeding subcategories, as detailed in Table 29. Among proprietary LLMs, GPT-4 achieves the highest average score on SeedBench (62.06), followed by GLM-4-plus (59.61). In contrast, open-source models show a different ranking: DeepSeek-V3 leads with an average score of 63.3, outperforming Qwen2.5-14B, which scores 54.3. Notably, DeepSeek-V3, despite being a recently released model with 671B parameters, surpasses GPT-4 on SeedBench. On the other hand, the three domain-specific LLMs perform relatively poorly, likely due to their limited conversational and instruction following capabilities. Interestingly, OpenAI o1, despite demonstrating strong reasoning abilities in mathematics and coding, scores lower than GPT-4 on SeedBench. This suggests that its reasoning strategy does not transfer effectively to breeding-related tasks. A similar trend is observed in Gemini-2.0-Flash Thinking and QwQ-32B, both of which exhibit explicit reasoning steps in their responses yet achieve only 34.24 and 33.55, respectively. A more detailed comparative analysis is provided in Appendix H, where we examine performance variations within the same model series, evaluate models of identical sizes, and assess the impact of task difficulty. And Appendix E provides a holistic error taxonomy and analysis of models’ failures, summarizing eight primary causes of errors, such as Gene Name Confusion. These additional analyses collectively offer deeper insights into the factors influencing model performance on SeedBench.

4.3 Empirical Analysis
4.3.1 Analysis on Reasoning Ability
Table 2 shows that LLMs with explicit “reasoning mode” do not consistently outperform those without specialized chain-of-thought mechanisms (RQ1). For instance, models designed for multi-step reasoning, such as Gemini-2.0-Flash (34.24) and QwQ-32B (33.55), achieve lower average scores than general-purpose LLMs such as GPT-4 (62.06) and the top-performing open-source model, DeepSeek-V3-671B (63.30). This discrepancy is

5https://www.anthropic.com/
6https://open.bigmodel.cn/dev/api/normal-model/glm-4
7https://huggingface.co/cropinailab/aksara_v1
8https://opencompass.org.cn/home
9See Tables 10 11 12 in Appendix H.8 for complete results.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="10">Breeding Subcategories</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>C1</th>
    <th>C2</th>
    <th>C3</th>
    <th>C4</th>
    <th>C5</th>
    <th>C6</th>
    <th>C7</th>
    <th>C8</th>
    <th>C9</th>
    <th>C10</th>
  </tr>
  <tr>
    <th colspan="12">Proprietary LLMs</th>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>48.77</td>
    <td>57.72</td>
    <td>66.02</td>
    <td>57.54</td>
    <td>47.82</td>
    <td>49.36</td>
    <td>57.47</td>
    <td>60.11</td>
    <td>58.06</td>
    <td>58.89</td>
    <td>55.45</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>47.00</td>
    <td>59.55</td>
    <td>62.42</td>
    <td>59.56</td>
    <td>43.11</td>
    <td>49.55</td>
    <td>53.41</td>
    <td>56.18</td>
    <td>52.51</td>
    <td>53.71</td>
    <td>53.58</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td>33.67</td>
    <td>27.37</td>
    <td>53.04</td>
    <td>32.07</td>
    <td>25.87</td>
    <td>44.41</td>
    <td>33.57</td>
    <td>36.77</td>
    <td>31.78</td>
    <td>31.70</td>
    <td>34.24</td>
  </tr>
  <tr>
    <td>GLM-4-Plus</td>
    <td>52.72</td>
    <td>59.62</td>
    <td>70.62</td>
    <td>60.11</td>
    <td>50.60</td>
    <td>56.75</td>
    <td>65.02</td>
    <td>64.17</td>
    <td>61.70</td>
    <td>62.90</td>
    <td>59.61</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>59.59</td>
    <td>60.55</td>
    <td>76.32</td>
    <td>61.16</td>
    <td>56.34</td>
    <td>59.35</td>
    <td>63.67</td>
    <td>64.74</td>
    <td>60.65</td>
    <td>67.66</td>
    <td>62.06</td>
  </tr>
  <tr>
    <td>GPT-4o mini</td>
    <td>54.24</td>
    <td>56.64</td>
    <td>72.11</td>
    <td>59.28</td>
    <td>53.00</td>
    <td>57.88</td>
    <td>58.38</td>
    <td>61.75</td>
    <td>57.50</td>
    <td>62.38</td>
    <td>58.40</td>
  </tr>
  <tr>
    <td>OpenAI o1-mini</td>
    <td>49.16</td>
    <td>55.58</td>
    <td>59.37</td>
    <td>54.77</td>
    <td>44.43</td>
    <td>50.73</td>
    <td>54.57</td>
    <td>55.36</td>
    <td>54.91</td>
    <td>54.19</td>
    <td>53.25</td>
  </tr>
  <tr>
    <th colspan="12">Open-Source LLMs</th>
  </tr>
  <tr>
    <td>DeepSeek-V3-671B</td>
    <td>56.03</td>
    <td>62.42</td>
    <td>74.81</td>
    <td>63.17</td>
    <td>55.23</td>
    <td>58.84</td>
    <td>68.23</td>
    <td>69.04</td>
    <td>66.46</td>
    <td>68.48</td>
    <td>63.30</td>
  </tr>
  <tr>
    <td>GLM-4-Chat-9B</td>
    <td>23.28</td>
    <td>21.31</td>
    <td>39.97</td>
    <td>26.13</td>
    <td>16.20</td>
    <td>34.15</td>
    <td>26.63</td>
    <td>29.60</td>
    <td>25.60</td>
    <td>26.68</td>
    <td>26.55</td>
  </tr>
  <tr>
    <td>InternLM-2.7B</td>
    <td>27.55</td>
    <td>21.14</td>
    <td>39.64</td>
    <td>28.57</td>
    <td>15.16</td>
    <td>36.12</td>
    <td>28.74</td>
    <td>30.80</td>
    <td>27.32</td>
    <td>29.22</td>
    <td>28.71</td>
  </tr>
  <tr>
    <td>InternLM2.5-7B</td>
    <td>51.71</td>
    <td>55.75</td>
    <td>67.88</td>
    <td>50.48</td>
    <td>44.14</td>
    <td>56.73</td>
    <td>51.28</td>
    <td>54.91</td>
    <td>52.46</td>
    <td>56.24</td>
    <td>53.51</td>
  </tr>
  <tr>
    <td>Llama3.1-8B</td>
    <td>43.89</td>
    <td>31.21</td>
    <td>42.53</td>
    <td>40.68</td>
    <td>38.47</td>
    <td>43.80</td>
    <td>42.87</td>
    <td>51.62</td>
    <td>41.88</td>
    <td>49.01</td>
    <td>42.23</td>
  </tr>
  <tr>
    <td>Llama3.1-70B</td>
    <td>48.72</td>
    <td>55.41</td>
    <td>64.77</td>
    <td>53.67</td>
    <td>46.73</td>
    <td>54.08</td>
    <td>56.94</td>
    <td>57.72</td>
    <td>55.31</td>
    <td>57.56</td>
    <td>54.30</td>
  </tr>
  <tr>
    <td>Llama3.3-70B</td>
    <td>45.32</td>
    <td>47.15</td>
    <td>60.62</td>
    <td>49.76</td>
    <td>40.90</td>
    <td>54.30</td>
    <td>52.79</td>
    <td>54.61</td>
    <td>49.98</td>
    <td>55.05</td>
    <td>50.53</td>
  </tr>
  <tr>
    <td>Mistral-v0.3-7B</td>
    <td>42.61</td>
    <td>38.28</td>
    <td>57.02</td>
    <td>40.41</td>
    <td>29.97</td>
    <td>44.22</td>
    <td>36.31</td>
    <td>43.98</td>
    <td>39.92</td>
    <td>43.51</td>
    <td>41.59</td>
  </tr>
  <tr>
    <td>Qwen2-0.5B</td>
    <td>32.84</td>
    <td>25.15</td>
    <td>40.19</td>
    <td>28.20</td>
    <td>27.62</td>
    <td>37.22</td>
    <td>33.81</td>
    <td>33.63</td>
    <td>28.25</td>
    <td>31.67</td>
    <td>31.44</td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>44.21</td>
    <td>40.41</td>
    <td>63.00</td>
    <td>47.36</td>
    <td>35.37</td>
    <td>52.30</td>
    <td>45.61</td>
    <td>48.73</td>
    <td>44.88</td>
    <td>46.89</td>
    <td>46.51</td>
  </tr>
  <tr>
    <td>Qwen2-57B</td>
    <td>53.67</td>
    <td>49.81</td>
    <td>74.30</td>
    <td>58.38</td>
    <td>39.34</td>
    <td>54.71</td>
    <td>63.89</td>
    <td>59.57</td>
    <td>59.22</td>
    <td>60.08</td>
    <td>57.20</td>
  </tr>
  <tr>
    <td>Qwen2-72B</td>
    <td>51.16</td>
    <td>58.10</td>
    <td>74.07</td>
    <td>59.72</td>
    <td>51.58</td>
    <td>57.76</td>
    <td>58.85</td>
    <td>61.63</td>
    <td>56.69</td>
    <td>59.11</td>
    <td>57.62</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>45.16</td>
    <td>39.50</td>
    <td>66.01</td>
    <td>44.61</td>
    <td>35.72</td>
    <td>50.00</td>
    <td>53.60</td>
    <td>53.31</td>
    <td>53.06</td>
    <td>51.05</td>
    <td>48.45</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>50.91</td>
    <td>50.73</td>
    <td>68.62</td>
    <td>52.15</td>
    <td>47.14</td>
    <td>54.54</td>
    <td>57.02</td>
    <td>62.05</td>
    <td>54.37</td>
    <td>54.15</td>
    <td>54.21</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>46.86</td>
    <td>47.41</td>
    <td>70.99</td>
    <td>51.89</td>
    <td>46.17</td>
    <td>57.50</td>
    <td>55.35</td>
    <td>56.31</td>
    <td>53.05</td>
    <td>54.75</td>
    <td>52.63</td>
  </tr>
  <tr>
    <td>QwQ-32B</td>
    <td>32.24</td>
    <td>21.06</td>
    <td>47.11</td>
    <td>29.14</td>
    <td>28.56</td>
    <td>39.68</td>
    <td>38.17</td>
    <td>39.56</td>
    <td>34.70</td>
    <td>34.52</td>
    <td>33.55</td>
  </tr>
  <tr>
    <th colspan="12">Domain Specific LLMs</th>
  </tr>
  <tr>
    <td>Aksara-v1-7B</td>
    <td>36.72</td>
    <td>36.69</td>
    <td>48.32</td>
    <td>35.41</td>
    <td>24.26</td>
    <td>36.83</td>
    <td>31.17</td>
    <td>34.64</td>
    <td>31.15</td>
    <td>34.14</td>
    <td>35.04</td>
  </tr>
  <tr>
    <td>PLLaMa-7B</td>
    <td>17.85</td>
    <td>13.69</td>
    <td>17.99</td>
    <td>16.81</td>
    <td>11.66</td>
    <td>21.67</td>
    <td>14.34</td>
    <td>17.36</td>
    <td>12.39</td>
    <td>16.11</td>
    <td>16.46</td>
  </tr>
  <tr>
    <td>PLLaMa-13B</td>
    <td>15.10</td>
    <td>14.18</td>
    <td>28.41</td>
    <td>18.83</td>
    <td>13.96</td>
    <td>23.28</td>
    <td>18.53</td>
    <td>17.37</td>
    <td>14.15</td>
    <td>18.51</td>
    <td>17.57</td>
  </tr>
</table>

Table 2: Evaluation of 26 LLMs on SeedBench. Performance (averaged across both zero-shot and one-shot configurations) is stratified by breeding subcategories, with open-source/domain-specific models evaluated through 3 repeated trials (mean scores reported). The scores represent averages across three different metrics for 11 task types. The columns delineate ten subcategories in breeding: (C1) Gene Basic Information Query, (C2) Gene Expression Pattern Query, (C3) Gene Product Cellular Localization Query, (C4) Gene Function Experimental Observation, (C5) Gene Product Regulation of Downstream Genes Analysis, (C6) Gene Function Prediction, (C7) Variety Breeding Process Query, (C8) Variety Agronomic Trait Query, (C9) Variety Cultivation and Technical Key Points Query, (C10) Variety Suitable Planting Area Recommendation. Top-3 performers per column are highlighted in red. Extended results (including standard deviations, separate zero-shot/one-shot scores, and task-type breakdowns) are provided in Appendix H.8.

evident in subcategories that primarily involve fact retrieval or straightforward inference, such as Gene Basic Information Query (C1) and Variety Cultivation and Technical Key Points Query (C9).

Furthermore, the results suggest potential drawbacks of verbose chain-of-thought reasoning in tasks that require single-step inferences. Lengthy reasoning chains can introduce unnecessary content, potentially reducing performance on precision-based metrics such as ROUGE. This effect is particularly evident in tasks like Gene Product Regulation of Downstream Genes Analysis (C5), where the best-performing models (e.g., GPT-4 and DeepSeek-V3-671B) maintain conciseness while effectively capturing key information. Overall, these findings indicate that while reasoning-centric prompts may be advantageous in complex multi-step tasks (e.g., coding or mathematical problem-solving), seed breeding queries rely on direct knowledge retrieval or limited inference. Future research could focus on adjusting the reasoning strategy to match task complexity, dynamically adjusting reasoning chains based on task difficulty, rather than using complex reasoning for all tasks (Chen et al., 2024b).

4.3.2 Impact of Domain-Specific Fine-Tuning

Contrary to expectations, domain-specific fine-tuned models (e.g., Aksara-v1-7B, PLLaMa-7B, PLLaMa-13B) perform worse than general-purpose models on SeedBench. As shown in Table 2, these specialized models exhibit significantly lower overall scores (e.g., 35.04 for Aksara-v1-7B and 17.57 for PLLaMa-13B). In comparison, mid-tier open-source LLMs such as Llama3.1-70B (54.30) and Qwen2.5-14B (54.21) outper-
Figure 4: Performance vs. Model Size. We empirically validate scaling laws in seed breeding tasks, showing a logarithmic correlation between model size and average scores. The optimal model size for breeding tasks lies between 7B and 14B, balancing performance and computational efficiency.

form them, while top-performing general-purpose models like DeepSeek-V3-671B (63.30) and GPT-4 (62.06) achieve even higher scores. Notably, Aksara-v1-7B’s best subcategory score (48.32 in C3: Gene Product Cellular Localization Query) remains below many open-source models’ performance in similar tasks. We hypothesize that a key factor behind this underperformance is the deterioration of conversational and instruction-following capabilities, after fine-tuning on domain-specific data. In one-shot evaluations, PLLaMas frequently misinterpret prompts, treating in-context examples as direct queries. This shows that domain specialization may weaken general instruction adherence, which is essential for tasks requiring customized outputs and complex user interactions. Additionally, post-training on narrowly defined tasks or using data not validated by experts may further diminish models’ performance. Thus, we suggest that maintaining general-purpose abilities during fine-tuning, expanding the breadth of training data, and incorporating expert validation may effectively improve the performance of domain-specific fine-tuned models. Further investigations into the fine-tuning strategy and the scope and quality of the training corpus are required to fully answer RQ2.

4.3.3 Impact of Model Size
As shown in Figure 4, models with 7B–14B parameters achieve the best trade-off between performance and computational efficiency (RQ3). Models in this range, such as Qwen2.5-14B and InternLM-2.5-7B, perform robustly with manageable resource requirements. In contrast, smaller models (e.g., Qwen2-0.5B) underperform, and much larger models (e.g., Qwen2.5-72B) show diminishing returns. Additionally, the lack of performance improvement across scaled Qwen models (14B to 72B) suggests a distribution mismatch between our benchmark data and Qwen’s enlarged training corpus. We conjecture that the quality of training corpus is more important than model size for domain-specific tasks. In addition, we analyze performance differences between models of varying sizes, series, and subcategory difficulty in Appendices H.1 to H.6.

5 Discussion
The discrepancy between the seed scientists’ expectations and the reality of LLMs is evident. From the perspective of breeding experts, the immediate applicability of domain-specific fine-tuned models to seed science remains constrained. This may arise from factors such as training on narrowly defined tasks, reliance on data lacking expert validation, or catastrophic forgetting of general capabilities during fine-tuning. Conversely, while current general LLMs have shown good potential in text understanding and basic reasoning, they still fall short of meeting the deep and specialized requirements of actual breeding work. Several key gaps between seed scientists’ expectations and the capabilities of general LLMs are highlighted below.

Domain Depth. While LLMs cover a broad range of agricultural topics, their knowledge depth is insufficient for specialized breeding tasks. Complex issues like molecular breeding or trait introgression often require high-quality domain knowledge repositories, an area where current models are lacking. One potential solution is to integrate structured knowledge graphs mapping relationships across phenotypic, genomic, and environmental data.

Multimodal Integration. Breeding in practice heavily relies on sensory evaluations (such as detecting grain morphology, texture, and odor) and environmental data (such as climate and soil conditions). Current LLMs, primarily based on text input, cannot effectively integrate images, sensor readings, and field observations, limiting their performance in multimodal decision-making.

Explainability and Risk Management. The breeding decision cycle is costly and prolonged. Without transparent reasoning and risk evaluation mechanisms, misleading outputs could result in substantial losses. Producers must be able to ver-
ify and trace model conclusions to ensure safe and controllable implementation, e.g., via safe RL.

6 Conclusion

In this study, we propose SeedBench, the first multi-task LLM benchmark tailored for seed science. The contributions of SeedBench to AI for Science are twofold. First, it demonstrates the complete benchmark construction process for seed science, a knowledge-intensive field. The knowledge behind can be transferred to other science domains, such as life science or physical science, to build a comprehensive and reliable scientific benchmark. Second, it evaluates the capabilities of LLMs on addressing seed breeding tasks. The results not only provide insightful empirical findings on the tasks on interest, but also pose future research directions on designing scientific LLM. By bridging the gap between the power of LLMs and the real-world scientific problem in seed science, we aim to make a foundational effort for successful implementation of LLMs for seed design in the future.

Limitations

Given the highly specialized nature of breeding research, we primarily use the peer-reviewed scientific literature to ensure the accuracy and credibility of the data. Online sources often lack systematic review and professional validation. In the future, we will explore more reliable online databases and expert knowledge bases to further diversify our data sources.

There remains a gap between the expectations of the researchers and the actual capabilities of LLMs. Closing this gap calls for more domain-focused and in-depth professional datasets, as well as additional breeding-specific knowledge during model training. Another key direction is to incorporate sequential decision making and iterative learning into the models, so that they can adapt to planting cycles and experimental feedback. Additionally, developing LLMs that support multi-modal inputs—such as phenotypic, genomic, and environmental data—would be crucial for complex breeding scenarios. Finally, improving model interpretability and safety mechanisms will be essential for building trust when the model provides breeding recommendations.

Ethical Considerations

This research adheres to the ethical principles outlined in the ACL Code of Ethics. We have taken steps to ensure that our work does not cause harm, particularly in the context of seed science, where the implications of our research could impact agricultural practices and food security.

We have carefully considered the potential risks of our approach, especially with respect to the use of LLMs in agricultural research. One of the main ethical concerns we addressed is the potential for bias in LLMs, which could affect seed breeding decisions or lead to misinterpretations in agricultural data. To mitigate this, we evaluated multiple LLMs, to better understand their limitations and potential biases, and have explicitly highlighted areas where further improvement is needed.

In addition, we ensured that all data used in our experiments were sourced responsibly, with due regard for privacy and intellectual property. The artifacts we used are governed under the CC-BY 4.0, Apache 2.0 and MIT licenses, which support open and ethical use of such resources. The creation of the SeedBench benchmark involved collaboration with domain experts to ensure that it accurately represents the complexities of seed breeding, while minimizing any unintended consequences Moreover, we have taken care to ensure that the use of these artifacts aligns with the intended purpose of advancing research in seed breeding and AI applications.

Finally, we have considered the broader social impact of our work, recognizing that our research could influence farming practices, genetic resource management, and food security. We have outlined the necessary steps to avoid harm and ensure responsible application of our findings.

Acknowledgments

This work was supported by Shanghai Artificial Intelligence Laboratory, the Yazhouwan National Laboratory Project (grant no. 2310CF01), and the Hainan Yazhou Bay Seed Laboratory Project (grant no. B21HJ0001). The authors thank anonymous reviewers for their constructive comments. The authors also thank Yazhouwan National Laboratory for data collection and validation of human experts. The authors also thank Songyang Zhang from Shanghai Artificial Intelligence Laboratory for academic discussion.
References

Josh Achiham, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Muhammad Arbab Arshad, Talukder Zaki Jubery, Tirtho Roy, Rim Nassiri, Asheesh K Singh, Arti Singh, Chinmay Hegde, Baskar Ganapathysubramanian, Aditya Balu, Adarsh Krishnamurthy, et al. 2024. Ageval: A benchmark for zero-shot and few-shot plant stress phenotyping with multimodal llms. arXiv preprint arXiv:2407.19617.

Zheng Cai, Maosong Cao, Haojing Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Interllm2 technical report. arXiv preprint arXiv:2403.17297.

Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyian Huang, Yanzhou Su, et al. 2024a. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. arXiv preprint arXiv:2408.03361.

Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qizhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. 2024b. Do not think that much for 2+ 3=7 on the overthinking of ol-like llms. arXiv preprint arXiv:2412.21187.

Lawrence O Copeland and Miller F McDonald. 2012. Principles of seed science and technology. Springer Science & Business Media.

Djavan De Clercq, Elias Nehring, Harry Mayne, and Adam Mahdi. 2024. Large language models can help boost food production, but be mindful of their risks. Frontiers in Artificial Intelligence, 7:1326153.

Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Lucy M. Egan, Rainer W. Hofmann, Warwick N. Stiller, and Valerio Hoyos-Villegas. 2024. Cultivating success: Bridging the gaps in plant breeding training in australia, canada, and new zealand. Crop Science, 64(4).

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. 2024. LawBench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7933–7962, Miami, Florida, USA.

Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. 2023. Large language models in education: Vision and opportunities. In 2023 IEEE international conference on big data (BigData), pages 4776–4785. IEEE.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

K Iversen, HWJ Cheng, K Helgason, and M LaFleur. 2021. Frontier technologies for smallholder farmers: Addressing information asymmetries and deficiencies. Frontier Technology Issues.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Josué Kpodo, Parisa Kordjamshidi, and A Pouyan Nejadhashemi. 2024. Agxqa: A benchmark for advanced agricultural extension question answering. Computers and Electronics in Agriculture, 225:109349.

Matheus Thomas Kuska, Mirwaes Wahabzada, and Stefan Paulus. 2024. Ai for crop production—where can large language models (llms) provide substantial value? Computers and Electronics in Agriculture, 221:108924.

Dongyang Li, Ruixue Ding, Qiang Zhang, Zheng Li, Boli Chen, Pengjun Xie, Yao Xu, Xin Li, Ning Guo, Fei Huang, et al. 2023a. Geolgue: A geographic language understanding evaluation benchmark. arXiv preprint arXiv:2305.06545.

Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023b. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance, pages 374–382.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3225, Dublin, Ireland.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.
Ummara Muntaz, Awais Ahmed, and Summaya Muntaz. 2024. Llms-healthcare: Current applications and challenges of large language models in various medical specialties. Artificial Intelligence in Health, 1(2):16–28.

Alec Radford. 2018. Improving language understanding by generative pre-training.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems.

Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. 2024a. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839.

Liangdong Wang, Bo-Wen Zhang, Chengwei Wu, Hanyu Zhao, Xiaofeng Shi, Shuhao Gu, Jijie Li, Quanyue Ma, TengFei Pan, and Guang Liu. 2024b. Cci3. 0-hq: a large-scale chinese dataset of high quality designed for pre-training large language models. arXiv preprint arXiv:2410.18505.

Zeyu Wang. 2024. Causalbench: A comprehensive benchmark for evaluating causal reasoning capabilities of large language models. In Proceedings of the 10th SIGNAH Workshop on Chinese Language Processing, pages 143–151.

Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yuere He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. 2024. The finben: An holistic financial benchmark for large language models. arXiv preprint arXiv:2402.12659.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Xianjun Yang, Junfeng Gao, Wenxin Xue, and Erik Alexandersson. 2024b. Pllama: An open-source large language model for plant science. arXiv preprint arXiv:2401.01600.

Hang Zhang, Jiawei Sun, Renqi Chen, Wei Liu, Zhonghang Yuan, Xinze Zheng, Zhefan Wang, Zhiyuan Yang, Hang Yan, Han-Sen Zhong, et al. 2024. Empowering and assessing the utility of large language models in crop science. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, pages 52670–52722.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
Appendix

Table of Contents

A  Background Definitions .................................................................................. 13
    A.1  Definition of Breeding Competencies ......................................................... 13
    A.2  Example Definitions of Breeding Terminology ............................................. 14

B  Source Data Collection .................................................................................. 15
    B.1  Construction and Annotation of High-Quality Text Segments ....................... 15
    B.2  Language Composition and Impact ............................................................. 16

C  Quality Verification ....................................................................................... 16
    C.1  Manual Quality Verification .................................................................... 17
    C.2  Descriptive Statistics ............................................................................... 18

D  Additional Experimental Setup ....................................................................... 19
    D.1  Model Hyperparameters .......................................................................... 19
    D.2  Evaluation Metrics .................................................................................. 19

E  Error Analysis of Large Language Model Outputs ........................................... 21
    E.1  Errors in the First Task Type .................................................................. 21
    E.2  Errors in the Second Task Type ............................................................... 27
    E.3  Errors in the Third Task Type ................................................................. 31

F  Prompt Templates .......................................................................................... 34
    F.1  Multiple-Choice Question Generation (Task QA-1) ..................................... 34
    F.2  Multiple-Answer Question Generation (Task QA-2) .................................. 35
    F.3  Fill-in-the-Blank Question Generation (Task QA-3) .................................. 36
    F.4  Text-based Q&A Generation (Task QA-4) ................................................ 37
    F.5  Naive Summarization (Task SUM-1) ........................................................ 38
    F.6  Key Information Extraction (Task SUM-2) ................................................ 39
    F.7  Question Rewriting (Tasks RC-1, RC-2, RC-3, RC-4) ............................... 40

G  Robustness Evaluation of Prompts ................................................................. 41

H  Detailed Performance Comparison and Analysis ............................................. 49
    H.1  Model Comparison on Different LLMs ...................................................... 49
    H.2  Comparison Between Different Model Sizes of the Same Model Series ...... 49
    H.3  Top Performers in Each Subcategory ....................................................... 49
    H.4  Comparison of Models with Same Parameter Scale .................................. 49
    H.5  Task-Specific Performance ..................................................................... 50
    H.6  Conclusion ............................................................................................. 50
    H.7  Assessment of the Generative Tasks using BERTScore ............................ 51
    H.8  Additional Quantitative Results ............................................................... 52
A  Background Definitions

A.1  Definition of Breeding Competencies

We define in Table 3 the core breeding competencies required to address complex, knowledge-intensive breeding tasks. The breeding process is divided into three steps: the acquisition and analysis of genetic information, the exploration of gene function and regulatory mechanisms, and variety selection alongside agronomic trait optimization. Each stage corresponds to specific knowledge and skill requirements.

<table>
  <tr>
    <th>Stage</th>
    <th>Corresponding Task</th>
    <th>Competency</th>
    <th>Description</th>
  </tr>
  <tr>
    <td rowspan="3">Acquisition and Analysis of Genetic Information</td>
    <td>Basic Gene Information Retrieval</td>
    <td>Ability to Retrieve Basic Gene Information</td>
    <td>Is able to retrieve and integrate a gene’s basic information (e.g., sequence, functional annotation, chromosomal location) based on user inputs such as gene names or identifiers.</td>
  </tr>
  <tr>
    <td>Gene Expression Pattern Retrieval</td>
    <td>Ability to Analyze Gene Expression Patterns</td>
    <td>Possesses the capacity to explore and interpret gene expression data under different tissues, developmental stages, or environmental conditions.</td>
  </tr>
  <tr>
    <td>Subcellular Localization of Gene Products</td>
    <td>Ability to Determine Gene Product Localization and Characteristics</td>
    <td>Uses existing protein or molecular data to infer the specific location of a gene product within the cell (e.g., nucleus, cytoplasm, cell membrane) and, by applying biological knowledge, deduces its potential functions (e.g., transcriptional regulation, signal transduction, material transport).</td>
  </tr>
  <tr>
    <td rowspan="3">Exploration of Gene Function and Regulatory Mechanisms</td>
    <td>Observations from Gene Function Experiments</td>
    <td>Ability to Interpret Experimental Results on Gene Function</td>
    <td>Accurately describes how a gene influences plant traits under specific conditions.</td>
  </tr>
  <tr>
    <td>Analysis of Gene Product Regulation on Downstream Genes</td>
    <td>Ability to Analyze Gene Regulatory Networks and Downstream Genes</td>
    <td>Relies on existing research or deductive reasoning to evaluate the regulatory effects of a target gene product on downstream genes or related pathways.</td>
  </tr>
  <tr>
    <td>Gene Function Prediction</td>
    <td>Ability to Predict and Validate Unknown Gene Functions</td>
    <td>Draws on known gene sequences, expression characteristics, and analogies with characterized genes to predict functions of genes that have not yet been extensively studied.</td>
  </tr>
  <tr>
    <td rowspan="4">Variety Selection and Agronomic Trait Optimization</td>
    <td>Retrieval of Variety Breeding Processes</td>
    <td>Ability to Oversee Variety Breeding Processes</td>
    <td>Collects and summarizes the breeding history, methods, and improvement goals for a specific variety.</td>
  </tr>
  <tr>
    <td>Querying Agronomic Traits of Varieties</td>
    <td>Ability to Screen and Evaluate Agronomic Traits</td>
    <td>Compares the agronomic traits (e.g., disease resistance, yield, drought tolerance) of a target variety or multiple varieties to assess their value in practical production, selecting the most suitable breeding materials or gene combinations.</td>
  </tr>
  <tr>
    <td>Querying Key Points of Variety Cultivation and Techniques</td>
    <td>Ability to Plan Variety Cultivation Management and Key Technical Points</td>
    <td>Gathers and analyzes crucial information on cultivation management (e.g., fertilization strategies, irrigation schedules, pest control) to optimize field trials or practical planting outcomes.</td>
  </tr>
  <tr>
    <td>Recommending Suitable Planting Regions for Varieties</td>
    <td>Ability to Evaluate and Recommend Planting Regions</td>
    <td>Takes into account environmental factors and agronomic traits to propose suitable planting regions for a given variety, while evaluating its potential for wider application.</td>
  </tr>
</table>

Table 3: Core Breeding Competencies
A.2 Example Definitions of Breeding Terminology

We define in Table 4 the specialized breeding terms mentioned in this paper, along with explanations to provide a unified reference for subsequent experimental design, task classification, and result interpretation.

<table>
  <tr>
    <th>Terminology</th>
    <th>Definition</th>
  </tr>
  <tr>
    <td><b>Gene Information</b></td>
    <td>Refers to the fundamental data of a given gene, including its sequence, structure, functional descriptions, and expression patterns. This information is typically obtained from genomic databases or sequencing results, serving as the starting point for subsequent gene function research and variety improvement.</td>
  </tr>
  <tr>
    <td><b>Gene Expression Pattern</b></td>
    <td>Describes a gene’s expression levels and dynamic changes across various tissues, developmental stages, or environmental conditions. By analyzing this pattern, one can determine the gene’s potential impact on target traits (e.g., drought tolerance).</td>
  </tr>
  <tr>
    <td><b>Gene Regulatory Network</b></td>
    <td>A molecular regulatory system formed by interactions among gene products (e.g., transcription factors, enzymes). It determines how crops respond to environmental stimuli or developmental requirements and serves as an essential foundation for precision breeding analyses.</td>
  </tr>
  <tr>
    <td><b>Hybridization & Back-crossing</b></td>
    <td>Involves breeding new offspring by crossing or pollinating different parental lines. Backcrossing is carried out with a desirable parent line over multiple generations to stabilize or strengthen specific traits. This approach is traditional yet effective, often combined with modern molecular marker techniques.</td>
  </tr>
  <tr>
    <td><b>Agronomic Trait</b></td>
    <td>Refers to key field performance characteristics of crops, such as plant height, number of tillers, yield, disease resistance, and maturity. These traits are crucial indicators for evaluating the economic value and potential for broader adoption of new varieties.</td>
  </tr>
  <tr>
    <td><b>Marker-Assisted Breeding</b></td>
    <td>Uses molecular markers (e.g., SNP, SSR) for genotypic screening to accelerate the breeding process and improve selection accuracy. This method is frequently employed during hybridization and backcrossing to rapidly pinpoint genes associated with targeted traits.</td>
  </tr>
  <tr>
    <td><b>LLM-Aided Breeding</b></td>
    <td>Leverages large language models (e.g., GPT) to analyze and reason over literature, databases, and experimental data, supporting gene information analysis, gene function prediction, and variety selection decisions. It is expected to shorten breeding cycles and enhance breeding efficiency.</td>
  </tr>
</table>

Table 4: Breeding Terminology Definitions
B  Source Data Collection

B.1  Construction and Annotation of High-Quality Text Segments

In Figure 5 below, Content is a snippet extracted by agricultural experts from selected scientific publications. Example Question is an illustrative question provided by the experts, demonstrating from which perspective or angle one may inquire about the snippet. Classification indicates the subfield of agriculture to which the snippet belongs, and Reference identifies the scientific publication from which the snippet originates.

Content:
A pathway composed of five subunits of the heterotrimeric G proteins that regulate grain length in rice. The Gβ protein is essential for plant survival and growth. Go provides a foundation for grain size expansion. Three Gy proteins, DEP1, GGC2 and GS3, antagonistically regulate grain size. DEP1 and GGC2, individually or in combination, increase grain length when in complex with Gβ. GS3, having no effect on grain size by itself, reduces grain length by competitively interacting with Gβ.
Example Question:
what rice gene GS3 (AGIS_Os12g031280) is?
Classification:
Basic Gene Information Query
Reference:
A G-protein pathway determines grain size in rice

Content:
Field tests of these plant lines in Beijing in 2018 revealed that OsDREB1C overexpression led to increases in grain yield per plant of 45.1 to 67.6% and in yield per plot of 41.3 to 68.3% compared with wild-type (WT) plants (Fig. 1C). Conversely, OsDREB1C KO resulted in yield decreases (from 16.1 to 29.1% in yield per plant and 13.8 to 27.8% in yield per plot) compared with the WT (Fig. 1, D and E, and table S1). A detailed phenotypic analysis showed that the higher yield of the OsDREB1C-OE lines was mainly attributable to an enhanced grain number per panicle and an increased 1000-grain weight (Fig. 1F and fig. S3C), traits apparently resulting from increased secondary branch number and grain length, width, thickness, and density (fig. S3, A and B and D to K, respectively). The OsDREB1C-OE plants exhibited higher grain yield but reduced strawweight compared with WT plants (Fig. 1G), thus leading to an increased harvest index (the ratio of grain yield to aboveground biomass; Fig. 1H) and raising the possibility that OsDREB1C controls resource allocation between vegetative and reproductive tissues. The harvest index of OsDREB1C-OE plants was increased by 40.3 to 55.7%, whereas it was decreased by 22.4 to 33.7% in OsDREB1C-KO plants (table S1). In addition, key grain quality traits were enhanced in OsDREB1C-OE plants, suggesting that yield improvement does not entail a quality penalty (table S2). Leaves of OsDREB1C-OE plants contained higher levels of photosynthetic pigments (chlorophylls and carotenoids) compared with WT plants, where-as pigment levels were reduced in OsDREB1C-KO plants (fig. S7A). Analysis of leaf mesophyll cells revealed that both chloroplast number and size were increased in OsDREB1C-OE plants( fig.S7, Band C).. Analysis of carbon and nitrogen distribution showed that the OsDREB1C-OE plants accumulated more carbon and nitrogen in the grains, but less in their mature leaves, without substantial alterations in the carbon-to-nitrogen ratio (Fig. 3H and fig. S11).
Example Question:
What phenotypes do transgenic plants, rice plants knockouting OsDREB1C genes or plants overexpressing OsDREB1C genes exhibit?
Classification:
Gene Function Experiment Observation
Reference:
A transcriptional regulator that boosts grain yields and shortens the growth duration of rice

Figure 5: An illustration of the Content, Example Question, Classification, and Reference fields.
B.2 Language Composition and Impact

(1) The initial corpus of 308,727 articles comprises 63% English and 37% Chinese, with this imbalance reflecting the greater availability of English publications; (2) After cleaning, the 1.1 billion-token corpus consists of 75% English and 25% Chinese, with this shift due to the higher accuracy of the MinerU in processing English texts; (3) The final 279 segments used in SeedBench include 49% English and 51% Chinese, achieving balance through manual selection by breeding experts; (4) The 2,264 questions in SeedBench include 45% English and 55% Chinese.

Although LLMs exhibit strong cross-lingual capabilities and the linguistic differences in breeding questions do not alter the underlying scientific logic, we observed response drift when posing the same question in English and Chinese (with cleared histories). This phenomenon suggests potential issues in cross-lingual consistency, which deserves further research—especially when applying LLMs in specific domains where alignment across languages is critical:

• Question1-EN: What effect did the overexpression of OsDREB1C have on the levels of photosynthetic pigments in the leaves of the plants?
  A. Increased pigment levels
  B. Decreased pigment levels
  C. Pigment levels fluctuated unpredictably
  D. No significant change in pigment levels

• Question1-CN: OsDREB1C 的过表达对植物叶片中光合色素水平有何影响?
  A. 增加色素水平
  B. 减少色素水平
  C. 色素水平不可预测地波动
  D. 色素水平没有显著变化

• Question2-EN: The expression profile of OsDT11 in different rice tissues was analyzed by ________________.

• Question2-CN: OsDT11 在不同水稻组织中的表达谱通过______________分析。

<table>
  <tr>
    <th>Model</th>
    <th>Ans1-EN</th>
    <th>Ans1-CN</th>
    <th>Ans2-EN</th>
    <th>Ans2-CN</th>
  </tr>
  <tr>
    <td>DeepSeek-V3-671B</td>
    <td>A</td>
    <td>A</td>
    <td>qRT-PCR</td>
    <td>实时荧光定量PCR</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>A</td>
    <td>A</td>
    <td>qRT-PCR</td>
    <td>转录组学</td>
  </tr>
  <tr>
    <td>OpenAI o1-mini</td>
    <td>A</td>
    <td>A</td>
    <td>qRT-PCR</td>
    <td>Northern blot分析</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>A</td>
    <td>A</td>
    <td>qRT-PCR</td>
    <td>qRT-PCR</td>
  </tr>
</table>

Table 5: Model responses to parallel English and Chinese domain-specific questions.

C Quality Verification

The manual QA verification process involves a comprehensive assessment by experts to ensure the clarity, relevance, and rationality of questions. This includes evaluating whether the questions are designed to elicit accurate and valuable answers. Experts also check the correctness of the answers by consulting authoritative sources and applying professional knowledge to ensure they are error-free. Additionally, for questions that include multiple-answer options, it is essential to assess the rationality of these options. This assessment guarantees a clear distinction between them and ensures their close relevance to the correct answer. To enhance transparency regarding the expert correction process and the resolution of disagreements, we outline the following three points: (1) The composition of the expert panel, consisting of six Ph.D.-level experts in seed breeding; (2) An iterative review process, wherein each question is independently evaluated by at least two experts; (3) A disagreement resolution mechanism that addresses subjectivity in expert assessments, resolved by adopting the intersection of differing expert corrections. Additionally, we have open-sourced samples discarded during the expert correction process as “bad cases” on GitHub, offering readers further insight and reference.
C.1 Manual Quality Verification

Question:
The results of regional trials can provide feedback to breeders, helping them understand the extent to which their breeding goals have been achieved and guiding the breeding direction for ________.
Answer:
the future

Expert Review:
This question does not assess any knowledge points related to Seed Science and Breeding.

Question:
Which disease is Ningxiangjing 9 (rice variety) resistant to?
A. Bacterial leaf blight of rice
B. Rice sheath blight
C. Rice stripe leaf blight
D. Rice blast
Answer:
C

Expert Review:
The question cannot be answered precisely and should be phrased as “Which disease is Ningxiangjing 9 most resistant to?”

Question:
What is the main focus of this study?
Answer:
The main focus of this study is to clone a dominant gene for resistance to BPH (Brown Planthopper) from the rice variety PtB33.

Expert Review:
The question contains an unclear reference error that cannot be understood as there is no information about “this study” in the context.
Question:
In which region is Yuanfengzao (rice variety) suitable for planting?
A. Pearl River Basin
B. Yangtze River Basin
C. Heilongjiang River Basin
D. Yellow River Basin
Answer:
B

Expert Review:
The options provided are not appropriate as there is little difference in the suitability of planting Yuanfengzao in the Yangtze River Basin, the Pearl River Basin, and the Yellow River Basin.
The QA should be transformed into:
Question:
In which region is Yuanfengzao suitable for planting?
A. Songhua River Basin
B. Liao River Basin
C. Hai River Basin
D. Yangtze River Basin
Answer:
D

C.2 Descriptive Statistics

Table 6: Descriptive Statistics of Generated Questions and Answers

<table>
  <tr>
    <th></th>
    <th>C1</th>
    <th>C2</th>
    <th>C3</th>
    <th>C4</th>
    <th>C5</th>
    <th>C6</th>
    <th>C7</th>
    <th>C8</th>
    <th>C9</th>
    <th>C10</th>
    <th>total</th>
  </tr>
  <tr>
    <td>QA-1</td>
    <td>37<br>(1.63%)</td>
    <td>9<br>(0.4%)</td>
    <td>0<br>(0.0%)</td>
    <td>23<br>(1.02%)</td>
    <td>2<br>(0.09%)</td>
    <td>33<br>(1.46%)</td>
    <td>15<br>(0.66%)</td>
    <td>32<br>(1.41%)</td>
    <td>38<br>(1.68%)</td>
    <td>11<br>(0.49%)</td>
    <td>200<br>(8.83%)</td>
  </tr>
  <tr>
    <td>QA-2</td>
    <td>37<br>(1.63%)</td>
    <td>10<br>(0.44%)</td>
    <td>5<br>(0.22%)</td>
    <td>44<br>(1.94%)</td>
    <td>2<br>(0.09%)</td>
    <td>26<br>(1.15%)</td>
    <td>17<br>(0.75%)</td>
    <td>11<br>(0.49%)</td>
    <td>9<br>(0.4%)</td>
    <td>26<br>(1.15%)</td>
    <td>187<br>(8.26%)</td>
  </tr>
  <tr>
    <td>QA-3</td>
    <td>38<br>(1.68%)</td>
    <td>32<br>(1.41%)</td>
    <td>0<br>(0.0%)</td>
    <td>34<br>(1.5%)</td>
    <td>5<br>(0.22%)</td>
    <td>13<br>(0.57%)</td>
    <td>20<br>(0.88%)</td>
    <td>23<br>(1.02%)</td>
    <td>34<br>(1.5%)</td>
    <td>25<br>(1.1%)</td>
    <td>224<br>(9.89%)</td>
  </tr>
  <tr>
    <td>QA-4</td>
    <td>66<br>(2.92%)</td>
    <td>12<br>(0.53%)</td>
    <td>14<br>(0.62%)</td>
    <td>47<br>(2.08%)</td>
    <td>5<br>(0.22%)</td>
    <td>34<br>(1.5%)</td>
    <td>14<br>(0.62%)</td>
    <td>16<br>(0.71%)</td>
    <td>14<br>(0.62%)</td>
    <td>20<br>(0.88%)</td>
    <td>242<br>(10.69%)</td>
  </tr>
  <tr>
    <td>SUM-1</td>
    <td>39<br>(1.72%)</td>
    <td>17<br>(0.75%)</td>
    <td>6<br>(0.27%)</td>
    <td>36<br>(1.59%)</td>
    <td>4<br>(0.18%)</td>
    <td>25<br>(1.1%)</td>
    <td>20<br>(0.88%)</td>
    <td>26<br>(1.15%)</td>
    <td>28<br>(1.24%)</td>
    <td>24<br>(1.06%)</td>
    <td>225<br>(9.94%)</td>
  </tr>
  <tr>
    <td>SUM-2</td>
    <td>39<br>(1.72%)</td>
    <td>17<br>(0.75%)</td>
    <td>6<br>(0.27%)</td>
    <td>36<br>(1.59%)</td>
    <td>4<br>(0.18%)</td>
    <td>25<br>(1.1%)</td>
    <td>20<br>(0.88%)</td>
    <td>26<br>(1.15%)</td>
    <td>28<br>(1.24%)</td>
    <td>24<br>(1.06%)</td>
    <td>225<br>(9.94%)</td>
  </tr>
  <tr>
    <td>RC-1</td>
    <td>25<br>(1.1%)</td>
    <td>7<br>(0.31%)</td>
    <td>0<br>(0.0%)</td>
    <td>15<br>(0.66%)</td>
    <td>1<br>(0.04%)</td>
    <td>17<br>(0.75%)</td>
    <td>9<br>(0.4%)</td>
    <td>16<br>(0.71%)</td>
    <td>17<br>(0.75%)</td>
    <td>6<br>(0.27%)</td>
    <td>113<br>(4.99%)</td>
  </tr>
  <tr>
    <td>RC-2</td>
    <td>18<br>(0.8%)</td>
    <td>4<br>(0.18%)</td>
    <td>4<br>(0.18%)</td>
    <td>32<br>(1.41%)</td>
    <td>1<br>(0.04%)</td>
    <td>15<br>(0.66%)</td>
    <td>7<br>(0.31%)</td>
    <td>6<br>(0.27%)</td>
    <td>7<br>(0.31%)</td>
    <td>14<br>(0.62%)</td>
    <td>108<br>(4.77%)</td>
  </tr>
  <tr>
    <td>RC-3</td>
    <td>38<br>(1.68%)</td>
    <td>32<br>(1.41%)</td>
    <td>0<br>(0.0%)</td>
    <td>33<br>(1.46%)</td>
    <td>5<br>(0.22%)</td>
    <td>13<br>(0.57%)</td>
    <td>20<br>(0.88%)</td>
    <td>23<br>(1.02%)</td>
    <td>34<br>(1.5%)</td>
    <td>23<br>(1.02%)</td>
    <td>221<br>(9.76%)</td>
  </tr>
  <tr>
    <td>RC-4</td>
    <td>65<br>(2.87%)</td>
    <td>12<br>(0.53%)</td>
    <td>14<br>(0.62%)</td>
    <td>47<br>(2.08%)</td>
    <td>5<br>(0.22%)</td>
    <td>33<br>(1.46%)</td>
    <td>14<br>(0.62%)</td>
    <td>16<br>(0.71%)</td>
    <td>14<br>(0.62%)</td>
    <td>20<br>(0.88%)</td>
    <td>240<br>(10.6%)</td>
  </tr>
  <tr>
    <td>RC-5</td>
    <td>58<br>(2.56%)</td>
    <td>26<br>(1.15%)</td>
    <td>24<br>(1.06%)</td>
    <td>43<br>(1.9%)</td>
    <td>4<br>(0.18%)</td>
    <td>26<br>(1.15%)</td>
    <td>20<br>(0.88%)</td>
    <td>26<br>(1.15%)</td>
    <td>28<br>(1.24%)</td>
    <td>24<br>(1.06%)</td>
    <td>279<br>(12.32%)</td>
  </tr>
  <tr>
    <td><b>total</b></td>
    <td><b>460<br>(20.32%)</b></td>
    <td><b>178<br>(7.86%)</b></td>
    <td><b>73<br>(3.22%)</b></td>
    <td><b>390<br>(17.23%)</b></td>
    <td><b>38<br>(1.68%)</b></td>
    <td><b>260<br>(11.48%)</b></td>
    <td><b>176<br>(7.77%)</b></td>
    <td><b>221<br>(9.76%)</b></td>
    <td><b>251<br>(11.09%)</b></td>
    <td><b>217<br>(9.58%)</b></td>
    <td><b>2264<br>(100.0%)</b></td>
  </tr>
</table>
D Additional Experimental Setup

D.1 Model Hyperparameters

We conduct evaluations on different large language models using opencompass as the primary tool. In order to ensure reproducibility and provide a reference for future research, the main hyperparameters used in our experiments, along with their meanings, are listed in Table 7:

<table>
  <tr>
    <th>Parameter</th>
    <th>Meaning</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>max_seq_len</td>
    <td>The maximum context length (upper limit of input tokens)</td>
    <td>7168</td>
  </tr>
  <tr>
    <td>max_out_len</td>
    <td>The maximum output length (upper limit of generated tokens)</td>
    <td>2048</td>
  </tr>
  <tr>
    <td>batch_size</td>
    <td>The batch size (number of requests processed per generation)</td>
    <td>80</td>
  </tr>
</table>

For proprietary LLMs, the maximum generation tokens for each model were set to 2048, with a batch size of 80. The temperature was set to 0.7, and top-\( p \) and top-\( k \) were set to 0.8 and 10, respectively. For open-source LLMs, the maximum generation tokens were also set to 2048, with a batch size of 80. The temperature was set to 0.7, and top-\( p \) and top-\( k \) were set to 0.8 and 10, respectively.

D.2 Evaluation Metrics

The evaluation process consists of two steps: the first step is answer extraction. After collecting the model’s responses, we first perform post-processing to extract the model’s replies. The second step is metric calculation. During evaluation, we have designed different evaluation methods for different question types:

Single-choice Questions. We use Accuracy as the evaluation metric to calculate the correctness of the model’s answers. The formula for Accuracy is:

\[
\text{Accuracy} = \frac{1}{N_{\text{total}}} \sum_{i=1}^{N_{\text{total}}} \delta(y_i^{\text{pred}}, y_i^{\text{true}})
\]

where \( N_{\text{total}} \) is the total number of questions, \( y_i^{\text{pred}} \) is the predicted answer for the \( i \)-th question by the model, and \( y_i^{\text{true}} \) is the true answer for the \( i \)-th question. \( \delta(a, b) \) is the Kronecker delta function, where \( \delta(a, b) = 1 \) if \( a = b \), and \( \delta(a, b) = 0 \) otherwise.

Multiple-choice Questions. We use the F1 score, which balances the correct and incorrect answers from the model to provide a more comprehensive reflection of model performance. The formula for the F1 score is:

\[
F1 = 2 \cdot \frac{P \cdot R}{P + R}
\]

where precision (\( P \)) is:

\[
P = \frac{1}{N} \sum_{i=1}^{N} \frac{|Y_i^{\text{pred}} \cap Y_i^{\text{true}}|}{|Y_i^{\text{pred}}|}
\]

and recall (\( R \)) is:

\[
R = \frac{1}{N} \sum_{i=1}^{N} \frac{|Y_i^{\text{pred}} \cap Y_i^{\text{true}}|}{|Y_i^{\text{true}}|}
\]

where \( N \) is the total number of samples, \( Y_i^{\text{pred}} \) is the set of predicted answers for the \( i \)-th sample, and \( Y_i^{\text{true}} \) is the set of true answers for the \( i \)-th sample.
Fill-in-the-Blank and Generation Questions. We use ROUGE for evaluation and calculate the average F1 score of ROUGE-L. For fill-in-the-blank questions, since answers typically contain multiple blanks or paragraphs, we adopt a segmented evaluation method, comparing the similarity of each segment between the model’s predicted answer and the reference answer, and averaging the scores of multiple segments to more accurately reflect the model’s performance for each answer item. For generation questions, we use a sentence-level evaluation method, directly calculating the similarity between the model’s generated full answer and the reference answer. This method allows for a comprehensive evaluation of the overall fluency and semantic accuracy of the generated content.

The formula for ROUGE-L F1 is:

\[
\text{ROUGE-L} = \frac{(1 + \beta^2) \cdot P \cdot R}{\beta^2 \cdot P + R}
\]

where precision (\(P\)) is:

\[
P = \frac{1}{N} \sum_{i=1}^{N} \frac{\text{LCS}(X_i, Y_i)}{|X_i|}
\]

and recall (\(R\)) is:

\[
R = \frac{1}{N} \sum_{i=1}^{N} \frac{\text{LCS}(X_i, Y_i)}{|Y_i|}
\]

where \(N\) is the total number of samples, \(X_i\) and \(Y_i\) represent the predicted and reference answers for the \(i\)-th sample, and \(\text{LCS}(X_i, Y_i)\) is the length of the longest common subsequence between \(X_i\) and \(Y_i\). \(|X_i|\) and \(|Y_i|\) are the lengths of \(X_i\) and \(Y_i\), respectively. \(\beta\) is the balance factor between precision and recall, typically set to 1.
E  Error Analysis of Large Language Model Outputs

In this chapter, we thoroughly analyze the errors produced by large language models in different scenarios, according to the three task types defined above. By presenting typical cases and discussing both the causes and potential improvements, we provide a reference for optimizing the quality of LLM outputs in subsequent work.

E.1  Errors in the First Task Type

The first task type primarily involves gene information retrieval capabilities. The model needs to demonstrate fundamental biological knowledge as well as accurate information retrieval. We observe several representative error cases, outlined below:

Gene Name Confusion  When dealing with similar or homonymous gene names or gene identifiers, the model incorrectly provides the wrong gene sequences and functional descriptions. This mistake can lead to misalignment in subsequent gene function validation.

Example 1

Example Question:
What is the identifier of the OsAL50 gene in The Rice Annotation Project Database (RAP-DB)?
Reference:
Os01g0306800
Answer:
Os03g0559000
Analysis:
The model mistakenly identified "Os01g0306800" as "Os03g0559000," resulting in gene identifier confusion.

Example 2

Example Question:
GWD1 gene is the same gene as ________?
Reference:
LSE1 (line leaf starch excess 1)
Answer:
GWD2
Analysis:
The model incorrectly identified GWD1 and GWD2 as the same gene, leading to confusion in gene naming/relationships.

Example 3

Example Question:
What is the identifier of the GWD1 gene in The Rice Annotation Project Database (RAP-DB)?
Reference:
Os06g0498400
Answer:
GRS0000G0001999, GRS0000G0002000, GRS0000G0002001
Analysis:
The model returned a set of non-existent identifiers (which also do not conform to the RAP-DB format), causing gene identifier confusion.
Example 4

Example Question:
What is the identifier of the OsWOX3A gene in The Rice Annotation Project Database (RAP-DB)?
Reference:
Os12g0101600
Answer:
Os03g0250600
Analysis:
The model incorrectly recognized Os12g0101600 as Os03g0250600, leading to gene identifier confusion.

Example 5

Example Question:
The official name of OsCS1 is ________.
Reference:
citrate synthase 1
Answer:
OsCS1's official name is OsCSI1, OsCs1, OsCSI1 gene
Analysis:
The model confused "citrate synthase 1" with abbreviations like "CSI1," leading to incorrect gene naming.

Example 6

Example Question:
________ is a functional homolog of AtHCAR.
Reference:
OsHCAR
Answer:
OsHCAR1, OsHCAR2, OsHCAR3
Analysis:
The model provided several variant names, none of which are the exact homolog of AtHCAR, indicating gene name confusion.

Example 7

Example Question:
What is the identifier of the Albino midrib 1 (AM1) gene in The Rice Annotation Project Database (RAP-DB)?
Reference:
Os04g0682800
Answer:
GRMZM2G094044
Analysis:
The correct identifier should be "Os04g0682800," but the model returned "GRMZM2G094044," which appears to be a maize gene identifier, indicating severe gene identifier confusion.

Figure 6: Example illustrating gene name confusion.
Errors in Gene Sequence and Positional Information   The model provides incorrect answers regarding the physical location information of genes (chromosome number, start and end coordinates, sequence length, etc.), such as incorrect gene coordinates or sequence length discrepancies with actual databases. This may impact genome annotation and structural variation analysis, leading to experimental design biases.

Example 1

Example Question:
OsAL50 gene plays an important role in ________.
Reference:
Regulation of photosynthetic pigment metabolism, photosynthesis, and chloroplast formation
Answer:
scutellarein pathway, decarboxylation pathway
Analysis:
The model only answered "Rice," deviating from the actual functional or process description. The question expects an answer related to physiological metabolism or processes, but the model provided a more general species name.

Example 2

Example Question:
What protein does the OsAL50 gene encode?
Reference:
Endoribonuclease E-like protein
Answer:
Rice disease resistance-related protein
Analysis:
The model confused the specific protein type encoded by the gene, providing an entirely different functional protein description.

Example 3

Example Question:
What enzyme does the GWD1 gene encode?
Reference:
α-glucan water dikinase
Answer:
Glutamine synthetase
Analysis:
The model confused enzyme categories, incorrectly answering "glutamine synthetase" instead of "α-glucan water dikinase," leading to a serious functional misjudgment.

Example 4

Example Question:
NAL2 (narrow leaf2) and NAL3 are paralogs. What transcriptional activator do they encode?
Reference:
OsWOX3A
Answer:
NAC-domain-containing protein
Analysis:
The model mistakenly assumed that the gene encodes a NAC family transcription factor rather than a WOX family gene product, resulting in a functional description deviation.
Example 5

Example Question:
AtHCAR is a key enzyme in the ________ pathway for Chl degradation.
Reference:
PAO/phyllobilin
Answer:
OsCSI1’s official name is OsCSI1, OsCs1, OsCSI1 gene
Analysis:
The model confused the major pathways for chlorophyll degradation, providing irrelevant or rare metabolic pathway names.

Example 6

Example Question:
AtHCAR catalyzes the conversion of ________ to Chl a in vivo and in vitro.
Reference:
7-HMC a
Answer:
protochlorophyllide, Pchlde, …
Analysis:
The model mistakenly confused the substrate "7-HMC a" with another chlorophyll precursor "Pchlde," leading to inaccurate functional descriptions.

Example 7

Example Question:
The athcar mutant shows a ________ phenotype during DIS.
Reference:
stay-green
Answer:
dwarfed, sterile flowering
Analysis:
The model incorrectly described the mutant phenotype, misidentifying "stay-green" as "dwarfed, sterile flowering," which are unrelated phenotypes.

Figure 7: Example illustrating cross-species gene information errors.
Errors in Gene Function and Regulation   The model exhibits errors in describing gene functions, protein products, or regulatory mechanisms, such as misclassifying gene functions, misunderstanding their roles in signaling pathways, or incorrectly predicting protein products. These issues may mislead researchers’ understanding of gene biological functions, potentially leading to incorrect downstream experiments and data analysis.

Example 1

Example Question:
On which chromosome is the OsAL50 gene located in rice? What is its specific position range?
Reference:
Chromosome 1
Answer:
Chromosome 4
Analysis:
The model made an error in answering the chromosome location, confusing the correct chromosome.

Example 2

Example Question:
What is the full sequence length of the OsAL50 gene?
Reference:
8917 bp
Answer:
1199
Analysis:
The model provided an incorrect sequence length number, which is significantly shorter than the actual length.

Example 3

Example Question:
On which chromosome strand is the GWD1 gene located?
Reference:
Chromosome 6
Answer:
5
Analysis:
The model provided an incorrect chromosome number.

Example 4

Example Question:
What is the full name of the OsUAM3 gene?
Reference:
UDP-ARABINOPYRANOSE MUTASE 3 gene
Answer:
The full name of the OsUAM3 gene is OsUAM3-like gene. Please note that this name may not be a widely recognized standard gene name, as OsUAM3-like suggests it might be a gene similar to OsUAM3 rather than a specifically recognized gene name. The exact name may need to be confirmed based on the latest scientific literature or databases.
Analysis:
In the actual database annotation, OsUAM3 corresponds to "UDP-ARABINOPYRANOSE MUTASE 3," but the model mistakenly assumed it to be an "OsUAM3-like" variant, indicating a lack of understanding of the full gene name.
Example Question:
How is IPA1 regulated in vivo?
Reference:
IPA1 is regulated by microRNA (OsmiR156) in vivo.
Answer:
IPA1 regulation in vivo is mainly achieved through transcriptional control. Specifically, IPA1 expression is regulated by multiple transcription factors, including members of the MYB, bHLH, and WD40 families. These transcription factors regulate IPA1 expression by binding to specific cis-acting elements in the IPA1 gene promoter region. Additionally, environmental factors such as light and hormone levels also influence the activity of these transcription factors, thereby indirectly regulating IPA1 expression. In some cases, IPA1 expression may also be regulated by microRNA.
Analysis:
According to literature, IPA1 is most classically and definitively regulated by miRNA156 in rice. However, the model provided a general "transcription factor group" regulation mechanism, which, although biologically plausible, deviates from the widely recognized primary regulatory pathway, making it an off-topic answer or a case of key information omission.

Figure 8: Examples of gene function and regulatory errors.
E.2 Errors in the Second Task Type
The second task type concerns gene function and regulatory mechanisms, requiring more advanced logical reasoning and deeper domain knowledge. The model is expected to demonstrate “causal understanding” or “upstream/downstream interactions” thinking in gene function validation and regulatory network analysis. Similarly, we observe several representative error types, listed below:

Knowledge Errors  Because the model lacks thorough understanding of the background or domain-specific knowledge, its answers can be incorrect. In Table X (not shown), we list erroneous examples where the model misinterprets domain-specific terminology or experimental data, resulting in inaccurate responses.

Example 1

Example Question:
GL10 overexpression results in a significant increase in grain length and ________ compared to NIL-gl10.
Reference:
grain weight
Answer:
grain width
Analysis:
The model confused "grain weight" with "grain width," indicating an inaccurate understanding of specific experimental results, which constitutes a clear professional knowledge error.

Example 2

Example Question:
To which protein family does the OsJRL45 gene belong?
Reference:
Jacalin-related lectin (JRL) family
Answer:
Photoreceptor family
Analysis:
The model incorrectly classified the OsJRL45 gene into the "Photoreceptor family." It confused the Jacalin-related lectin family with other protein families that may contain keywords such as "light" or "pigment," demonstrating a significant background knowledge bias.

Example 3

Example Question:
What method was used in the research on LOC_Os06g03240 (Bph32 gene)?
Reference:
The method used was the transformation of a susceptible indica variety (Kasalath) with the cDNA sequence of the Ptb33 Bph32 gene.
Answer:
CRISPR/Cas9 gene-editing technology
Analysis:
The core method described in the literature was the introduction of the "cDNA sequence of the Bph32 gene" into a susceptible indica rice variety to verify its function. However, the model answered with CRISPR/Cas9 editing, which contradicts known facts, indicating a lack of understanding of literature details or professional background.
Example 4

Example Question:
In which database is OsJRL45 recorded under the gene identifier T2T?
Reference:
RiceSuperPIRdb
Answer:
T2T, OsJRL45
Analysis:
The question asks for the database where the gene identifier T2T is recorded, but the model only repeated "T2T, OsJRL45" without providing the correct database name. This represents a clear knowledge error or a failure to distinguish between databases and identifiers.

Example 5

Example Question:
Which protein domain does the OsJRL45 gene-encoded protein contain?
Reference:
jacalin-like lectin
Answer:
RNA-binding domain
Analysis:
The model confused the "jacalin-like lectin" domain with the "RNA-binding domain." This is a typical domain knowledge error, indicating that the model failed to correctly recognize or understand the protein domain annotation related to this gene.

Figure 9: Example illustrating knowledge errors in gene function and regulation.
Insufficient Answer Precision   When the question requires detailed specifics, such as particular experimental methods, tools, or procedures, the model fails to provide precise information and instead offers overly generic descriptions. As a result, the answer does not meet the exact requirements of the question, indicating over-generalization.

Example 1

Example Question:
What were the findings regarding the effect of elevated DEP1 accumulation on grain length?
Reference:
Elevated DEP1 accumulation increased the grain length by 6.85 – 9.58% with a normal plant stature.
Answer:
Elevated DEP1 accumulation was found to significantly increase grain length in the studied plants. Specifically, the grains from plants with increased DEP1 levels were approximately 10-15% longer compared to the control group. This effect was consistent across different genotypes and environmental conditions, suggesting a robust and reliable outcome. However, further research is needed to understand the underlying molecular mechanisms and potential impacts on grain yield and quality.
Analysis:
While the model's response is directionally correct, the specific values (10-15%) do not match the actual literature (6.85-9.58%). Additionally, the model adds extraneous, generalized information that was not required by the question, failing to provide the exact percentage increase in grain length.

Example 2

Example Question:
According to Gene Ontology (GO) analysis, what are the predicted functions of the APX9 gene?
Reference:
Examples include the top five GO annotations (e.g., response to reactive oxygen species, catalytic activity, peroxidase activity, etc.).
Answer:
The predicted functions of the APX9 gene in GO analysis mainly include antioxidant activity, hydrogen peroxide metabolism, and antioxidant defense systems.
Analysis:
The question requires specific GO functional entries or IDs, but the model only provides broad descriptions related to "antioxidant activity" without matching the corresponding GO terms, making the answer imprecise.

Example 3

Example Question:
The significant reductions in C30, C32, and C34 waxes in wsl4-1 leaves were ________ in the complementation line.
Reference:
rescued
Answer:
restored
Analysis:
In the context of genetics and functional validation, "rescued" specifically means "restoring the original phenotype through exogenous gene expression or treatment." While "restored" is a similar term, it is not equivalent in this precise scientific context, making this a vocabulary accuracy issue.

Figure 10: Example illustrating lack of precision in model answers.
Limited Reasoning Ability   The model does not incorporate the given information to perform logical analysis and contextual interpretation, thereby producing responses that do not meet practical needs. It either fails to link the contextual clues correctly or ignores implicit details in the question.

Example 1

Example Question:
What was the effect of overexpressing RFT1 in the study?
Reference:
Overexpression of RFT1 resulted in the direct formation of spikelets from most of the transgenic calli.
Answer:
Overexpressing RFT1 in the study led to earlier flowering in plants, which is beneficial for breeding programs in temperate regions where early flowering can help avoid summer heat stress. Additionally, it enhanced flowering under long-day conditions, indicating its potential role in regulating flowering time under different photoperiods.
Analysis:
The most significant and specific experimental observation was "the direct formation of spikelets from transgenic calli." However, the model omitted this core experimental result, instead providing a general discussion of "earlier flowering," indicating that it failed to deeply analyze or match the contextual information in the question.

Example 2

Example Question:
What is the name of the gene discussed in the text?
Reference:
The name of the gene discussed in the text is OsCHR709.
Answer:
Unable to answer because the text does not provide it.
Analysis:
The question explicitly refers to "OsCHR709" as the target gene; the model should have been able to answer by connecting contextual information. However, it claimed it "could not answer," demonstrating insufficient contextual information extraction and reasoning ability.

Figure 11: Example illustrating limited reasoning ability.
E.3 Errors in the Third Task Type
The third task type encompasses a wide range of needs related to variety selection and agronomic traits, requiring the model to integrate gene-level knowledge with real-world agricultural management experience. We observe several representative error cases, listed below:

Insufficient Semantic Understanding  The model does not fully grasp the core descriptions or context in the question, leading to incorrect interpretations of critical information. Consequently, the final output deviates from the reference solution.

Example 1

Example Question:
How is gene editing technology performed?
Reference:
Gene editing is a process that modifies specific targets in an organism’s genome using gene editing technologies. It enables precise and efficient gene insertion, deletion, or replacement, thereby altering genetic information and phenotypic traits.
Answer:
Gene editing technology typically follows these steps:
    1. Target Gene Selection: Identify the target gene to be edited, usually based on an understanding of specific traits or functions.
    2. Guide Sequence Design: Use specific sequences (such as guide RNA in the CRISPR-Cas9 system) to precisely locate the target gene.
    3. Delivery of Editing Tools: Introduce the editing tools (such as Cas9 enzyme) and guide sequences into target cells. This can be achieved through various methods, including direct injection, viral vectors, or electroporation.
    4. Gene Editing: The editing tool makes cuts at the target gene site, triggering the cell’s repair mechanisms to insert, delete, or replace the gene.
    5. Screening and Validation: Molecular biology techniques are used to screen successfully edited cells and verify whether the editing occurred as expected.
This technology can be applied to improving disease resistance, stress tolerance, yield, or nutritional value in crops.
Analysis:
Both the prompt and the reference answer specify that “the response should only include the answer to the question, without additional content.” However, the model provides a structured step-by-step explanation and additional applications of the technology, failing to strictly follow the instruction of “only providing the answer itself without unnecessary elaboration.” This is a typical example of inadequate comprehension of the prompt instructions.

Example 2

Example Question:
The paternal parent of Koshihikari is _______, and the maternal parent is ________.
Reference:
Norin 1, Norin 22.
Answer:
Dek, Dongnong 288.
Analysis:
The documented pedigree of Koshihikari is well established, yet the model provides completely unrelated parent names, showing a fundamental misunderstanding of rice variety lineage.

Figure 12: Example illustrating insufficient semantic understanding in variety selection tasks.
Knowledge Errors   Similar to the other task types, this category also shows frequent knowledge-related mistakes. Lacking the necessary domain background or relevant information leads the model to give incorrect answers.

Example 1

Example Question:
During the breeding process of Longjing 13, in 1995, Longhua 91-340 was used as the maternal parent, and ________ was used as the paternal parent for sexual hybridization.
Reference:
Kongyu 139.
Answer:
Hei 2499.
Analysis:
The official breeding records clearly state that “Kongyu 139” was used as the paternal parent. The model’s incorrect response of “Hei 2499” indicates a significant knowledge gap in breeding lineage.

Example 2

Example Question:
The paternal parent of Koshihikari is _______, and the maternal parent is ________.
Reference:
Norin 1, Norin 22.
Answer:
Dek, Dongmong 288.
Analysis:
The documented pedigree of Koshihikari is well established, yet the model provides completely unrelated parent names, showing a fundamental misunderstanding of rice variety lineage.

Example 3

Example Question:
Yuxiangyouzhan was developed through the hybridization of_______ and ________, followed by multiple generations of directional selection to create a photoperiod-sensitive indica conventional rice variety.
Reference:
TY36, IR100.
Answer:
Early indica rice, late indica rice.
Analysis:
The model’s response is overly generic, providing broad rice categories rather than the specific parent varieties “TY36” and “IR100,” highlighting a deficiency in precise knowledge of hybrid breeding lineages.
Example 4

Example Question:
Longjing 13 is a rice variety bred by ________.
Reference:
Heilongjiang Academy of Agricultural Sciences, Rice Research Institute.
Answer:
Liaoning Academy of Agricultural Sciences, Rice Research Institute.
Analysis:
The correct breeding institution for this variety is the “Heilongjiang Academy of Agricultural Sciences, Rice Research Institute,” but the model incorrectly states “Liaoning Academy of Agricultural Sciences, Rice Research Institute,” demonstrating a factual knowledge error or confusion of institutions.

Example 5

Example Question:
The breeding technology of Longjing 13 is a comprehensive breeding technology that combines conventional techniques with ________.
Reference:
Anther culture technology.
Answer:
Hybridization technology.
Analysis:
The model mistakenly identifies “hybridization technology” instead of the correct “anther culture technology,” revealing a misunderstanding of the breeding process. This is a classic example of a domain-specific knowledge error.

Figure 13: Example illustrating knowledge errors in variety selection and agronomic traits.
F  Prompt Templates

F.1  Multiple-Choice Question Generation (Task QA-1)

Prompt for Task QA-1

Topic Description:
Instruct the model to generate multiple-choice questions.

User:
Please generate independent multiple-choice questions based on the provided context. Each multiple-choice question should contain four options, with one correct answer and three distractors. Generate {num_of_questions} questions for each context.

Please note:
1. If the context is in Chinese, the questions should also be in Chinese. If the context is in English, the questions should be in English.
2. The correct answer for each question should strictly follow the specified format, with no extra output.
3. Each question should be complete and independent, without relying on the answers to other questions.

{example_1}

{example_2}

Contextual materials:
{context}

Below are the {num_of_questions} multiple-choice questions generated based on the given description:

Example:
Question: The field test conducted in Beijing in 2018 on plant lines found that overexpression of OsDREB1C led to what effect on grain yield per plant?
A. Decrease by 45.1 to 67.6%
B. Increase by 45.1 to 67.6%
C. No significant change
D. Decrease by 16.1 to 29.1%
Answer: B

Question: In the same field test, knockout (KO) of OsDREB1C resulted in what effect on yield per plot?
A. Increase by 13.8 to 27.8%
B. Increase by 41.3 to 68.3%
C. No significant change
D. Decrease by 13.8 to 27.8%
Answer: D
...

Figure 14: Illustration for multiple-choice question generation.
F.2  Multiple-Answer Question Generation (Task QA-2)

Prompt for Task QA-2

Topic Description:
Instruct the model to generate multiple-answer questions.

User:
Please generate independent multiple-answer questions based on the context, with each question containing four options, some of which are correct answers while others are distractors. Generate {num_of_questions} multiple-choice questions for each piece of context.

Please note:
1. If the context is in Chinese, the questions should also be in Chinese. If the context is in English, the questions should be in English.
2. The correct answer for each question should strictly follow the specified format, with no extra output.
3. Each question should be complete and independent, without relying on the answers to other questions.

{example_1}

{example_2}

Contextual materials:
{context}

Below are the {num_of_questions} multiple-answer questions generated based on the given description:

Example:
Question: What are the observed characteristics of the Bg1-D mutant compared to the wild type (WT)?
A. Increased grain size
B. Decreased plant height
C. Longer and wider leaves
D. Smaller panicles
Answer: A, C

Question: What changes were observed in the cells of the Bg1-D mutant?
A. Increased area of parenchyma cells
B. Decreased number of parenchyma cells
C. Down-regulation of genes associated with cell cycle and cell expansion
D. Longer epidermis cells of both palea and
Answer: A, D
...

Figure 15: Illustration for multiple-answer question generation.
F.3  Fill-in-the-Blank Question Generation (Task QA-3)

Prompt for Task QA-3

Topic Description:
Instruct the model to generate fill-in-the-blank questions.

User:
Please generate independent fill-in-the-blank questions based on the provided context. Fill-in-the-blank questions typically require filling in the gaps according to the requirements of the question. Generate {num_of_questions} fill-in-the-blank questions for each contextual material.

Please note:
1. If the context is in Chinese, the questions should also be in Chinese. If the context is in English, the questions should be in English.
2. The correct answer for each question should strictly follow the specified format, with no extra output.
3. Each question should be complete and independent, without relying on the answers to other questions.

{example_1}

{example_2}

Contextual materials:
{context}

Below are the {num_of_questions} fill-in-the-blank questions generated based on the given description:

Example:
Question: AFD1 transcripts were found in all examined ________ and organs.
Answer: tissues

Question: AFD1 exhibited more abundant expressions in _______, young panicles, lemmas, paleae, and lodicules than those in other tissues or organs examined.
Answer: internodes
...

Figure 16: Illustration for fill-in-the-blank question generation.
F.4 Text-based Q&A Generation (Task QA-4)

Prompt for Task QA-4

Topic Description:
Instruct the model to generate text generation questions.

User:
Please generate independent knowledge-based questions and answers based on the provided context. Both the input and output of the questions and answers should be in text format. Generate {num_of_questions} questions and answers for each context.

Please note:
1. If the context is in Chinese, the questions should also be in Chinese. If the context is in English, the questions should be in English.
2. The correct answer for each question should strictly follow the specified format, with no extra output.
3. Each question should be complete and independent, without relying on the answers to other questions.

{example_1}

{example_2}

Contextual materials:
{context}

Below are the {num_of_questions} text generation questions generated based on the given description:

Example:
Question: What role does the Gα protein play in the heterotrimeric G proteins pathway?
Answer: The Gα protein provides a foundation for grain size expansion.

Question: How do the three Gγ proteins DEP1, GGC2 and GS3 regulate grain size?
Answer: DEP1 and GGC2 increase grain length when in complex with Gβ, either individually or in combination. GS3 reduces grain length by competitively interacting with Gβ, but has no effect on grain size by itself.
...

Figure 17: Illustration for text-based Q&A generation.
F.5  Naive Summarization (Task SUM-1)

Prompt for Task SUM-1

Topic Description:
Instruct the model to summarize the given text.

User:
You are a text reading assistant, and your task is to summarize the text concisely while including all key points.

Please note:
1. You are a very rigorous text reading assistant, never fabricate content.
2. Please summarize the text concisely while including all key points.
3. If the source material is in English, the generated content should also be in English. If the source material is in Chinese, the generated content should also be in Chinese.

{example_1}

{example_2}

Source Material:
{}

Summary:

Example:
Input: The qPCR analysis showed that AFD1 transcripts appeared in all examined tissues and organs, including roots, internodes, leaves, panicles at different developmental stages, main rachillae of panicles, lemmas, paleae, lodicules, stamens and pistils (Figure 9A). However, the results revealed that AFD1 exhibited more abundant expressions in internodes, young panicles, lemmas, paleae, and lodicules than those in other tissues or organs examined (Figure 9A), consistent with the phenotypes defects in the afdl mutant.
Output: The qPCR analysis revealed that AFD1 transcripts are present in all examined tissues and organs, including roots, internodes, leaves, and various parts of panicles. However, AFD1 showed higher expression in internodes, young panicles, lemmas, paleae, and lodicules. This pattern is consistent with the phenotype defects observed in the afdl mutant.
...

Figure 18: Illustration for naive summarization.
F.6   Key Information Extraction (Task SUM-2)

Prompt for Task SUM-2

Topic Description:
Instruct the model to extract key information.

User:
Please extract the key information from the following description.
---
Task: Extract key information.
Goal: The extracted content should be as comprehensive as possible and cover the given text.
Method: Analyze the text, identify and extract key information points from the text.
Language restriction: If the description text is in Chinese, the extracted key information should also be in Chinese. If the description text is in English, the extracted key information should also be in English.
---
{example_1}
{example_2}

Description:
{}

Extracted key information:

Example:
Research topic: qPCR analysis of AFD1 transcripts
Findings:
- AFD1 transcripts were found in all examined tissues and organs, including roots, internodes, leaves, panicles at different developmental stages, main rachillae of panicles, lemmas, paleae, lodicules, stamens, and pistils.
- AFD1 exhibited more abundant expressions in internodes, young panicles, lemmas, paleae, and lodicules than those in other tissues or organs examined.
- The results are consistent with the phenotypes defects in the afd1 mutant.
...

Figure 19: Illustration for key information extraction.
F.7  Question Rewriting (Tasks RC-1, RC-2, RC-3, RC-4)

Prompt for Task RC-1, RC-2, RC-3, RC-4

Topic Description:
Instruct the model to rephrase the question.

User:
You are a language expert responsible for rephrasing specific questions. The requirements are as follows:
1. Change the wording and sentence structure without altering the original meaning.
2. Maintain the core meaning of the original sentence without introducing new concepts or distorting the original intent.
3. Ensure fluency and accuracy in the language.
4. The original questions can be multiple-choice, single-choice, fill-in-the-blank, or open-ended; when rephrasing, the original question type must be preserved.
5. Do not change the language of the question; if the original question is in Chinese, the rephrased question should also be in Chinese, and if the original question is in English, the rephrased question should be in English as well.

{example_1}

{example_2}

{original question}

Rephrased question:

Example:
Input: The field test conducted in Beijing in 2018 on plant lines found that overexpression of OsDREB1C led to what effect on grain yield per plant?
Output: What was the impact on grain yield per plant as a result of overexpressing OsDREB1C in the 2018 Beijing field test on plant lines?
...

Figure 20: Illustration for question rewriting across multiple subtasks.
G  Robustness Evaluation of Prompts

This section analyzes the sensitivity of model outputs to different prompt templates and evaluates how various prompt styles impact model performance. Specifically, we modify the wording and style of prompts and compare how different styles affect the consistency and stability of model-generated results.

Example 1

Question:
a pathway composed of five subunits of the heterotrimeric G proteins that regulate grain length in rice. The Gβ protein is essential for plant survival and growth. Gα provides a foundation for grain size expansion. Three Gy proteins, DEP1, GGC2 and GS3, antagonistically regulate grain size. DEP1 and GGC2, individually or in combination, increase grain length when in complex with Gβ. GS3, having no effect on grain size by itself, reduces grain length by competitively interacting with Gβ. How does GS3 influence the length of the grain?
Reference:
GS3 influences the length of the grain by competitively interacting with Gβ, thereby reducing grain length.

Instruction A:
As a researcher in the field of agricultural bioengineering, please provide a professional response to the following agricultural question.
Important notes: Your response must be precise and limited strictly to the answer itself. The question is as follows:

Answer A:
GS3 influences the length of the grain by competitively interacting with Gβ, thereby reducing grain length.

Score A:
54.9

Instruction B:
This is a question about agriculture. As an agricultural university student, you should answer according to the following rules:
The input and output of the question are both in text format. Note: Your answer should include only the response to the question, without any additional content.
Question:

Answer B:
GS3 reduces grain length by competitively interacting with Gβ.

Score B:
68.7
Question:
OsSPL14 is localized to the nucleus (Fig. 1g), consistent with a role as a transcription factor. RNA in situ hybridization revealed that OsSPL14 was predominantly expressed in the shoot apex at both the vegetative (Fig. 2a) and reproductive (Fig. 2b) stages. It was also highly expressed in the promordia of primary and secondary branches (Fig. 2c). Expression patterns of OsSPL14 and OsmiR156 in various rice organs revealed by real-time PCR and miRNA gel blot analyses showed that OsSPL14 was highly expressed in the culm and shoot apex, which is complementary with the expression pattern of OsmiR156 in vivo (Fig. 2e,f). Consistently, overexpression of OsmiR156 resulted in a substantial decrease in OsSPL14 transcripts (Supplementary Fig. 7a), whereas the interruption of OsmiR156 (overexpression of MIM156) led to a marked increase in OsSPL14 transcripts (Supplementary Fig. 7b). These results indicated that OsSPL14 was regulated by the OsmiR156-directed cleavage in vivo. In which part of the plant is OsSPL14 mainly expressed?
A. In the flower apex
B. In the leaf apex
C. In the root apex
D. In the shoot apex

Reference:
D

Instruction A:
Now, please act as an agricultural university student and complete this multiple-choice question. The task requires selecting all correct answers from the four options: A, B, C, and D.
Answering rules: Your answer should only include the correct option letters. If multiple answers are correct, separate them with a comma. Do not add any extra text or spaces. For example, if A and B are correct, write: A, B. Please begin answering:

Answer A:
D

Score A:
100

Instruction B:
This is a multiple-choice question about agriculture. As an agricultural university student, you should answer according to the following rules: Select one or more correct options from A, B, C, and D, and respond accordingly. Note that your answer should include only the correct option letters, without extra spaces or other characters. When there are multiple correct options, separate them with a comma. For example: Correct answer: A, B Strictly follow this format and provide no additional output.

Question:

Answer B:
D, B, A

Score B:
50
Question:
How is OsSPL14 regulated?
A. By the OsmiR159-directed cleavage in vivo.
B. By the OsmiR157-directed cleavage in vivo.
C. By the OsmiR156-directed cleavage in vivo.
D. By the OsmiR158-directed cleavage in vivo.

Reference:
C

Instruction A:
Below is a multiple-choice question on agricultural knowledge. Answer it from the perspective of an agricultural student.
Carefully read the question and select all correct options from A, B, C, and D.
Answer format: Only list the correct option letters, with multiple answers separated by commas. Standard format: Correct answer: A, B.

Answer A:
A

Score A:
0

Instruction B:
This is a multiple-choice question about agriculture. As an agricultural university student, you should answer according to the following rules: Select one or more correct options from A, B, C, and D, and respond accordingly. Note that your answer should include only the correct option letters, without extra spaces or other characters. When there are multiple correct options, separate them with a comma. For example: Correct answer: A, B Strictly follow this format and provide no additional output.

Question:

Answer B:
A, C

Score B:
50
Question:
What changes were observed in the cells of the Bg1-D mutant?
A. Increased area of parenchyma cells
B. Decreased number of parenchyma cells
C. Down-regulation of genes associated with cell cycle and cell expansion
D. Longer epidermis cells of both palea and lemma.

Reference:
D, A

Instruction A:
Here is a multiple-choice question in the field of agriculture. As an agricultural student, please select all the appropriate answers from A, B, C, and D.
Note: Your answer should be concise and clear, with multiple correct options separated by commas. Example format: Correct answer: A, B. Please follow this format when answering the question.

Answer A:
C

Score A:
0

Instruction B:
This is a multiple-choice question about agriculture. As an agricultural university student, you should answer according to the following rules: Select one or more correct options from A, B, C, and D, and respond accordingly. Note that your answer should include only the correct option letters, without extra spaces or other characters. When there are multiple correct options, separate them with a comma. For example: Correct answer: A, B Strictly follow this format and provide no additional output.

Question:

Answer B:
A, C

Score B:
50
Question:
Where is OsSPL14 predominantly expressed?
A. In the flower apex
B. In the leaf apex
C. In the root apex
D. In the shoot apex.
Reference:
D

Instruction A:
Let’s solve this multiple-choice question in the field of agriculture together. Your role is that of an agricultural university student, and you need to select all the correct answers from options A, B, C, and D. Remember, your answer should only include the corresponding letters. If there are multiple correct answers, separate them with a comma. Reference format: Correct answer: A, B

Answer A:
A, B, D

Score A:
50

Instruction B:
This is a multiple-choice question about agriculture. As an agricultural university student, you should answer according to the following rules: Select one or more correct options from A, B, C, and D, and respond accordingly. Note that your answer should include only the correct option letters, without extra spaces or other characters. When there are multiple correct options, separate them with a comma. For example: Correct answer: A, B Strictly follow this format and provide no additional output.

Question:

Answer B:
A, B, D

Score B:
50
Question:
What growth changes were observed in one-week-old seedlings of the Bg1-D mutant compared with WT?
A. Decreased length of leaf and root
B. No significant changes in leaf and root length
C. Decreased plant height
D. Increased length of leaf and root.
Reference:
D

Instruction A:
As an agricultural student, carefully read the question and answer this multiple-choice question. Select all the correct answers from the given options A, B, C, and D. If multiple answers are correct, separate them with a comma and list only the letter options. Example format: Correct answer: A, B Please strictly follow this answer format.

Answer A:
A, C

Score A:
0

Instruction B:
This is a multiple-choice question about agriculture. As an agricultural university student, you should answer according to the following rules: Select one or more correct options from A, B, C, and D, and respond accordingly. Note that your answer should include only the correct option letters, without extra spaces or other characters. When there are multiple correct options, separate them with a comma. For example: Correct answer: A, B Strictly follow this format and provide no additional output.
Question:

Answer B:
A, C

Score B:
0
Question:
OsSPL14 is localized to the nucleus (Fig. 1g), consistent with a role as a transcription factor. RNA in situ hybridization revealed that OsSPL14 was predominantly expressed in the shoot apex at both the vegetative (Fig. 2a) and reproductive (Fig. 2b) stages. It was also highly expressed in the promordia of primary and secondary branches (Fig. 2c). Expression patterns of OsSPL14 and OsmiR156 in various rice organs revealed by real-time PCR and miRNA gel blot analyses showed that OsSPL14 was highly expressed in the culm and shoot apex, which is complementary with the expression pattern of OsmiR156 in vivo (Fig. 2e,f). Consistently, overexpression of OsmiR156 resulted in a substantial decrease in OsSPL14 transcripts (Supplementary Fig. 7a), whereas the interruption of OsmiR156 (overexpression of MIM156) led to a marked increase in OsSPL14 transcripts (Supplementary Fig. 7b). These results indicated that OsSPL14 was regulated by the OsmiR156-directed cleavage in vivo. What function does OsSPL14 serve?
A. It acts as a termination factor.
B. It acts as a translation factor.
C. It acts as a replication factor.
D. It acts as a transcription factor.
Reference:
D

Instruction A:
Here is a multiple-choice test on agricultural knowledge. As an agricultural student, please read the question carefully and select all the correct answers from A, B, C, and D. Answer requirements: Only write the letter(s) of the correct option(s). If multiple answers are correct, separate them with a comma. For example, if A and B are correct, write: A, B. Keep your answer concise and do not add any explanations.
The question is:

Answer A:
D

Score A:
100

Instruction B:
This is a multiple-choice question about agriculture. As an agricultural university student, you should answer according to the following rules: Select one or more correct options from A, B, C, and D, and respond accordingly. Note that your answer should include only the correct option letters, without extra spaces or other characters. When there are multiple correct options, separate them with a comma. For example: Correct answer: A, B Strictly follow this format and provide no additional output.
Question:

Answer B:
D

Score B:
100
Question:
Among them, Bg1-D has the most obviously increased grain size (Fig. 1A). Compared with WT (Nipponbare, Oryza sativa L. ssp. japonica), the 1,000-grain weight of Bg1-D has increased about 33.8%, with the grain length and grain width increased about 15.2% and 17.0%, respectively (Fig. 1B). Observation of spikelet hull by scanning electron microscopy showed that the epidermis cells of both palea and lemma in Bg1-D are much longer than that of the WT (Fig. S1). Careful examination of the hull cross-section before flowering revealed significant increases of both number and area of the parenchyma cells in the Bg1-D mutant (Fig. 1C and D). Consistent with this, a number of genes associated with cell cycle and cell expansion were up-regulated in the panicles of the Bg1-D mutant (Fig. S2). Despite the greatly enlarged grain size, Bg1-D also exhibits enhanced growth of other tissues at both the vegetative and reproductive stages (Fig. 1E and Fig. S3). One-week-old seedlings of the Bg1-D mutant have obviously increased length of leaf and root compared with WT (Fig. S3 A–D). When plants enter reproductive stage, the leaves of Bg1-D are longer and wider than those of the WT plants, and the Bg1-D also has increased plant height (Fig. 1E, Fig. S3E, and Table S1). In addition, the Bg1-D panicles are much larger than those of the WT (Fig. S3F and G). We overexpressed BG1 cDNA under the control of the rice ACTIN1 promoter in WT and found that the BG1-overexpressing plants (BG1-OE) showed obviously increased grain size and other phenotypes, such as enhanced plant height, longer leaves, and larger panicles, which resemble the Bg1-D mutant phenotypes (Fig. 2C and Fig. S6A and B). Importantly, severity of the phenotypes was apparently correlated with the BG1 expression level; that is, plants with higher BG1 expression have bigger grain size as well as increased 1,000-grain weight (Fig. 2D–G), demonstrating that activation of BG1 is the cause for the Bg1-D mutant phenotypes. What were the observed outcomes when the BG1 cDNA was overexpressed, regulated by the rice ACTIN1 promoter in WT?
A. Enhanced plant height
B. Larger panicles
C. Decreased grain size
D. Longer leaves.
Reference:
A, B, D

Instruction A:
Below is a multiple-choice test in the field of agriculture. As an agricultural student, identify all the correct answers from the four options: A, B, C, and D. Note: Your answer must be concise—only include the correct option letters. If there are multiple correct answers, separate them with a comma. For example, if A and B are correct, write: A, B. Please follow this format and do not add any explanations. The question is:

Answer A:
A, D, B

Score A:
100

Instruction B:
This is a multiple-choice question about agriculture. As an agricultural university student, you should answer according to the following rules: Select one or more correct options from A, B, C, and D, and respond accordingly. Note that your answer should include only the correct option letters, without extra spaces or other characters. When there are multiple correct options, separate them with a comma. For example: Correct answer: A, B Strictly follow this format and provide no additional output. Question:

Answer B:
A, D, B

Score B:
100
H  Detailed Performance Comparison and Analysis

To analyze the experiment results from the provided table, we can examine several aspects, including the performance of different models, the comparison between different sizes of the same series, the variation in performance across tasks, and the relationship between model size and performance.

H.1  Model Comparison on Different LLMs

Proprietary Models: Among the proprietary models, GPT-4 consistently performs well across most of the tasks, often achieving top or second place in many breeding subcategories (C1, C3, C5, C6). The model’s average score is notably high at 62.06, which is the highest among proprietary models.
Open-Source Models: Models such as DeepSeek-V3-671B and Qwen2-57B exhibit strong performance in certain categories (e.g., Qwen2-57B performs exceptionally well in C3 and C7).

H.2  Comparison Between Different Model Sizes of the Same Model Series

Qwen Models: The different model sizes of the Qwen models show varying performance trends. While Qwen2-7B, a mid-tier model, significantly outperforms Qwen2-0.5B, it falls short of Qwen2-72B, the largest model in the series. However, contrary to expectations, increasing model size does not guarantee progressive performance gains. For instance, Qwen2.5-72B fails to demonstrate clear improvements over Qwen2.5-14B in many subcategories, suggesting that parameter count alone may not dictate performance in this domain. Notably, Qwen2.5-14B excels in several subcategories (e.g., C6, C8), achieving top-tier rankings, whereas Qwen2.5-72B shows inconsistent superiority—exhibiting only marginal gains in C3 and C6. This pattern implies diminishing returns for larger models in specific subcategories, highlighting the importance of training data and strategies beyond mere scale.

H.3  Top Performers in Each Subcategory

C1: (Gene Basic Information Query: GPT-4, 59.59)
C2: (Gene Expression Pattern Query: DeepSeek-V3, 62.42)
C3: (Gene Product Cellular Localization Query: GPT-4, 76.32)
C4: (Gene Function Experimental Observation: DeepSeek-V3, 63.17)
C5: (Gene Product Regulation of Downstream Genes Analysis: GPT-4, 56.34)
C6: (Gene Function Prediction: GPT-4, 59.35)
C7: (Variety Breeding Process Query: DeepSeek-V3, 68.23)
C8: (Variety Agronomic Trait Query: DeepSeek-V3, 69.04)
C9: (Variety Cultivation and Technical Key Points Query: DeepSeek-V3, 66.46)
C10: (Variety Suitable Planting Area Recommendation: DeepSeek-V3, 68.48)

Consistent Top Performers: DeepSeek-V3-671B is a standout performer, securing top-3 positions in all ten subcategories and ranking first in C2 (62.42), C4 (63.17), C7 (68.23), C8 (69.04), C9 (66.46), and C10 (68.48). Its dominance in moderately hard to medium tasks (e.g., C7, C8, C10) underscores its strength in breeding-related applications.

Other Consistent Top Performers: GPT-4 also demonstrates exceptional consistency, appearing in the top-3 for nine out of ten subcategories (C1, C2, C3, C4, C5, C6, C8, C9, C10) and ranking first in C1 (59.59), C3 (76.32), C5 (56.34), and C6 (59.35). Its strong performance across both simpler tasks (e.g., C3) and more complex ones (e.g., C5) highlights its versatility and robustness. GLM-4-Plus frequently appears in the top-3 for six subcategories (C2, C4, C7, C8, C9, C10), with notable scores such as 65.02 in C7 and 64.17 in C8, making it a reliable performer, particularly in medium-difficulty tasks. Additionally, Qwen2-57B and Qwen2-72B occasionally appear in the top-3 (e.g., C3, C7 for Qwen2-57B; C5 for Qwen2-72B), but their consistency is less pronounced compared to GPT-4 and DeepSeek-V3-671B.

H.4  Comparison of Models with Same Parameter Scale

<table>
  <tr>
    <th>Parameter Level</th>
    <th>Best Model (Avg.)</th>
    <th>Second Best (Avg.)</th>
    <th>Advantaged Domains</th>
  </tr>
  <tr>
    <td>7B</td>
    <td>InternLM2.5-7B (53.51)</td>
    <td>Qwen2.5-7B (48.45)</td>
    <td>C3, C9</td>
  </tr>
  <tr>
    <td>70B</td>
    <td>Qwen2-72B (57.62)</td>
    <td>Llama3.1-70B (54.30)</td>
    <td>C2, C5</td>
  </tr>
  <tr>
    <td>large-scale</td>
    <td>DeepSeek-V3-671B (63.30)</td>
    <td>GPT-4 (62.06)</td>
    <td>C5, C10</td>
  </tr>
</table>
H.5 Task-Specific Performance

Task Difficulty Spectrum (Based on average scores across all models):

<table>
  <tr>
    <th>Difficulty Level</th>
    <th>Subcategory</th>
    <th>Avg. Score</th>
  </tr>
  <tr>
    <td>Most Difficult</td>
    <td>C5</td>
    <td>37</td>
  </tr>
  <tr>
    <td>Moderately Hard</td>
    <td>C1, C2, C4, C9</td>
    <td>42-45</td>
  </tr>
  <tr>
    <td>Medium</td>
    <td>C6, C7, C8,C10</td>
    <td>46-49</td>
  </tr>
  <tr>
    <td>Easiest</td>
    <td>C3</td>
    <td>58</td>
  </tr>
</table>

Model Performance on Specific Subcategories (C1–C10)

The performance of large language models (LLMs) across the ten breeding subcategories (C1–C10) reveals distinct patterns in task difficulty and model capability. By classifying tasks based on average model performance, we observe that larger models excel in complex tasks, while smaller models can achieve competitive results in simpler ones. The following analysis aligns with the provided difficulty classification, adjusted to reflect observed performance trends.

Most Difficult Task (C5, Avg. Score 37): Gene Product Regulation of Downstream Genes Analysis (C5) is the most complex, with a wide performance gap. Larger models like GPT-4 (56.34) and DeepSeek-V3-671B (55.23) lead, but smaller models, such as Qwen2-0.5B (27.62), GLM-4-Chat-9B (16.20), and PLLaMa-7B (11.66), score significantly lower, aligning with the 37 average for mid-to-low-tier models. This task demands advanced reasoning and knowledge integration, favoring larger models.

Easiest Task (C3, Avg. Score 58): Gene Product Cellular Localization Query (C3) is the least challenging, with top models like GPT-4 (76.32) and DeepSeek-V3-671B (74.81) achieving high scores. Even smaller models, such as InternLM2.5-7B (67.88) and Qwen2.5-7B (66.01), perform well, indicating that C3 requires less computational capacity and is accessible to models with fewer parameters. The average score of 58 reflects mid-tier model performance, though top performers significantly exceed this.

Additional Insights: Proprietary models (e.g., GPT-4) and large open-source models (e.g., DeepSeek-V3-671B) dominate across all categories, particularly in difficult tasks like C5. Domain-specific models, such as Aksara-v1-7B (24.26 in C5) and PLLaMa-13B (13.96 in C5), underperform, suggesting limited generalization. The consistent presence of GPT-4 and DeepSeek-V3-671B in top-3 rankings underscores the advantage of model scale in complex tasks, while smaller models remain viable for easier tasks.

In summary, the difficulty classification highlights that smaller models can perform competitively in easier tasks (e.g., C3), but larger models are essential for moderately hard to difficult tasks (e.g., C5, C1, C2). This emphasizes the importance of selecting appropriately scaled models to match task complexity in breeding-related applications.

H.6 Conclusion

Model Choice: DeepSeek-V3-671B emerges as the top performer overall and should be considered the best model for most tasks. However, models like GPT-4 and GLM-4-Plus demonstrate competitive performance and could be preferable for certain tasks.

Task-Level Analysis: For more difficult tasks such as C1 and C5, larger models like GPT-4 tend to excel, while Qwen2-57B performs strongly in easier categories such as C3.

Insights from Scaling Law: Conventional scaling laws suggest that larger models, such as Qwen2.5-72B, should consistently outperform smaller models like Qwen2.5-7B and Qwen2.5-14B due to their greater model sizes and capacity. However, the experimental results challenge this expectation, as Qwen2.5-7B often performs comparably to or even surpasses its larger counterparts in several subcategories (e.g., 53.06 in C9 vs. 53.05 for Qwen2.5-72B). This discrepancy indicates that model size alone does not guarantee superior performance. Instead, factors such as training data distribution, task-specific optimization, and the nature of the tasks play critical roles in determining model effectiveness.

The findings suggest that larger models may not be fully optimized for the specific task set evaluated here, potentially due to mismatches between their training data and the demands of breeding-related subcategories. For certain tasks, smaller models like Qwen2.5-7B may be better suited, particularly when tasks align closely with their training or require less complex reasoning. These results challenge the
universal applicability of scaling laws across all task types, highlighting the need for careful consideration of training data characteristics and task design when selecting model size for optimal performance.

H.7 Assessment of the Generative Tasks using BERTScore
We have conducted an assessment of the generative tasks (QA-4, SUM-1, SUM-2, RC-3, RC-4) using BERTScore, and calculated the Pearson correlation coefficient between BERTScore and ROUGE-L. The results are as follows:

Table 8: Pearson correlation between BERTScore and ROUGE-L on generative tasks.

<table>
  <tr>
    <th>Task</th>
    <th>Pearson Correlation (BERTScore vs. ROUGE-L)</th>
  </tr>
  <tr>
    <td>QA-4</td>
    <td>0.7937</td>
  </tr>
  <tr>
    <td>SUM-1</td>
    <td>0.6770</td>
  </tr>
  <tr>
    <td>SUM-2</td>
    <td>0.5811</td>
  </tr>
  <tr>
    <td>RC-3</td>
    <td>0.7515</td>
  </tr>
  <tr>
    <td>RC-4</td>
    <td>0.7338</td>
  </tr>
</table>

The following table reports the BERTScore (%) for five generative task subsets in SeedBench.

Table 9: BERTScore (%) across models on five generative tasks.

<table>
  <tr>
    <th>Model</th>
    <th>QA-4</th>
    <th>SUM-1</th>
    <th>SUM-2</th>
    <th>RC-3</th>
    <th>RC-4</th>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>48.43</td>
    <td>50.74</td>
    <td>51.83</td>
    <td>43.28</td>
    <td>48.47</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>72.39</td>
    <td>74.89</td>
    <td>73.89</td>
    <td>95.59</td>
    <td>79.48</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td>64.60</td>
    <td>74.84</td>
    <td>70.85</td>
    <td>61.71</td>
    <td>72.00</td>
  </tr>
  <tr>
    <td>GLM-4-Plus</td>
    <td>79.70</td>
    <td>83.42</td>
    <td>80.19</td>
    <td>95.03</td>
    <td>84.34</td>
  </tr>
  <tr>
    <td>GPT-4o mini</td>
    <td>81.10</td>
    <td>83.50</td>
    <td>82.79</td>
    <td>93.98</td>
    <td>86.01</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>79.66</td>
    <td>87.45</td>
    <td>85.95</td>
    <td>96.02</td>
    <td>88.27</td>
  </tr>
  <tr>
    <td>OpenAI o1-mini</td>
    <td>77.36</td>
    <td>75.18</td>
    <td>69.91</td>
    <td>94.61</td>
    <td>81.97</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>81.22</td>
    <td>83.04</td>
    <td>81.55</td>
    <td>94.25</td>
    <td>85.89</td>
  </tr>
  <tr>
    <td>GLM-4-Chat-9B</td>
    <td>56.10</td>
    <td>69.45</td>
    <td>74.72</td>
    <td>51.17</td>
    <td>60.93</td>
  </tr>
  <tr>
    <td>InternLM2-7B</td>
    <td>57.49</td>
    <td>72.76</td>
    <td>74.04</td>
    <td>53.83</td>
    <td>61.34</td>
  </tr>
  <tr>
    <td>InternLM2.5-7B</td>
    <td>76.97</td>
    <td>84.07</td>
    <td>81.63</td>
    <td>91.01</td>
    <td>85.72</td>
  </tr>
  <tr>
    <td>Llama3.1-8B</td>
    <td>74.91</td>
    <td>82.11</td>
    <td>77.87</td>
    <td>83.28</td>
    <td>81.18</td>
  </tr>
  <tr>
    <td>Llama3.1-70B</td>
    <td>77.97</td>
    <td>83.84</td>
    <td>82.62</td>
    <td>91.12</td>
    <td>82.98</td>
  </tr>
  <tr>
    <td>Llama3.3-70B</td>
    <td>76.35</td>
    <td>81.89</td>
    <td>79.08</td>
    <td>87.91</td>
    <td>82.52</td>
  </tr>
  <tr>
    <td>Mistral-v0.3-7B</td>
    <td>75.13</td>
    <td>81.98</td>
    <td>84.54</td>
    <td>75.94</td>
    <td>83.66</td>
  </tr>
  <tr>
    <td>Qwen2-0.5B</td>
    <td>75.27</td>
    <td>78.37</td>
    <td>76.46</td>
    <td>64.88</td>
    <td>79.64</td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>77.21</td>
    <td>78.71</td>
    <td>78.85</td>
    <td>84.85</td>
    <td>82.60</td>
  </tr>
  <tr>
    <td>Qwen2-57B</td>
    <td>80.31</td>
    <td>82.82</td>
    <td>83.93</td>
    <td>92.00</td>
    <td>85.83</td>
  </tr>
  <tr>
    <td>Qwen2-72B</td>
    <td>78.75</td>
    <td>80.23</td>
    <td>85.11</td>
    <td>94.51</td>
    <td>84.53</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>77.66</td>
    <td>81.19</td>
    <td>82.19</td>
    <td>82.82</td>
    <td>83.35</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>75.30</td>
    <td>75.38</td>
    <td>62.66</td>
    <td>83.46</td>
    <td>82.80</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>79.50</td>
    <td>84.38</td>
    <td>83.50</td>
    <td>93.16</td>
    <td>83.14</td>
  </tr>
  <tr>
    <td>QwQ-32B</td>
    <td>70.16</td>
    <td>70.43</td>
    <td>69.57</td>
    <td>65.62</td>
    <td>73.06</td>
  </tr>
  <tr>
    <td>Aksara-v1-7B</td>
    <td>71.48</td>
    <td>77.60</td>
    <td>80.33</td>
    <td>70.86</td>
    <td>79.36</td>
  </tr>
  <tr>
    <td>PLLaMa-7B</td>
    <td>69.53</td>
    <td>64.44</td>
    <td>63.62</td>
    <td>62.41</td>
    <td>69.60</td>
  </tr>
  <tr>
    <td>PLLaMa-13B</td>
    <td>61.73</td>
    <td>62.46</td>
    <td>59.66</td>
    <td>56.95</td>
    <td>66.64</td>
  </tr>
</table>
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="10">Breeding Subcategories</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>C1</th><th>C2</th><th>C3</th><th>C4</th><th>C5</th><th>C6</th><th>C7</th><th>C8</th><th>C9</th><th>C10</th>
  </tr>
  <tr>
    <th colspan="11">Proprietary LLMs</th>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td><td>48.77</td><td>57.72</td><td>66.02</td><td>57.54</td><td>47.82</td><td>49.36</td><td>57.47</td><td>60.11</td><td>58.06</td><td>58.89</td><td>55.45</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td><td>47.00</td><td>59.55</td><td>62.42</td><td>59.56</td><td>43.11</td><td>49.55</td><td>53.41</td><td>56.18</td><td>52.51</td><td>53.71</td><td>53.58</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td><td>33.67</td><td>27.37</td><td>53.04</td><td>32.07</td><td>25.87</td><td>44.41</td><td>33.57</td><td>36.77</td><td>31.78</td><td>31.70</td><td>34.24</td>
  </tr>
  <tr>
    <td>GLM-4-Plus</td><td>52.72</td><td>59.62</td><td>70.62</td><td>60.11</td><td>50.60</td><td>56.75</td><td>65.02</td><td>64.17</td><td>61.70</td><td>62.90</td><td>59.61</td>
  </tr>
  <tr>
    <td>GPT-4</td><td>59.59</td><td>60.55</td><td>76.32</td><td>61.16</td><td>56.34</td><td>59.35</td><td>63.67</td><td>64.74</td><td>60.65</td><td>67.66</td><td>62.06</td>
  </tr>
  <tr>
    <td>GPT-4o mini</td><td>54.24</td><td>56.64</td><td>72.11</td><td>59.28</td><td>53.00</td><td>57.88</td><td>58.38</td><td>61.75</td><td>57.50</td><td>62.38</td><td>58.40</td>
  </tr>
  <tr>
    <td>OpenAI o1-mini</td><td>49.16</td><td>55.58</td><td>59.37</td><td>54.77</td><td>44.43</td><td>50.73</td><td>54.57</td><td>55.36</td><td>54.91</td><td>54.19</td><td>53.25</td>
  </tr>
  <tr>
    <th colspan="11">Open-Source LLMs</th>
  </tr>
  <tr>
    <td>DeepSeek-V3-671B</td><td>56.03</td><td>62.42</td><td>74.81</td><td>63.17</td><td>55.23</td><td>58.84</td><td>68.23</td><td>69.04</td><td>66.46</td><td>68.48</td><td>63.30</td>
  </tr>
  <tr>
    <td>(±0.11)</td><td>(±0.12)</td><td>(±0.43)</td><td>(±0.12)</td><td>(±0.01)</td><td>(±0.12)</td><td>(±0.08)</td><td>(±0.01)</td><td>(±0.02)</td><td>(±0.05)</td><td>(±0.03)</td><td></td>
  </tr>
  <tr>
    <td>GLM-4-Chat-9B</td><td>23.28</td><td>21.31</td><td>39.97</td><td>26.13</td><td>16.20</td><td>34.15</td><td>26.63</td><td>29.60</td><td>25.60</td><td>26.68</td><td>26.55</td>
  </tr>
  <tr>
    <td>(±0.01)</td><td>(±0.12)</td><td>(±0.43)</td><td>(±0.12)</td><td>(±0.01)</td><td>(±0.12)</td><td>(±0.08)</td><td>(±0.01)</td><td>(±0.02)</td><td>(±0.05)</td><td>(±0.03)</td><td></td>
  </tr>
  <tr>
    <td>InternLM2-7B</td><td>27.55</td><td>21.14</td><td>39.64</td><td>28.57</td><td>15.16</td><td>36.12</td><td>28.74</td><td>30.80</td><td>27.32</td><td>29.22</td><td>28.71</td>
  </tr>
  <tr>
    <td>(±0.01)</td><td>(±0.01)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.03)</td><td>(±0.02)</td><td>(±0.04)</td><td>(±0.01)</td><td>(±0.02)</td><td>(±0.01)</td><td></td>
  </tr>
  <tr>
    <td>InternLM2.5-7B</td><td>51.71</td><td>55.75</td><td>67.88</td><td>50.48</td><td>44.14</td><td>56.73</td><td>51.28</td><td>54.91</td><td>52.46</td><td>56.24</td><td>53.51</td>
  </tr>
  <tr>
    <td>(±0.01)</td><td>(±0.03)</td><td>(±0.02)</td><td>(±0.14)</td><td>(±0.01)</td><td>(±0.01)</td><td>(±0.02)</td><td>(±0.03)</td><td>(±0.01)</td><td>(±0.01)</td><td>(±0.03)</td><td></td>
  </tr>
  <tr>
    <td>Llama3.1-8B</td><td>43.89</td><td>31.21</td><td>42.53</td><td>40.68</td><td>38.47</td><td>43.80</td><td>42.87</td><td>51.62</td><td>41.88</td><td>40.91</td><td>42.23</td>
  </tr>
  <tr>
    <td>(±0.35)</td><td>(±1.07)</td><td>(±0.30)</td><td>(±0.16)</td><td>(±0.38)</td><td>(±0.03)</td><td>(±0.05)</td><td>(±0.51)</td><td>(±0.14)</td><td>(±0.23)</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3.1-70B</td><td>48.72</td><td>55.41</td><td>64.77</td><td>53.67</td><td>46.73</td><td>54.08</td><td>56.94</td><td>57.72</td><td>55.31</td><td>57.56</td><td>54.30</td>
  </tr>
  <tr>
    <td>(±0.07)</td><td>(±0.01)</td><td>(±0.00)</td><td>(±0.29)</td><td>(±0.00)</td><td>(±0.01)</td><td>(±0.12)</td><td>(±0.09)</td><td>(±0.11)</td><td>(±0.56)</td><td>(±0.13)</td><td></td>
  </tr>
  <tr>
    <td>Llama3.3-70B</td><td>45.32</td><td>47.15</td><td>60.62</td><td>49.76</td><td>40.90</td><td>54.30</td><td>52.79</td><td>54.61</td><td>49.98</td><td>55.05</td><td>50.53</td>
  </tr>
  <tr>
    <td>(±0.00)</td><td>(±0.06)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.09)</td><td>(±0.06)</td><td>(±0.00)</td><td>(±0.02)</td><td>(±0.00)</td><td>(±0.01)</td><td></td>
  </tr>
  <tr>
    <td>Mistral-v0.3-7B</td><td>42.61</td><td>38.28</td><td>57.02</td><td>40.41</td><td>29.97</td><td>44.22</td><td>36.31</td><td>43.98</td><td>39.92</td><td>43.51</td><td>41.59</td>
  </tr>
  <tr>
    <td>(±0.32)</td><td>(±0.19)</td><td>(±0.14)</td><td>(±0.15)</td><td>(±0.21)</td><td>(±0.06)</td><td>(±0.39)</td><td>(±0.23)</td><td>(±0.10)</td><td>(±0.05)</td><td>(±0.06)</td><td></td>
  </tr>
  <tr>
    <td>Qwen2-0.5B</td><td>32.84</td><td>25.15</td><td>40.19</td><td>28.20</td><td>27.62</td><td>37.22</td><td>33.81</td><td>33.63</td><td>28.25</td><td>31.67</td><td>31.44</td>
  </tr>
  <tr>
    <td>(±0.26)</td><td>(±0.64)</td><td>(±0.24)</td><td>(±0.49)</td><td>(±0.19)</td><td>(±0.58)</td><td>(±0.21)</td><td>(±0.33)</td><td>(±0.22)</td><td>(±0.13)</td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2-7B</td><td>44.21</td><td>40.41</td><td>63.00</td><td>47.36</td><td>35.37</td><td>52.30</td><td>45.61</td><td>48.73</td><td>44.88</td><td>46.89</td><td>46.51</td>
  </tr>
  <tr>
    <td>(±0.03)</td><td>(±0.10)</td><td>(±0.11)</td><td>(±0.10)</td><td>(±0.12)</td><td>(±0.27)</td><td>(±0.06)</td><td>(±0.14)</td><td>(±0.13)</td><td>(±0.17)</td><td>(±0.02)</td><td></td>
  </tr>
  <tr>
    <td>Qwen2-57B</td><td>53.67</td><td>49.81</td><td>74.30</td><td>58.38</td><td>39.34</td><td>54.71</td><td>63.89</td><td>59.57</td><td>59.22</td><td>60.08</td><td>57.20</td>
  </tr>
  <tr>
    <td>(±0.21)</td><td>(±0.05)</td><td>(±0.37)</td><td>(±0.04)</td><td>(±0.40)</td><td>(±0.16)</td><td>(±0.37)</td><td>(±0.04)</td><td>(±0.01)</td><td>(±0.07)</td><td>(±0.01)</td><td></td>
  </tr>
  <tr>
    <td>Qwen2-72B</td><td>51.16</td><td>58.10</td><td>74.07</td><td>59.72</td><td>51.58</td><td>57.76</td><td>58.85</td><td>61.63</td><td>56.69</td><td>59.11</td><td>57.62</td>
  </tr>
  <tr>
    <td>(±1.70)</td><td>(±0.47)</td><td>(±0.04)</td><td>(±0.44)</td><td>(±0.42)</td><td>(±0.75)</td><td>(±2.35)</td><td>(±4.89)</td><td>(±2.16)</td><td>(±3.33)</td><td>(±2.60)</td><td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td><td>45.16</td><td>39.50</td><td>66.01</td><td>44.61</td><td>35.72</td><td>50.00</td><td>53.60</td><td>53.31</td><td>53.06</td><td>51.05</td><td>48.45</td>
  </tr>
  <tr>
    <td>(±0.43)</td><td>(±0.19)</td><td>(±0.36)</td><td>(±0.19)</td><td>(±0.51)</td><td>(±0.24)</td><td>(±0.29)</td><td>(±0.21)</td><td>(±0.31)</td><td>(±0.10)</td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td><td>50.91</td><td>50.73</td><td>68.62</td><td>52.15</td><td>47.14</td><td>54.54</td><td>57.02</td><td>62.05</td><td>54.37</td><td>54.15</td><td>54.21</td>
  </tr>
  <tr>
    <td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.06)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.03)</td><td>(±0.00)</td><td>(±0.12)</td><td>(±0.00)</td><td>(±0.02)</td><td></td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td><td>46.86</td><td>47.41</td><td>70.99</td><td>51.89</td><td>46.17</td><td>57.60</td><td>55.35</td><td>56.31</td><td>53.05</td><td>54.75</td><td>52.63</td>
  </tr>
  <tr>
    <td>(±0.01)</td><td>(±0.01)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.13)</td><td>(±0.00)</td><td>(±0.01)</td><td>(±0.01)</td><td>(±0.01)</td><td>(±0.02)</td><td></td>
  </tr>
  <tr>
    <td>QwQ-32B</td><td>32.24</td><td>21.06</td><td>47.11</td><td>29.14</td><td>28.56</td><td>39.68</td><td>38.17</td><td>39.56</td><td>34.70</td><td>34.52</td><td>33.55</td>
  </tr>
  <tr>
    <td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td>(±0.00)</td><td></td>
  </tr>
  <tr>
    <th colspan="11">Domain Specific LLMs</th>
  </tr>
  <tr>
    <td>Aksara-v1-7B</td><td>36.72</td><td>36.69</td><td>48.32</td><td>35.41</td><td>24.26</td><td>36.83</td><td>31.17</td><td>34.64</td><td>31.15</td><td>34.14</td><td>35.04</td>
  </tr>
  <tr>
    <td>(±0.18)</td><td>(±0.10)</td><td>(±0.27)</td><td>(±0.06)</td><td>(±0.02)</td><td>(±0.22)</td><td>(±0.03)</td><td>(±0.02)</td><td>(±0.01)</td><td>(±0.05)</td><td>(±0.05)</td><td></td>
  </tr>
  <tr>
    <td>PLLaMa-7B</td><td>17.85</td><td>13.69</td><td>17.99</td><td>16.81</td><td>11.66</td><td>21.67</td><td>14.34</td><td>17.36</td><td>12.39</td><td>16.11</td><td>16.46</td>
  </tr>
  <tr>
    <td>(±0.03)</td><td>(±0.07)</td><td>(±0.44)</td><td>(±0.14)</td><td>(±0.20)</td><td>(±0.34)</td><td>(±0.21)</td><td>(±0.03)</td><td>(±0.17)</td><td>(±0.22)</td><td>(±0.01)</td><td></td>
  </tr>
  <tr>
    <td>PLLaMa-13B</td><td>15.10</td><td>14.18</td><td>28.41</td><td>18.83</td><td>13.96</td><td>23.28</td><td>18.53</td><td>17.37</td><td>14.15</td><td>18.51</td><td>17.57</td>
  </tr>
  <tr>
    <td>(±0.04)</td><td>(±0.03)</td><td>(±0.36)</td><td>(±0.03)</td><td>(±0.01)</td><td>(±0.07)</td><td>(±0.12)</td><td>(±0.21)</td><td>(±0.08)</td><td>(±0.03)</td><td>(±0.03)</td><td></td>
  </tr>
</table>

Table 10: Evaluation of 26 LLMs on SeedBench. Performance is stratified by breeding subcategories, with open-source/domain-specific models evaluated through 3 repeated trials (mean scores reported). The scores represent averages across three different metrics for 11 task types. The columns delineate ten subcategories in breeding: (C1) Gene Basic Information Query, (C2) Gene Expression Pattern Query, (C3) Gene Product Cellular Localization Query, (C4) Gene Function Experimental Observation, (C5) Gene Product Regulation of Downstream Genes Analysis, (C6) Gene Function Prediction, (C7) Variety Breeding Process Query, (C8) Variety Agronomic Trait Query, (C9) Variety Cultivation and Technical Key Points Query, (C10) Variety Suitable Planting Area Recommendation. Top-3 performers per column are highlighted in red.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="5">Question Type</th>
    <th colspan="5">Average</th>
  </tr>
  <tr>
    <th>QA-1</th>
    <th>QA-2</th>
    <th>QA-3</th>
    <th>QA-4</th>
    <th>SUM-1</th>
    <th>SUM-2</th>
    <th>RC-1</th>
    <th>RC-2</th>
    <th>RC-3</th>
    <th>RC-4</th>
    <th>RC-5</th>
  </tr>
  <tr>
    <td>Proprietary LLMs</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>57.50</td><td>74.68</td><td>25.84</td><td>21.82</td><td>33.97</td><td>44.94</td><td>98.23</td><td>97.67</td><td>85.97</td><td>44.58</td><td>88.17</td><td>61.22</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>56.50</td><td>73.26</td><td>27.37</td><td>16.96</td><td>22.99</td><td>28.88</td><td>99.12</td><td>97.01</td><td>86.75</td><td>43.18</td><td>84.95</td><td>57.91</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td>62.00</td><td>58.59</td><td>1.83</td><td>11.04</td><td>31.01</td><td>27.76</td><td>97.35</td><td>65.25</td><td>7.37</td><td>25.74</td><td>95.34</td><td>43.93</td>
  </tr>
  <tr>
    <td>GLM-4-Plus</td>
    <td>64.50</td><td>75.40</td><td>25.82</td><td>32.51</td><td>48.53</td><td>51.53</td><td>99.12</td><td>97.02</td><td>84.39</td><td>49.31</td><td>85.30</td><td>64.85</td>
  </tr>
  <tr>
    <td>GPT-4o mini</td>
    <td>57.50</td><td>72.33</td><td>17.88</td><td>44.47</td><td>49.51</td><td>60.66</td><td>97.35</td><td>95.38</td><td>85.30</td><td>57.73</td><td>84.23</td><td>65.67</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>60.50</td><td>73.87</td><td>21.35</td><td>36.07</td><td>58.73</td><td>62.89</td><td>100.00</td><td>96.44</td><td>87.86</td><td>62.29</td><td>86.74</td><td>67.88</td>
  </tr>
  <tr>
    <td>OpenAI o1-mini</td>
    <td>57.50</td><td>73.81</td><td>22.25</td><td>27.93</td><td>38.40</td><td>37.80</td><td>100.00</td><td>96.17</td><td>83.46</td><td>36.40</td><td>82.80</td><td>59.68</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>72.50</td><td>79.84</td><td>29.29</td><td>40.63</td><td>48.06</td><td>54.67</td><td>100.00</td><td>97.22</td><td>87.89</td><td>55.19</td><td>86.74</td><td>68.37</td>
  </tr>
  <tr>
    <td>Open-Source LLMs</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>GLM-4-Chat-9B</td>
    <td>53.00</td><td>62.41</td><td>0.95</td><td>1.57</td><td>5.69</td><td>13.38</td><td>98.53</td><td>68.51</td><td>6.26</td><td>2.29</td><td>93.55</td><td>36.92</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.10)</td><td>(±0.00) (±0.01)</td><td>(±0.03) (±0.01)</td><td>(±0.03) (±0.01)</td><td>(±0.02) (±0.01)</td><td>(±0.29) (±0.01)</td><td>(±0.01) (±0.36)</td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>InternLM2-7B</td>
    <td>56.50</td><td>66.93</td><td>1.85</td><td>2.14</td><td>8.66</td><td>13.88</td><td>98.23</td><td>85.88</td><td>14.31</td><td>6.51</td><td>96.77</td><td>41.06</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.17)</td><td>(±0.00) (±0.00)</td><td>(±0.07) (±0.00)</td><td>(±0.07) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.02) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>InternLM2.5-7B</td>
    <td>58.50</td><td>65.86</td><td>12.99</td><td>32.69</td><td>50.98</td><td>53.64</td><td>99.12</td><td>94.48</td><td>73.39</td><td>55.41</td><td>70.37</td><td>60.67</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00)</td><td>(±0.05) (±0.07)</td><td>(±0.10) (±0.02)</td><td>(±0.02) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.01) (±0.21)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3.1-8B</td>
    <td>48.00</td><td>64.33</td><td>10.60</td><td>24.57</td><td>48.11</td><td>42.62</td><td>94.69</td><td>81.05</td><td>54.31</td><td>45.63</td><td>74.67</td><td>53.51</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.30)</td><td>(±0.24) (±0.12)</td><td>(±0.14) (±0.09)</td><td>(±0.09) (±0.09)</td><td>(±0.09) (±0.09)</td><td>(±0.76) (±0.21)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3.1-70B</td>
    <td>56.00</td><td>73.73</td><td>19.53</td><td>34.65</td><td>52.09</td><td>54.83</td><td>99.12</td><td>96.62</td><td>74.76</td><td>51.28</td><td>83.15</td><td>63.25</td>
  </tr>
  <tr>
    <td>(±0.71) (±0.07)</td><td>(±0.09) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.02) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.98) (±0.01)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3.3-70B</td>
    <td>58.50</td><td>71.50</td><td>18.65</td><td>26.55</td><td>47.79</td><td>53.03</td><td>99.12</td><td>93.88</td><td>64.92</td><td>49.96</td><td>80.29</td><td>60.38</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00)</td><td>(±0.00) (±0.01)</td><td>(±0.13) (±0.01)</td><td>(±0.01) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.09) (±0.02)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Mistral-v0.3-7B</td>
    <td>39.50</td><td>58.14</td><td>5.91</td><td>30.41</td><td>45.02</td><td>62.03</td><td>83.19</td><td>76.43</td><td>38.51</td><td>50.31</td><td>73.12</td><td>51.11</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.19)</td><td>(±0.18) (±0.42)</td><td>(±0.05) (±0.03)</td><td>(±0.00) (±0.00)</td><td>(±0.22) (±0.53)</td><td>(±0.00) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2-0.5B</td>
    <td>40.50</td><td>62.20</td><td>2.82</td><td>31.21</td><td>44.72</td><td>53.73</td><td>55.46</td><td>65.89</td><td>15.98</td><td>42.04</td><td>50.18</td><td>42.25</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.29)</td><td>(±0.07) (±0.47)</td><td>(±0.20) (±0.34)</td><td>(±1.35) (±0.61)</td><td>(±0.08) (±0.76)</td><td>(±0.72) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>57.00</td><td>69.44</td><td>12.72</td><td>32.42</td><td>31.18</td><td>40.30</td><td>97.35</td><td>82.36</td><td>57.83</td><td>46.96</td><td>84.59</td><td>55.65</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.09)</td><td>(±0.17) (±0.08)</td><td>(±0.00) (±0.08)</td><td>(±0.00) (±0.17)</td><td>(±0.20) (±0.28)</td><td>(±0.00) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2-57B</td>
    <td>56.00</td><td>74.79</td><td>21.24</td><td>39.56</td><td>46.04</td><td>60.27</td><td>99.12</td><td>95.95</td><td>76.55</td><td>56.66</td><td>80.83</td><td>64.27</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.03)</td><td>(±0.11) (±0.23)</td><td>(±0.03) (±0.64)</td><td>(±0.00) (±0.00)</td><td>(±0.09) (±0.25)</td><td>(±0.00) (±0.25)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2-72B</td>
    <td>59.50</td><td>75.98</td><td>19.55</td><td>31.62</td><td>31.08</td><td>63.09</td><td>99.12</td><td>94.24</td><td>72.20</td><td>51.58</td><td>89.96</td><td>62.54</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>57.00</td><td>71.19</td><td>17.46</td><td>32.47</td><td>43.96</td><td>54.70</td><td>96.46</td><td>88.33</td><td>54.47</td><td>47.97</td><td>79.21</td><td>58.47</td>
  </tr>
  <tr>
    <td>(±1.00) (±0.24)</td><td>(±0.10) (±0.09)</td><td>(±0.01) (±0.07)</td><td>(±0.00) (±0.07)</td><td>(±0.00) (±0.02)</td><td>(±0.00) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>57.00</td><td>72.49</td><td>17.43</td><td>22.66</td><td>54.88</td><td>56.20</td><td>99.12</td><td>93.36</td><td>67.72</td><td>51.57</td><td>83.51</td><td>61.45</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.25) (±0.00)</td><td>(±0.00) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>70.50</td><td>73.71</td><td>17.86</td><td>29.84</td><td>51.33</td><td>59.52</td><td>100.00</td><td>87.79</td><td>57.07</td><td>48.41</td><td>83.51</td><td>61.77</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00)</td><td>(±0.00) (±0.01)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>QwQ-32B</td>
    <td>61.50</td><td>58.81</td><td>3.00</td><td>14.54</td><td>15.60</td><td>26.93</td><td>91.15</td><td>64.17</td><td>19.23</td><td>19.09</td><td>91.04</td><td>42.28</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Domain Specific LLMs</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Aksara-v1-7B</td>
    <td>34.167</td><td>53.4</td><td>4.28</td><td>17.263</td><td>26.503</td><td>38.32</td><td>87.61</td><td>72.89</td><td>30.037</td><td>35.397</td><td>66.67</td><td>42.41</td>
  </tr>
  <tr>
    <td>(±0.289) (±0.139)</td><td>(±0.061) (±0.025)</td><td>(±0.006) (±0.000)</td><td>(±0.00) (±0.00)</td><td>(±0.075) (±0.396)</td><td>(±0.00) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>PLLaMa-7B</td>
    <td>6.5</td><td>35.47</td><td>2.613</td><td>20.223</td><td>13.753</td><td>20.577</td><td>6.19</td><td>35.03</td><td>12.76</td><td>24.583</td><td>4.78</td><td>16.59</td>
  </tr>
  <tr>
    <td>(±0.5) (±0.485)</td><td>(±0.032) (±0.326)</td><td>(±0.126) (±0.046)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.167)</td><td>(±0.208) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>PLLaMa-13B</td>
    <td>24.167</td><td>50.433</td><td>1.013</td><td>18.17</td><td>13.37</td><td>14.84</td><td>38.94</td><td>57.37</td><td>6.95</td><td>18.007</td><td>40.98</td><td>25.84</td>
  </tr>
  <tr>
    <td>(±0.289) (±0.133)</td><td>(±0.04) (±0.171)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.00)</td><td>(±0.00) (±0.243)</td><td>(±0.416) (±0.00)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
</table>

Table 11: Evaluation results (zero-shot) on different question type. Top-3 per column are highlighted.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="8">Question Type</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>QA-1</th>
    <th>QA-2</th>
    <th>QA-3</th>
    <th>QA-4 SUM-TSUM-2</th>
    <th>RC-1</th>
    <th>RC-2</th>
    <th>RC-3</th>
    <th>RC-4 RC-5</th>
  </tr>
  <tr>
    <td>Proprietary LLMs</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>53.50</td><td>75.54</td><td>27.92</td><td>23.99</td><td>32.36</td><td>42.88</td><td>98.23</td><td>97.67</td><td>85.68</td><td>45.40</td><td>86.02</td><td>60.84</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>55.50</td><td>73.49</td><td>27.13</td><td>27.03</td><td>28.98</td><td>35.65</td><td>99.12</td><td>97.45</td><td>86.98</td><td>46.08</td><td>86.74</td><td>60.38</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td>64.50</td><td>59.30</td><td>1.73</td><td>12.99</td><td>31.99</td><td>36.43</td><td>97.35</td><td>65.91</td><td>6.25</td><td>26.63</td><td>93.19</td><td>45.12</td>
  </tr>
  <tr>
    <td>GLM-4-Plus</td>
    <td>67.00</td><td>76.46</td><td>26.47</td><td>46.13</td><td>51.35</td><td>52.73</td><td>99.12</td><td>96.62</td><td>85.99</td><td>59.14</td><td>80.29</td><td>67.39</td>
  </tr>
  <tr>
    <td>GPT-4o mini</td>
    <td>55.00</td><td>71.89</td><td>19.10</td><td>41.59</td><td>50.21</td><td>57.59</td><td>97.35</td><td>95.00</td><td>86.00</td><td>53.76</td><td>82.08</td><td>64.51</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>57.50</td><td>73.24</td><td>23.43</td><td>46.19</td><td>63.93</td><td>68.07</td><td>100.00</td><td>96.86</td><td>88.97</td><td>64.53</td><td>83.15</td><td>69.63</td>
  </tr>
  <tr>
    <td>OpenAI o1-mini</td>
    <td>61.50</td><td>74.78</td><td>19.82</td><td>35.75</td><td>32.89</td><td>25.91</td><td>99.12</td><td>96.84</td><td>83.31</td><td>47.12</td><td>80.65</td><td>59.79</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>71.50</td><td>79.16</td><td>32.28</td><td>46.59</td><td>50.16</td><td>58.52</td><td>100.00</td><td>97.00</td><td>86.85</td><td>60.59</td><td>85.30</td><td>69.81</td>
  </tr>
  <tr>
    <td>Open-Source LLMs</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>GLM-4-Chat-9B</td>
    <td>55.50</td><td>59.31</td><td>0.51</td><td>1.24</td><td>4.46</td><td>11.12</td><td>97.35</td><td>64.35</td><td>7.67</td><td>1.92</td><td>91.16</td><td>35.87</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00) (±0.00) (±0.01) (±0.01) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.41)</td>
  </tr>
  <tr>
    <td>InternLM2-7B</td>
    <td>50.50</td><td>59.30</td><td>0.16</td><td>1.55</td><td>7.82</td><td>12.76</td><td>94.69</td><td>75.12</td><td>6.35</td><td>2.98</td><td>88.17</td><td>36.31</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.03) (±0.00) (±0.01) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.01) (±0.07)</td>
  </tr>
  <tr>
    <td>InternLM2.5-7B</td>
    <td>58.50</td><td>69.42</td><td>14.74</td><td>35.05</td><td>53.63</td><td>56.07</td><td>99.12</td><td>95.18</td><td>75.04</td><td>57.03</td><td>68.22</td><td>62.00</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00) (±0.00) (±0.12) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.11) (±0.06)</td>
  </tr>
  <tr>
    <td>Llama3.1-8B</td>
    <td>35.50</td><td>49.10</td><td>7.97</td><td>32.90</td><td>49.22</td><td>46.62</td><td>86.43</td><td>75.39</td><td>54.04</td><td>41.27</td><td>68.10</td><td>49.68</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00) (±0.01) (±0.63) (±0.07) (±0.01) (±0.51) (±0.38) (±0.00) (±0.01) (±0.62)</td>
  </tr>
  <tr>
    <td>Llama3.1-70B</td>
    <td>55.00</td><td>70.01</td><td>19.60</td><td>33.17</td><td>52.24</td><td>59.04</td><td>99.12</td><td>96.02</td><td>74.49</td><td>51.42</td><td>80.65</td><td>62.80</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.06) (±0.00) (±0.07) (±0.00) (±0.06) (±0.00) (±0.00) (±0.33) (±0.00) (±0.33)</td>
  </tr>
  <tr>
    <td>Llama3.3-70B</td>
    <td>54.75</td><td>73.55</td><td>17.35</td><td>30.61</td><td>50.09</td><td>45.12</td><td>99.12</td><td>93.52</td><td>63.22</td><td>45.62</td><td>79.21</td><td>59.29</td>
  </tr>
  <tr>
    <td>(±0.35) (±0.13) (±0.00) (±0.06) (±0.00) (±0.01) (±0.00) (±0.00) (±0.00) (±0.06) (±0.22)</td>
  </tr>
  <tr>
    <td>Mistral-v0.3-7B</td>
    <td>39.50</td><td>60.85</td><td>3.82</td><td>31.09</td><td>45.74</td><td>62.27</td><td>79.06</td><td>75.79</td><td>35.65</td><td>48.81</td><td>74.55</td><td>50.65</td>
  </tr>
  <tr>
    <td>(±0.50) (±0.13) (±0.01) (±0.12) (±0.00) (±0.51) (±0.00) (±0.30) (±0.10) (±0.75)</td>
  </tr>
  <tr>
    <td>Qwen2-0.5B</td>
    <td>44.50</td><td>62.94</td><td>2.28</td><td>33.36</td><td>36.05</td><td>38.12</td><td>67.26</td><td>68.93</td><td>12.10</td><td>40.83</td><td>25.09</td><td>39.22</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.31) (±0.25) (±0.65) (±0.38) (±0.44) (±0.00) (±0.42) (±0.98) (±0.83) (±0.46)</td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>52.17</td><td>65.62</td><td>14.13</td><td>34.50</td><td>43.35</td><td>52.33</td><td>88.50</td><td>78.53</td><td>52.03</td><td>48.77</td><td>85.42</td><td>55.94</td>
  </tr>
  <tr>
    <td>(±0.58) (±0.79) (±0.02) (±0.18) (±0.00) (±0.00) (±0.00) (±0.08) (±0.24) (±0.01) (±0.31)</td>
  </tr>
  <tr>
    <td>Qwen2-57B</td>
    <td>58.00</td><td>76.42</td><td>21.17</td><td>43.49</td><td>49.60</td><td>60.53</td><td>97.35</td><td>95.28</td><td>76.86</td><td>56.87</td><td>77.42</td><td>64.82</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00) (±0.09) (±0.36) (±0.01) (±0.12) (±0.00) (±0.16) (±0.16) (±0.63)</td>
  </tr>
  <tr>
    <td>Qwen2-72B</td>
    <td>65.00</td><td>77.73</td><td>24.51</td><td>41.70</td><td>54.36</td><td>66.48</td><td>99.12</td><td>96.13</td><td>78.54</td><td>57.85</td><td>84.41</td><td>67.80</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.13) (±2.57) (±0.96) (±0.07) (±0.01) (±0.00) (±1.57) (±7.40) (±0.69) (±0.57)</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>57.667</td><td>70.32</td><td>16.867</td><td>38.27</td><td>43.897</td><td>59.597</td><td>92.92</td><td>80.093</td><td>52.003</td><td>52.46</td><td>75.987</td><td>58.19</td>
  </tr>
  <tr>
    <td>(±0.00) (±1.11) (±0.62) (±0.29) (±0.18) (±0.40) (±0.00) (±0.30) (±0.33) (±0.39)</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>63.5</td><td>72.57</td><td>18.79</td><td>38.4</td><td>56.24</td><td>62.14</td><td>99.12</td><td>94.39</td><td>69.877</td><td>56.83</td><td>82.08</td><td>64.90</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.335) (±0.00) (±0.00)</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>67.50</td><td>74.79</td><td>21.39</td><td>40.425</td><td>55.86</td><td>62.01</td><td>99.12</td><td>89.11</td><td>55.995</td><td>51.38</td><td>82.8</td><td>63.67</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00) (±0.00) (±0.078) (±0.00) (±0.00) (±0.00) (±0.00) (±0.07) (±0.028) (±0.00)</td>
  </tr>
  <tr>
    <td>QwQ-32B</td>
    <td>63.50</td><td>60.26</td><td>6.81</td><td>25.35</td><td>13.12</td><td>20.35</td><td>93.81</td><td>65.07</td><td>22.72</td><td>27.84</td><td>75.99</td><td>43.17</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00) (±0.00)</td>
  </tr>
  <tr>
    <td>Domain Specific LLMs</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Aksara-v1-7B</td>
    <td>41.667</td><td>55.627</td><td>2.653</td><td>23.35</td><td>38.7</td><td>57.14</td><td>84.07</td><td>70.88</td><td>27.037</td><td>40.087</td><td>71.33</td><td>50.65</td>
  </tr>
  <tr>
    <td>(±0.289) (±0.182) (±0.072) (±0.157) (±0.00) (±0.00) (±0.00) (±0.167) (±0.389) (±0.00)</td>
  </tr>
  <tr>
    <td>PLLaMa-7B</td>
    <td>11.167</td><td>37.22</td><td>6.437</td><td>25.203</td><td>30.42</td><td>27.71</td><td>5.31</td><td>28.38</td><td>20.27</td><td>24.723</td><td>29.75</td><td>22.42</td>
  </tr>
  <tr>
    <td>(±0.577) (±0.859) (±0.202) (±0.117) (±0.00) (±0.00) (±0.00) (±0.208) (±0.067) (±0.00)</td>
  </tr>
  <tr>
    <td>PLLaMa-13B</td>
    <td>24.0</td><td>52.697</td><td>0.507</td><td>7.167</td><td>12.34</td><td>13.73</td><td>30.09</td><td>50.51</td><td>3.81</td><td>15.55</td><td>32.26</td><td>22.06</td>
  </tr>
  <tr>
    <td>(±0.00) (±0.029) (±0.006) (±0.086) (±0.00) (±0.00) (±0.00) (±0.035) (±0.00) (±0.00)</td>
  </tr>
</table>

Table 12: Evaluation results (one-shot) on different question type. Top-3 per column are highlighted.
Figure 22: Evaluation of Proprietary LLMs on SeedBench. Performance is stratified by task-type.

Figure 23: Evaluation of Open-Source LLMs on SeedBench. Performance is stratified by task-type.

Reviewer paper summaries: This work presents SeedBench, the first multitask benchmark designed to evaluate the performance of LLMs in seed science, a crucial field for agriculture and food security. Built using data from scientific articles in English and Chinese, SeedBench addresses the lack of specific benchmarks for seed genetic improvement by simulating three key research stages: genetic information retrieval, functional and regulatory analysis, and variety improvement. It includes 11 expert-validated task types and evaluates 26 LLMs, including proprietary, open-source, and domain-specific fine-tuned models. The results reveal significant gaps between model capabilities and real-world seed science needs, with domain-specific models underperforming expectations and no clear advantage observed in models incorporating explicit reasoning. The study highlights the need for further research to effectively apply LLMs to seed design, while SeedBench establishes a crucial foundation for future advancements in the field.
Summary of strengths: This work is highly interesting because, as far as I know, there are few resources available for tasks related to Natural Language Processing (NLP) and seed science, as this is a complex field requiring expert supervision. Additionally, the article is clear and easy to read. Below, I highlight some of its key strengths:
- It introduces SeedBench, the first multitask benchmark specifically designed to evaluate LLMs in seed science. This is a significant contribution, addressing a gap in LLM evaluation tools by focusing on a domain critical to agriculture and global food security. The methodology behind SeedBench, which involves collaboration with domain experts, the simulation of real genetic improvement processes, and a rigorous two-stage validation process, is also innovative and could serve as a model for developing benchmarks in other scientific fields. 
- The comprehensive evaluation of 26 LLMs (spanning proprietary, open-source, and domain-specific models) yields valuable empirical insights. The findings highlight the current limitations of LLMs in tackling the complex challenges of seed science. Notably, domain-specific models underperformed compared to general-purpose ones, and models with explicit reasoning modes did not consistently outperform others in genetic improvement tasks, providing crucial insights for future research.
- The differences between models of different sizes are analyzed, reaching the conclusion that it is not always better to have larger and larger models, but rather quality data to train these models. They reach this conclusion because with small models and quality data they are able to outperform models with more parameters.
- Beyond assessing the current state of LLMs in seed science, the study lays a solid foundation for future research on LLM design and application in seed development and the broader field of AI for Science. By identifying gaps between seed scientists’ expectations and LLM capabilities, it defines clear directions for future work, such as enhancing domain knowledge, integrating multimodal approaches, and improving explainability.
Summary of weaknesses: Although this work is strong, there are some aspects that could be improved: 
- The finding that domain-specific fine-tuned models performed worse than general-purpose models could suggest a limitation in the immediate applicability of current LLMs to seed science, even with fine-tuning. However, if these models were trained on narrowly defined tasks or with data not validated by experts, their lower performance might not be entirely conclusive. 
- While a rigorous validation process was implemented, a significant portion of the benchmark’s questions were automatically generated using GPT-4. This could introduce biases and limit SeedBench's ability to serve as a truly independent and objective measure of LLM performance in seed science. 
- The study acknowledges the importance of multimodal information in seed genetic improvement, yet SeedBench primarily focuses on text-based tasks. Since seed science relies heavily on sensory evaluations and environmental data, the lack of multimodal integration in the benchmark could restrict its ability to fully capture real-world challenges. 
- To evaluate generative text tasks, SeedBench uses RougeL, which, as has been shown in a large number of studies, does not adapt well to the variety of our language when it comes to expressing itself. So from my point of view it would be more correct to use automatic metrics such as BERTScore or MoverScore that take into account the semantic similarities of the generated texts and not only that the generated text is as it appears in the reference text.
- The paper includes an extremely extensive appendix (over 40 pages) which, while providing valuable details on the experiments, is almost a separate article, making it somewhat difficult to navigate.
Comments, suggestions and typos: In addition to all the points mentioned above, I include some comments that I believe are worth considering:
- In some parts of the article, different points are distinguished using bolded keywords followed by explanatory text (e.g., Section 4.1, lines 359 and 375). In these cases, I suggest adding a period between the bolded word and the explanatory text to improve readability.
- In Section 4.1 (Implementation Details), it is stated that models are evaluated using two different strategies: zero-shot learning and one-shot learning. Is there a specific reason for selecting one-shot learning rather than including or testing with a larger number of examples?
- Section 4.2 mentions that Appendix H provides a detailed analysis of the performance differences between models of varying sizes and series. In my opinion, this is an important aspect of the paper and should not be placed in an appendix. Appendices are typically used for additional data, but analytical discussions should be included in the main text.
- In line 535, there is a grammatical mistake: "Pproducers" should be corrected to "Producers."
Overall assessment scores: 4
Metareview score: 4
Metareview: This paper introduces SeedBench, the first benchmark for evaluating LLMs in seed science, a domain critical to agricultural research food security. SeedBench addresses the significant gap in specialized benchmarks for seed genetic improvement by simulating three essential research phases: genetic information retrieval, functional and regulatory analysis, and variety improvement. 

The benchmark includes 11 expert-validated task categories and evaluates 26 different LLMs, including proprietary, open-source and domain-specialized fine-tuned models. 

The reported results show disparities between current model capabilities and the practical requirements of seed science research, with domain-specific models performing quite low and no substantial advantage observed in "reasoning" models.
Summary of reasons to publish: SeedBench is an interesting contribution, as it constitutes the first multitask benchmark specifically designed to evaluate LLMs in seed science. 

The paper's experimental methodology is sound and it includes domain experts, simulation of actual genetic improvement processes, and validation protocols. 

They evaluate 26 different LLMs and the obtained results show a large room for improvement. Interesting results show that domain-specific models underperformed, and that open-weight models are not that far in terms of performance to proprietary ones.  

I agree that the paper can be a good example to be taken as a foundation for future developments in other scientific disciplines.