A conversational multi-agent AI system for automated plant phenotyping

Domain: biology
Published: 2026-04-03
Editorial outcome: major_revision

Abstract
Plant phenotyping increasingly relies on (semi-)automated image-based analysis workflows to improve its accuracy and scalability. However, many existing solutions remain overly complex, difficult to reimplement and maintain, and pose high barriers for users without substantial computational expertise. To address these challenges, we introduce PhenoAssistant: a pioneering AI-driven system that streamlines plant phenotyping via intuitive natural language interaction. PhenoAssistant leverages a large language model to orchestrate a curated toolkit supporting tasks including automated phenotype extraction, data visualisation and automated model training. We validate PhenoAssistant through several representative case studies and a set of evaluation tasks. By lowering technical hurdles, PhenoAssistant underscores the promise of AI-driven methodologies to democratising AI adoption in plant biology.

Introduction
Plant phenotyping aims to quantify functional and structural traits (phenotypes) of crops and plants, which result from complex interactions between genetics and environmental factors1. Phenotype analysis allows breeders and researchers to disentangle genetic effects from environmental adaptation, enabling the development of crops with improved yields and climate resilience -- an urgent need given that the global population is projected to reach 9.7 billion by 20502 and extreme weather events become increasingly frequent3.

Accurate plant phenotyping often relies on computational workflows that chain multiple software tools for image processing to measure plant traits and subsequently perform data analysis. However, the steep learning curve associated with these workflows such as programming, machine learning, and data science may prevent plant phenotyping researchers and practitioners from fully leveraging them4. The complexity of operating several software tools across different platforms present additional communication obstacles for both intra and inter-domain collaboration (e.g. between plant scientists with different specialisations, or between plant scientists and data scientists). Moreover, existing workflows typically employ fixed pipelines, which are difficult to extend or modify, restricting their applicability to broader tasks and scenarios. Despite calls for more accessible and versatile phenotyping systems5, this need remains inadequately addressed.

A promising approach to lowering technical barriers, bridging domain expertise, and enabling more flexible phenotype analysis is by harnessing the power of large language models (LLMs) 6,7. LLMs have emerged as powerful tools capable of solving diverse tasks from natural language instructions. Rather than focusing on developing computational skills, researchers can focus on critical scientific discovery and exploration, by simply prompting an LLM to address the necessary computational and analytical tasks. However, this promise faces the challenge that LLMs alone often lack the specialised domain knowledge and requisite computational workflows to fulfil user-specified tasks8,9. An effective strategy is to augment LLMs with instructions and external tools, forming AI agents capable of automatically executing tailored computational routines and data analysis pipelines.

An AI agent is a goal-directed software designed to perform tasks within a defined digital environment10,11. LLMs have recently become the core intelligence for many such agents (and hence also termed LLM agents) due to their strong abilities in natural language understanding and reasoning. These agents can receive task instructions, interpret and reason over inputs, and act to achieve specific objectives. They can also utilise external tools such as search engines or databases to extend their abilities12. A multi-agent AI system extends this paradigm by integrating several specialised LLM agents and tools within a coordinated framework, enabling division of labour across subtasks and allowing the system to address broader and more complex tasks than a single agent could manage alone11. Agentic AI represents multi-agent AI systems that exhibit higher adaptability and autonomy. Rather than operating with fixed roles, such systems often allow agents to interact with each another flexibly, negotiate responsibilities, and adapt their behaviour to dynamic contexts12–14. These different levels of complexity and flexibility distinguish the three paradigms and make them suitable for different scenarios: AI agents can operate independently on single, bounded tasks; multi-agent AI systems are well suited for tasks requiring multi-step workflows; and agentic AI leverages emergent behaviours to support more exploratory or creative tasks.

Recently, LLM-based agent systems have successfully been applied across different scientific domains, including chemistry15,16, material sciences17,18, biology19,20, and healthcare21–23. They not only reduce the technical difficulties of domain-specific computation and data analysis, but also serve as effective platforms for multidisciplinary collaboration. Experts from diverse fields can contribute to the analytical process by interacting with the system through natural language. For example, a plant scientist unsure about which data analysis approach to use could consult a data scientist, then relay the detailed instructions to the AI system to automate the task. Feature comparison of several existing LLMs and LLM-based agent systems for plant science and agriculture is presented at Supplementary Table 1. While LLMs and LLM-based agent systems have also been applied to agriculture and plant sciences (e.g. agricultural Q&A24–28, disease and stress phenotyping29–32, genome prediction33–35), their potential for automating complex and data-intensive phenotype analysis workflows remains underexplored.

In this work, we introduce PhenoAssistant, an open-source multi-agent AI system for automating plant phenotype analysis. By integrating a generalist LLM with a specialised toolkit for plant research, PhenoAssistant allows plant scientists to use free-text task descriptions to prompt a wide range of phenotyping-related data analysis pipelines, such as phenotype extraction from images, phenotypic statistics analysis, and data visualisation. The toolkit features cutting-edge deep learning models and LLM agents to support users’ dynamic needs in extracting and analysing plant traits. PhenoAssistant can be extended (e.g. through built-in model training functions) to accommodate customised and emerging needs. To improve reproducibility, it also allows users to save the generated pipelines and re-execute them on similar datasets. Overall, PhenoAssistant serves as a pioneer for leveraging multi-agent systems in plant phenotyping. We demonstrate its potential to streamline complex workflows and democratise AI adoption in plant biology through three case studies and evaluations covering three aspects. By lowering technical barriers and facilitating collaborations across different areas of expertise, PhenoAssistant highlights the promising future of LLM-based agent systems in accelerating scientific discovery in plant science.

Results
Design of PhenoAssistant
The goal of PhenoAssistant is to generate and execute suitable workflows by chaining AI models and other computational tools to address plant phenotype analysis requests from users. To achieve this, PhenoAssistant is built around a centralised multi-agent architecture, comprising an augmented LLM (the Manager) and a specialised toolkit consisting of dedicated computational modules and LLM agents designed to enhance its capabilities for image-based plant phenotyping, as illustrated in Fig. 1.

In this architecture, a single Manager agent is responsible for coordinating workflow creation and execution. The role of each LLM agent in the system is pre-defined and fixed. This architecture is chosen because it best supports our aim of reliable workflow generation and execution. It offers several advantages for plant phenotype analysis: it is less prone to coordination errors, produces more consistent workflows for similar task descriptions, and operates within a clearly defined set of available tools and agents. This architecture also increases transparency and user control by allowing users to intervene at any step of the generated workflow to correct errors or unexpected outcomes simply by interacting with the Manager.

Specifically, upon receiving user-provided task descriptions and accompanying data, the Manager first creates a plan detailing the necessary steps to fulfil the task. It then selects and executes the suitable tools with appropriate parameters, producing various types of outputs that can be saved for further analysis (either by the user or PhenoAssistant). Finally, the Manager summarises these outputs, presenting essential information for task completion. Users retain control throughout this process, and can provide textual feedback to refine plans or select alternative tools and parameters, minimising errors from PhenoAssistant.

A core component of many plant phenotype analysis workflows is accurate visual information extraction. While recent generalist LLMs and agents often include vision capabilities6,36,37, they are not always able to generalise to extracting plant phenotypes due to the limited representation of plant-specific data and tasks during their development. To demonstrate this limitation, we prompt the state-of-the-art generalist agent, ChatGPT 4o, in various ways, to extract the projected leaf area (PLA) and leaf count from an image sourced from the same dataset that will later be used in Case Study 1 in the Case studies subsection of the Results section. The actual PLA and leaf count for this image are 14,260 pixels and 14, respectively.

As illustrated in Fig. 2a, ChatGPT is directly prompted to extract the requested phenotypes without additional instructions. However, it cannot correctly predict either the leaf count or PLA, despite it invokes the code writing function attempting to use image processing techniques to solve the tasks. Furthermore, ChatGPT produces inconsistent results when processing the same prompt repeatedly (Fig. 2b). Next, as shown in Fig. 2c, we prompts ChatGPT to perform a computer vision pre-task (leaf instance segmentation), which allows for manual extraction of PLA and leaf count from the segmentation results. However, the results are inadequate for further analysis: several edges are incorrectly segmented, and not all individual leaves are successfully identified. Finally, we prompts ChatGPT to rely solely on its vision capabilities to extract the phenotypes, but the results are still unsatisfactory: tasks such as leaf instance segmentation cannot be performed using ChatGPT's vision abilities (Fig. 2d), and leaf count predictions are incorrect (Fig. 2e).

To overcome this limitation in generalist LLMs and agents, PhenoAssistant is equipped with phenotype extraction tools, incorporating a model zoo of computer vision models trained on plant-specific datasets, with utilities for computing traits such as plant area and diameter from these models' outputs. The toolkit also supports automatic model training on new data, enabling users to expand PhenoAssistant's vision capabilities to their unique data and analytical requirements.

Complementing these phenotype extraction tools, PhenoAssistant is also augmented with tools for phenotype analysis. This includes dedicated modules with pre-defined logic and deterministic processes for statistical tests (e.g. ANOVA). In addition, the toolkit contains various LLM agents engineered for different purposes, including the Table Analyser for extracting information from CSV files, the Data Visualiser for generating plots, and the Plot Analyser for interpreting and analysing generated plots. Additionally, the Code Writer agent supports essential tasks necessary for developing comprehensive end-to-end analysis workflows, such as merging and saving files, as well as executing novel functions not predefined within the toolkit. To enhance PhenoAssistant's expertise in plant-related domains, a retrieval augmented generation (RAG) agent provides access to scientific literature. The Pipeline Reproducer agent enables users to extract and re-execute previously conducted phenotype extraction and analyses, including all tool calls and generated code, ensuring reproducibility for similar data.

Case studies
Traditionally, executing plant phenotyping tasks requires substantial expertise in computer science, including image processing, machine learning, and coding. With PhenoAssistant, users can complete these tasks effortlessly by interacting through natural language. To demonstrate PhenoAssistant's effectiveness, we present three case studies showcasing its ability to streamline plant phenotyping analysis.

In Case Study 1, we use PhenoAssistant to replicate part of the Phenotiki analysis workflow38, aiming to visualise and analyse the growth patterns of different Arabidopsis thaliana ecotypes. We provide PhenoAssistant with the same dataset as in Minervini et al.38, which contains 24 plants from 5 different ecotypes: wild type (Col-0, 5 plants); constitutive triple response 1 (ctr1, 5 plants); ethylene insensitive 2 (ein2.1, 5 plants); pgm (mutation in the plastid ic isoform of phosphoglucomutase, 5 plants); and adh1 (mutation causing defects in alcohol dehydrogenase, 4 plants). Each plant was grown for 26 days with images captured every 12 hours, resulting in a total of 1,248 images.

As shown in Fig. 3, we prompt PhenoAssistant to address five relevant tasks, each including a task description, tools used by PhenoAssistant, and generated results. We begin with the fundamental step of any plant phenotyping task: extracting phenotypes from data: in Task 1, PhenoAssistant is prompted to extract phenotypes including projected leaf area (PLA) and leaf count from images. It identifies the need for a leaf instance segmentation model, and hence selects the one suitable for Arabidopsis from the model zoo, executes it, and computes the requested phenotypes from the segmentation results. The Code Writer is then employed to write and execute Python code to merge the computed phenotypes with metadata and save them into a CSV file. To illustrate how users can interact with PhenoAssistant in practice, we provide the chat logs for Case Study 1 Task 1 in Supplementary Note 1. At the end of Task 1, the user can request PhenoAssistant to save the executed pipeline (i.e. from instance segmentation to saving the extracted phenotypes, as demonstrated at Supplementary Note 2), which can later be reapplied to a different dataset.

From Tasks 2 to 5, we demonstrate PhenoAssistant's capabilities for processing and analysing the extracted phenotypes (i.e. from Task 1), which are essential for identifying plant growth trends and key differences among ecotypes. In Task 2, PhenoAssistant leverages the Table Analyser and Data Visualiser agents to compute and plot statistics based on user-specified plot requirements to visualise the growth patterns of different ecotypes. It can further use the Plot Analyser to interpret data visualisations, such as ranking ecotypes according to their PLA (Task 3). PhenoAssistant can also invoke relevant tools to conduct statistical analyses (e.g. ANOVA and Tukey-Kramer) using user-defined parameters and then summarise the key findings (Task 4). For example, it categorises ecotypes by PLA size as large (ein2.1, Col-0), medium (adh1, pgm), and small (ctr1), based on Tukey–Kramer post hoc tests (p < 0.05). Furthermore, PhenoAssistant can validate these findings against previous literature, by leveraging the RAG agent to embed paper contents and retrieve related knowledge (Task 5). In Tasks 4 and 5, PhenoAssistant performs the same statistical tests and reaches conclusions consistent with the Phenotiki paper38.

In Case Study 2, we show that PhenoAssistant can handle other data (e.g. non-model plant) and tasks when provided with appropriate tools, such as vision models developed on other datasets. To illustrate this flexibility, we prompt PhenoAssistant to replicate part of the workflow in Williams et al.39 for potato leaves, assessing the correlation between manually measured leaf area (𝐴) and dry weight (𝑊), and evaluating if this correlation holds with algorithmically computed PLA. This analysis can help plant scientists identify potential discrepancies between manual and automated measurements. For this demonstration, we integrate the Leaf-only SAM model (for potato leaf segmentation) and use a dataset consisting of 32 potato images with 𝐴 and 𝑊 annotated; both the model and dataset originate from Williams et al.39.

As shown in Fig. 4, PhenoAssistant is asked to compute PLA and it follows the similar procedure to the previous case study while using a different segmentation model (Task 1). It is then tasked to measure and analyse the Pearson correlation coefficient between 𝐴 and 𝑊, as well as PLA and 𝑊 (Task 2). As this correlation analysis is not predefined in the toolkit, PhenoAssistant leverages the Code Writer to accomplish it. Finally, PhenoAssistant summarises the results, highlighting the correlation coefficients and providing further explanations and context. The findings indicate that while the PLA derived from Leaf-only SAM is useful, it may introduce errors compared to manually measured leaf area when predicting dry weight. In Task 2, PhenoAssistant performs the same correlation analysis and reaches conclusions consistent with Williams et al.39.

In Case Study 3, we demonstrate that PhenoAssistant can expand its vision capabilities via the built-in automatic vision model training function, enabling it to tackle previously unseen phenotyping challenges (e.g. experiments with new data or in new environments).

As shown in Fig. 5, PhenoAssistant is prompted to identify nutrient deficiency for winter wheats in field, where no capable model is integrated in the toolkit. Recognising the need for image classification, it instructs the user to upload labelled data in the required format. After receiving the dataset, it invokes tools to automatically preprocess the data (e.g. splitting into training and evaluation subsets, applying data augmentation), initiate model training using the user-specified finetuning method (full-parameter finetuning or low-rank adaptation (LoRA)40; their differences are described in the Automatic vision model training subsection of the Methods section), and evaluate the model’s performance. Finally, the evaluation results are presented to the user, and the trained model is added to the model zoo for future use. The data used for this case study are generated from Yi et al.41,42.

In summary, we present three case studies illustrating how PhenoAssistant streamlines plant phenotyping with only natural language instructions, allowing plant researchers to concentrate on scientific discovery rather than technical training. These cases span various plant species, environmental conditions, and tasks, including phenotype extraction, pipeline reproduction, statistics computation and visualisation, plot analysis, statistical testing, knowledge retrieval, correlation assessment, and model training. Chat logs for all case studies are available in our code repository.

Evaluation of PhenoAssistant
Beyond the presented case studies, we evaluate PhenoAssistant’s adaptability and potential for broader applications via a series of tasks. Since PhenoAssistant leverages rapidly evolving AI technologies, this evaluation also offers a method for assessing new AI components (e.g. DeepSeek43,44) as replacements for PhenoAssistant's original modules. The evaluation covers three aspects: tool selection, Vision Model Selection, and data analysis. Prompts and detailed results for the evaluation are presented at Supplementary Note 3-5. Chat logs for the evaluation are available in our code repository. During the evaluation, no human feedback (e.g. corrections to tool selection or tool arguments) is provided.

We evaluate the tool selection ability of PhenoAssistant by assessing whether it can select appropriate tools with the correct parameters and in the correct order to complete a task. This evaluates PhenoAssistant's abilities of chaining multiple tools for task completion rather than the correctness of task outputs.

To conduct this evaluation, we create 20 tasks to represent various phenotyping requests with different plant species. For each task, we prompt PhenoAssistant to generate a sequence of tools with their arguments in a standalised format. The outputs from PhenoAssistant are evaluated by an LLM-based evaluator, termed Tool Evaluator (LLM-based evaluators have been applied in different domains15,45). The Tool Evaluator is an LLM (GPT-4o) provided with detailed information about PhenoAssistant's toolkit (e.g. descriptions of tool functionalities and arguments) and the user-specified task description. It is prompted to score the response of PhenoAssistant for each task on the following four aspects using a scale of 1–5 (where 1 is the lowest and 5 the highest):

1. Overall Chain: whether the tool chain is reasonable for addressing the task, and whether any crucial steps are missing or incorrectly ordered.
2. Tool Existence: whether each selected tool exists within PhenoAssistant’s toolkit.
3. Tool Appropriateness: whether each tool is suitable for the specific step where it is applied.
4. Arguments: whether the arguments for each tool are correctly specified.

The average scores across the 20 tasks are reported in Table 1, under the row Manager. Overall, PhenoAssistant consistently achieves a full score in tool existence correctness (5), indicating that it does not hallucinate tools outside its toolkit. Slightly lower scores in overall chain correctness (4.25) and argument correctness (4.3) suggest that PhenoAssistant might generate toolchains in an inappropriate order, omit certain steps, or provide less accurate arguments. These issues may also be caused by vague task descriptions, highlighting the importance of allowing users to clarify tasks through interaction with PhenoAssistant. Tool appropriateness correctness also performs well (4.65), but still indicates potential limitations in the tool understanding abilities of LLMs and the clarity and comprehensiveness of the tool descriptions provided within toolkit in PhenoAssistant.

Beyond evaluating the original design of PhenoAssistant, we also assess the potential of introducing a Critic agent. This agent is implemented as a prompted LLM (GPT-4o), provided with both the user-specified task and PhenoAssistant's toolkit information, to critique and refine the toolchains generated by the Manager. Each toolchain produced by the Manager is reviewed by the Critic and will be refined if issues are identified. The toolchains generated by the Manager-Critic interactions are assessed with the same Tool Evaluator in the Manager-only setting.

The average scores with the Critic are presented in Table 1, under the row Manager + Critic. The results show consistent improvements across all aspects compared to using the Manager solely. This suggests that strategy verification mechanisms (e.g. adding a Critic agent) can enhance the reliability of PhenoAssistant on broader tasks and reduce the burden and frequency of human verification, which should be considered in future extensions.

The system prompts for the Tool Evaluator and the Critic are provided in Supplementary Note 6. Detailed per-task scores for the tool selection evaluation are reported in Supplementary Table 2.

Next, we evaluate whether PhenoAssistant can recommend the appropriate type of computer vision model (instance segmentation, image classification, or image regression) for a given plant phenotyping task. We term this evaluation Vision Model Selection I. It is crucial for automatic model training, as users may not have substantial machine learning knowledge to associate plant tasks with computer vision models. Accurate suggestions from PhenoAssistant prevent users from preparing incorrect training data.

To conduct this evaluation, we create 50 plant tasks expressed in varied formats: short phrases (e.g. leaf counting), statements (e.g. highlight the weeds in these field images), and questions (e.g. what is the maturity status of these pods?). This variation aims to cover different ways users might naturally express tasks. PhenoAssistant is then asked to select the most appropriate model type corresponding to each task description. The outputs of PhenoAssistant are manually inspected.

PhenoAssistant achieves 100% accuracy in Vision Model Selection I, indicating its potential to activate the correct model training function given a wide range of plant tasks.

In addition to evaluating PhenoAssistant's ability to select the correct model type, we further assess whether it can identify the most suitable specific model from a vision model zoo for a given plant phenotyping task. We term this evaluation Vision Model Selection II. It is designed to test whether PhenoAssistant can handle broader phenotyping scenarios when its model zoo expands.

To perform this evaluation, we adapt tasks from Vision Model Selection I and create 20 tasks covering a wide range of plant species and phenotyping needs (e.g. Identify all the wheat spikes in these images, and How badly are these sunflowers affected by disease? Provide a score for each image.). We also create a vision model zoo containing 30 model identifiers. Each identifier follows the convention {plant-species}_{task}_{model} (e.g. wheat_spike-instance-segmentation_m2fb, sunflower_disease-severity-regress_dino2b). The naming convention of PhenoAssistant's vision model zoo is described in the Vision model zoo subsection of the Methods section. Among these 30 model identifiers, 15 identifiers correspond exactly to 15 of the 20 plant tasks, while the remaining 15 identifiers serve as distractors.

PhenoAssistant is then asked to match each plant task with the most suitable model identifier from the model zoo. It achieves 100% accuracy in this evaluation: for the 15 tasks with corresponding model identifiers, it provides the correct match; for the remaining 5 tasks without a direct match, it not only recognises that no suitable model is available but also recommends an appropriate model type for fine-tuning. This result demonstrates PhenoAssistant's potential to address a wider range of plant phenotyping tasks with an extended vision model zoo.

Finally, we assess whether PhenoAssistant can produce the correct outputs for different data analysis requests. This aims to assess the reliability of the built-in LLM agents (e.g. Data Visualiser) for data analysis.

To conduct this evaluation, we create 20 phenotype analysis tasks using actual data. These tasks cover a range of operations, extracting values from files, computing statistics, generating plots, analysing plots, converting file formats, computing and analysing new metrics. The outputs of PhenoAssistant are manually inspected.

PhenoAssistant achieves 85% (17/20) accuracy in this evaluation. All failed cases fall within the plot analysis category. These tasks require the use of the plot analyser agent (a prompted GPT-4o) to interpret and extract fine-grained information from figures. This suggests that fine-grained visual reasoning remains a bottleneck for LLMs.

In summary, our evaluation shows that PhenoAssistant, as a multi-agent AI system, exhibits promising adaptability to broader plant phenotyping scenarios and tasks, while strategy verification mechanisms and improved visual interpretation capabilities should be further explored.

Methods
LLMs and agents
LLMs are deep neural networks with billions of parameters trained on large-scale text corpora to predict the next token in a sequence. Although this objective is simple, it enables LLMs to acquire emergent abilities such as natural language understanding, reasoning, and code generation. Recent models, such as GPT-4o6, have also extended to multi-modal training (e.g. using both text and image data), and hence expanded their capabilities to vision-language interpretation, making them suitable to support more complex tasks.

The versatile abilities of LLMs make them effective foundations for AI agents (and hence also termed LLM agents). These agents leverage LLMs as core intelligence for reasoning, decision-making, and performing actions to achieve specified goals such as operating structural data or retrieving knowledge from literature. This is typically achieved by providing an LLM with (i) a carefully crafted system prompt, which defines the agent's role and acting logics (e.g. to follow specific instructions or avoid common mistakes), and (ii) access to external tools, such as computational functions, APIs, or databases, which extend the abilities of an LLM beyond those acquired from model training. When multiple agents and tools with specialised functionalities are integrated into a multi-agent AI system, the system as a whole can collectively address complex tasks that exceed the capacity of a single agent. To ensure that a such system aligns with its objectives, both system-level design (e.g. incorporating human-in-the-loop mechanisms) and element-level design (i.e., selecting appropriate agents and tools) need to be carefully considered.

Implementation of PhenoAssistant
PhenoAssistant comprises a Manager agent and a specialised toolkit designed for plant phenotyping tasks. The Manager provides an interactive interface for handling user requests, planning tasks, selecting and executing tools, and summarising results. The Manager is implemented using GPT-4o (version: 2024-08-06, via OpenAI Azure API), which was the state-of-the-art LLM when PhenoAssistant was developed, with the model temperature set to 0.1, guided by a system prompt (Supplementary Note 6) that instructs it to ensure desired functionality and behaviour.

Specifically, the system prompt instructs the Manager agent to (i) generate a step-by-step plan and refine it iteratively, (ii) coordinate the use of the toolkit by chaining outputs and inputs across different tools and agents, (iii) follow specialised guidelines when using specific tools or agents (e.g. the Manager must verify the availability of a vision model before invoking it), and (iv) conclude interactions with a concise summary.

The toolkit consists of Python-based functions, each accompanied by structured schema (name, description, parameters, input/output format). This enables the Manager agent to understand the tools' capabilities and compose them properly into valid workflows. Notably, PhenoAssistant's toolkit includes both deterministic modules (e.g. instance segmentation and statistical tests) and LLM agents (e.g. code writer and plot analyser).

In terms of software and packages, PhenoAssistant is implemented in Python, with tool invocation and multi-agent orchestration supported by AutoGen. Computer vision models and model training functions are implemented using PyTorch and the Transformers library. The Table Analyser agent is implemented using Pandas AI. We introduce the key tools below; other tools as well as a complete list of software dependencies and version numbers are available in our code repository.

Vision model zoo
Many plant phenotypes cannot be extracted without the aid of image processing or computer vision techniques. Although recent LLMs have incorporated vision capabilities, they are not yet powerful enough for effective extraction of plant traits. For example, tasks such as delineating individual leaf boundaries (known as instance segmentation in computer vision), which is a preliminary task for computing leaf area and count, remain challenging.

To overcome this limitation, PhenoAssistant integrates external computer vision models. Each vision model is uniquely identified using the naming convention: {plant-species}_{task}_{(optional) training-dataset}_{model}_{(optional) finetuning-method}. These identifiers are maintained within a structured file (e.g. model_zoo.json, which the Manager agent accesses whenever a vision model is required, ensuring appropriate selection for the given plant task.

For demonstration purposes, in Case Study 1, we integrate Mask2Former71 for segmenting individual Arabidopsis leaves. Following the similar practice in72, Mask2Former is fine-tuned on subsets A1 and A4 of the publicly available CVPPP LSC dataset73. In Case Study 2, we integrate Leaf-only SAM39 for potato leaf instance segmentation.

Automatic vision model training
Given the diverse conditions encountered in plant phenotyping, such as varying illumination, indoor/outdoor environments, species diversity, and diverse phenotyping needs, it is practically impossible to pre-integrate every required vision model into PhenoAssistant. Currently, there is also no universal vision model capable of solving all phenotyping tasks. Furthermore, new tasks and customised user requirements continuously emerge, highlighting the need for a convenient mechanism to expand PhenoAssistant’s vision model zoo.

To satisfy this need, PhenoAssistant incorporates an automated model training pipeline for common phenotyping tasks (e.g. image classification). When users identify the need for a new vision model, or when PhenoAssistant determines that its current model zoo cannot adequately solve a provided task, it prompts the user to upload a dataset formatted according to predefined specifications. PhenoAssistant calls tools to automatically split the uploaded dataset into training and validation sets and initiate model training. Upon completion, the newly trained model is automatically added to the vision model zoo using the naming convention described earlier and becomes available for inference.

The core idea behind this automated training pipeline is to fine-tune pre-trained vision models using plant-specific datasets74. These models are initially trained on large-scale general data and can be fine-tuned to specific plant analysis tasks with limited additional data, thereby minimising development and deployment efforts. Specifically, for image classification (as shown in Case Study 3), we employ the DINOv2-base model75 as the pre-trained model.

PhenoAssistant supports two fine-tuning strategies: low-rank adaptation (LoRA)40 and full fine-tuning, to accommodate users with varying levels of computational resources. LoRA updates only a small subset of parameters by inserting low-rank trainable matrices into existing model layers, substantially reducing computational and memory requirements during training. This approach enables rapid adaptation at the cost of a modest performance trade-off. In contrast, full fine-tuning updates all model parameters, potentially achieving higher accuracy while at increased computational and memory demands.

LLM agents for phenotype analysis
Beyond phenotype extraction, PhenoAssistant supports diverse data analysis tasks such as statistics computations, data visualisation, and interpretative plot analysis. These analyses often involve complex and customised requirements that cannot always follow pre-defined logic. For example, users may request specific visualisation styles (e.g. plotting plant growth curves for different ecotypes using specific colour), or novel analyses prompted spontaneously during exploration.

To support these dynamic and evolving analytical needs, we repurpose GPT-4o (same as the manger agent, version 2024-08-06 via the OpenAI Azure API with temperature set to 0.1) into different roles using different system prompts (Supplementary Note 6). This configuration creates a set of distinct LLM agents, each of which can be invoked by the Manager when required:

1. Code Writer: Generates and executes Python code for performing tasks that are not previously defined in the toolkit. Code execution is performed with automatic error handling and retry logic. Its system prompt explicitly instructs the agent to output complete, runnable scripts.
2. Data Visualiser: Generates and executes Python code for producing plots with user specification. Guidance for creating clear and well-designed plots (e.g. including legends and beautifying layout) is provided within its system prompt.
3. Plot Analyser: Interprets plots using GPT-4o’s visual understanding capabilities.
4. Table Analyser: Queries values and computes statistics from CSV files using Pandas AI. Pandas AI is an agent framework that augments an LLM (GPT-4o in our case) with predefined logic and access to specialised tools (e.g. Pandas data analysis library) to allow natural language queries for data analysis.
5. Pipeline Reproducer: Enhances result reproducibility by extracting executed function calls and code from the chat history, organising them into a new Python function that can be reused in the future. Its system prompt instructs it to extract only the functions and code snippets that were actually executed, while excluding failed ones. The system prompt also provides an example of an extracted pipeline to ensure the output is valid and can be re-run on new datasets without modification.
6. RAG Agent: Expands PhenoAssistant’s domain-specific knowledge by embedding information from literature and retrieving relevant knowledge to support user's tasks.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "This paper presents PhenoAssistant, an innovative AI-driven platform that simplifies plant phenotyping through natural language interfaces. By integrating a large language model (LLM) with a curated toolkit for phenotype extraction, visualization, and model training, the system reduces technical barriers and aim to enhance accessibility for researchers with limited computational expertise. The tool is demonstrated by three case studies and three aspects were evaluated as of its performance. These, both test cases and evaluation tasks, are all reasonable and represents an array of different needs in plant phenotyping. The main value of this paper is the first case to demonstrate AI agent applications for plant phenotyping (the application value). PhenoAssistant fills a gap which is likely needed, \"democratising\", the use of plant phenotyping. Usage will tell to what extent these types of tool will accelerate the advancement, incorporation and use of plant phenotyping in the plant sciences.\nMajor comment:\nAs this is a collection of tools into a kit, there are no clear contribution on new methods or methods as it is mainly based on using GPT-4o, and as such difficult to benchmark against other tools even if an internal evaluation is done on performance. Can the generalizaion ability of the AI agent be ensured by further tests?\nMinor comments:\nIt is not stated who performed the case studies. Probably by the authors who have great expertise in plant phenotyping methods and thinking, and thus could be criticised not to be the target group the PhenoAssistant tool is aimed at.\nIt could be more clearly stated in the introduction that this tool and other AI agents also bridges different expertise.\nTo more clearly highlight that that ChatGPT cannot correctly extract phenotypes from images (Supplementary Section A.1), this should be lifted in the text.\nHow is the tool maintained and envisaged to be developed?\nIndicate in the Appendix which specific tasks as part of the evaluation failed.\nThe methods are even if clearly aimed at non-LLM experts quite generically presented in the methods section. A refinement could be considered.", "reviewer_2": "This is a good paper, and timely needed content. However I have following concerns as a MAJOR concerns before I read the revised version for further publication:\n1) While different agents (e.g., Code Writer, RAG Agent, Data Visualizer) handle specific subtasks, they follow fixed responsibilities. There is no dynamic role negotiation or emergent division of laborhallmarks of Agentic AI.\n2) Although multiple agents are involved, they operate under a central LLM \"manager\" with no decentralized decision- making or peer-to-peer coordination, which constrains scalability and emergent intelligence.\n3) The paper lacks a proper literature in introduction to differentiate what AI Agents are and how are AI Agents different than Agentic AI.\n4) There is no mechanism for the agents to simulate tasks, explore solution spaces, or verify strategies before execution which is a key requirement for high-assurance applications and agentic behavior.\n5) Introduce agent-based simulation tools where agents can virtually test phenotyping hypotheses (e.g., nutrient deficiency responses(this is just an example, you can consider relevant example to your dataset but the example should be applied) before deploying real-world models\n6) Try to integrate causal modeling agents that reason about cause-effect relationships in plant physiology and validate hypotheses using both statistical and symbolic methods.\n7) Write more discussion on Agentic AI how will it be different than a normal AI Agent, present some recent findings and literatures how they define this both terms and also write the discussion as Scalable Infrastructure and Governance in the future discussion part.\n8) You could potentially think about shifting from a central-manager paradigm to decentralized agent collaboration with local autonomy, communication protocols, and a shared task environment—more aligned with Agentic AI frameworks such as ChatDev or CrewAI or more.", "reviewer_3": "Overall, this is an interesting and timely work that leverages the new wave of Agentic AI pipelines for automate plant phenotyping tasks. However, the main concerns are: (1) evaluation does not seem comprehensive, and (2) unclear how it may scale to add new tools and models to the toolkit. Hence, the claim seems a bit overblown.\nBelow are some detailed comments and suggestions.\n• In Figure 4, i.e., case study 3, what framework is being used to make the models? For example, how is the appropriate model size selected?\n• L218: It seems it is only predicting the type of model to select (e.g., instance segmentation, image classification, etc.), but which exact model is selected from the model zoo is not being evaluated.\n• The granularity of prediction in invasion model selection and the number of tool selections (currently 10) and data analysis tasks (currently 10 as well) should be evaluated.\n• L254: The accuracy of reproduced pipelines should also be given.\n• L304: It is rightly noted that it is practically impossible to pre-integrate every required vision model. To solve this problem, a mechanism for dynamically fetching onion models could help. The publicly available open-sourced models with the ’agriculture’ tag, for example, could be fetched from Kaggle. https://www.kaggle.com/models?query=agriculture&owner- type=user\n• L324: Where is the use of LoRA fine-tuning demonstrated in the presented use cases?\n• L580 - L635: The pipeline is supposed to predict one of these labels: ’instance segmentation, image classification, and image regression,’ but the inputs themselves give it away. For example, the inputs were: Flower color classification, Leaf disease segmentation, Fruit ripening stage classification. So, even though it is reported that this evaluation is pretty surface level, the reported performance of 98% does not mean much.\n• How is the model going to know if the image being asked for is trained for it? VLMs can take general input, but these specialized trained models cannot, so how are they fed? For example, if any image from another dataset is captured which does not correspond to training data, then it is a known limitation with traditional models.\n• Make a clear distinction between a module and an agent. For example, why is statistical test listed as an agent?\n• Model Context Protocol (MCP) is a standardized way to connect to external data sources and tools through a unified interface. Instead of PhenoAssistant creating its own custom toolkit integration system, they could have used MCP to connect their LLM to the vision models, statistical tools, etc.\n• Chat logs and evaluation results are provided in a public repository, but the software/tool itself is not available. Also, it would help to publicly document the method to add new agents in the toolkit, after making it publicly accessible.\n• A matrix table containing the feature comparison of PhenoAssistant with other existing tools would be a great addition. Some tools available online in this space are: https://github.com/Taranshah9/ MumbaiHacks2024_LlamaMia https://github.com/nalinrajendran/eco-agri-bot https://github. com/jayeshbhandarkar/AgriGenius", "reviewer_4": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts."}, "author_response": "Responses to Reviewers’ Comments\nTo All Reviewers:\nWe greatly appreciate your recognition of the value of our work and sincerely thank you for\nthe time and effort dedicated to reading and reviewing our manuscript. Taking the input from\nthe reviewers, we have now submitted a revised manuscript and supplementary information.\nFurthermore, we provide a point-by-point response to all the concerns raised by the\nreviewers below.\nBefore reading our responses, we kindly ask you to read the following notes:\nNote 1. How changes are indicated in the revised manuscript:\nWherever a response required modifications to the manuscript, the corresponding changes\nhave been highlighted in Orange and labelled with Rm.n, where m refers to the index of\nreviewer, and n represents the index of comment. For example, R1.2 refers to Reviewer 1’s\nComment 2.\nWe also revised other parts of the manuscript to better fit the context while addressing\nreviewers’ comments, correct typographical errors, and ensure consistent word usage.\nThese changes are highlighted in Orange but not necessarily labelled with Rm.n.\nNote 2. Reorganisation of sections:\nTo comply with the formatting requirements of Nature Communications, we reorganised\nthese sections:\n(i) We created a new Section 2 Results by consolidating the original Section 2 Design of\nPhenoAssistant (now Section 2.1), Section 3 Case Studies (now Section 2.2), and\nSection 4 Evaluation of PhenoAssistant (now Section 2.3).\n(ii) The layout of the new Section 2.3 Evaluation of PhenoAssistant has been revised for\nbetter clarity and comprehensiveness. Previously, we presented only a brief description\nof each evaluation and then discussed the results collectively at the end. In the new\nversion, we have expanded the descriptions, discussed the results of each evaluation in\ndetail separately, and provided a summary at the end.\n(iii) We created a new Section 3 Discussion by merging the original Section 5 (Conclusion)\nwith additional discussion on PhenoAssistant’s limitations and potential future directions.\n(iv) We moved the original Appendix A Supplementary Materials into a separate file titled\nSupplementary Information. The section labels in this new file have been updated\naccordingly (e.g. previously: A.2 Chat Log for Case Study 1 Task 1, now: Supplementary\nNote 2 Chat Log for Case Study 1 Task 1), and the corresponding cross-references in\nthe main text have been updated.\nReviewer #1 (Remarks to the Author):\n[... Author responses to Reviewer 1 ...]\nReviewer #2 (Remarks to the Author):\n[... Author responses to Reviewer 2 ...]\nReviewer #3 (Remarks to the Author):\n[... Author responses to Reviewer 3 ...]\nReviewer #4 (Remarks to the Author):\n[... Author responses to Reviewer 4 ...]"}, {"round": 2, "reviewer_comments": {"reviewer_1": "I am pleased with the extensive response and also the changes made to the manuscript based on my comments. I have no further comments", "reviewer_2": "Great Revision. However, there are still spaces to improve the paper with minor and important changes\nIn the abstract, a proper GitHub repository link could be added as the last sentence.\nFigure 2, 3 and 4 caption could be more detailed for better read(basic to advanced reader audience should be focused for captioning such figures)\nFigure 5, the leaf count, plant diameter and plant perimeter graphs are hard to see and understand even with glasses. Please enlarge all those graphs to PLA size graph at least for better visibility", "reviewer_3": "Here are my feedback on the authors' responses to my questions.\n• In Figure 4, i.e., case study 3, what framework is being used to make the models? For example, how is the appropriate model size selected?\nResponse Assessment:\n– Satisfied/Unsatisfied: Satisfied\n– Additional Comments: I agree the model size and optimizations become secondary in this context.\nThe authors say ”we do integrate both LoRA [39] and full-parameter finetuning options to allow\nusers adapt to their hardware (e.g. GPUs), dataset size, and requirements for model performance.”\nHowever, there is a caveat that it takes more effort on the logistical side of setting up everything\non the computer, which could vary from one system to another, to use the full capability of the\nhardware.\n• L218: It seems it is only predicting the type of model to select (e.g., instance segmentation, image classification, etc.), but which exact model is selected from the model zoo is not being evaluated.\nResponse Assessment:\n– Satisfied/Unsatisfied: Unsatisfied\n– Additional Comments: While I acknowledge the authors’ explanation that they evaluate model\ntype prediction rather than exact model selection, this evaluation task is too simplistic to properly assess the framework’s\ncapabilities. The high accuracy numbers reported don’t provide meaningful confidence in the system because the task of\ncategorizing into broad model types (instance segmentation, image classification, etc.) is relatively easy. A more challenging\nevaluation would better demonstrate the system’s real-world utility.\n• The granularity of prediction in invasion model selection and the number of tool selections (currently 10) and data analysis tasks (currently 10 as well) should be evaluated.\nResponse Assessment:\n– Satisfied/Unsatisfied: Satisfied\n– Additional Comments: I acknowledge that the authors added meaningful numbers and evaluation\ndetails for the current scope of their toolkit. However, this does not fully address the scalability\nissue since the process remains manual. For the scope of this paper, the evaluation is adequate,\nbut this manual limitation should be noted as a caveat for future scalability.\n• L254: The accuracy of reproduced pipelines should also be given.\nResponse Assessment:\n– Satisfied/Unsatisfied: Indifferent\n– Additional Comments: Case study 1 provides anecdotal evidence that sparks confidence, but\nreproduction is limited to one case only.\n• L304: It is rightly noted that it is practically impossible to pre-integrate every required vision model. To solve this problem, a mechanism for dynamically fetching onion models could help. The publicly available open-sourced models with the ’agriculture’ tag, for example, could be fetched from Kaggle. https://www.kaggle.com/models?query=agriculture&owner-type=user\nResponse Assessment:\n– Satisfied/Unsatisfied: Satisfied\n– Additional Comments: The authors rightly noted the limitations of fetching online models and\nthus pre-fetching is indeed necessary in this context to add reliability.\n• L324: Where is the use of LoRA fine-tuning demonstrated in the presented use cases?\nResponse Assessment:\n– Satisfied/Unsatisfied: Satisfied\n– Additional Comments: It has now been added in revision.\n• L580 - L635: The pipeline is supposed to predict one of these labels: ’instance segmentation, image classification, and image regression,’ but the inputs themselves give it away. For example, the inputs were: Flower color classification, Leaf disease segmentation, Fruit ripening stage classification. So, even though it is reported that this evaluation is pretty surface level, the reported performance of 98% does not mean much.\nResponse Assessment:\n– Satisfied/Unsatisfied: Satisfied\n– Additional Comments: The list of evaluation tasks has been diversified. However, the question of\nusing this as a metric to validate the framework is still questionable (as shared above). But I am\nsatisfied with the concern of having revealing labels in the input.\n• How is the model going to know if the image being asked for is trained for it? VLMs can take general input, but these specialized trained models cannot, so how are they fed? For example, if any image from another dataset is captured which does not correspond to training data, then it is a known limitation with traditional models.\nResponse Assessment:\n– Satisfied/Unsatisfied: Indifferent\n– Additional Comments: This is rightly noted as a limitation (out of distribution). However, the\nexpectation that experts will be able to identify issues from intermediary outputs is not reasonable\nbecause images could be taken from different cameras and human eye/expert cursory evaluation\nmight not catch such cases. This could be a low hanging fruit for further improvement. If the\nsystem could detect out of distribution cases, that would add hugely to the reliability of the\nsystem.\n• Draw a line between calling it a module and an agent. For example, why is statistical test listed as an agent\nResponse Assessment:\n– Satisfied/Unsatisfied: Satisfied\n– Additional Comments: Thanks for clarifying in text.\n• Model Context Protocol (MCP) is a standardized way to connect to external data sources and tools\nthrough a unified interface. Instead of PhenoAssistant creating its own custom toolkit integration\nsystem, they could have used MCP to connect their LLM to the vision models, statistical tools, etc.\nResponse Assessment:\n– Satisfied/Unsatisfied: Satisfied\n– Additional Comments: The authors acknowledge MCP as a valid alternative but note their custom\nintegration was developed before MCP became widely adopted. While MCP would indeed reduce\nredundant integration code, the current system’s modular design allows for migration to MCP in\nfuture versions.\nThe code is now public on GitHub along with v1 and v2 releases. Instructions to run the tool\nand add new agents are included. Chat logs and evaluation results are also available.\n• A matrix table containing the feature comparison of PhenoAssistant with other existing tools would\nbe a great addition. Some tools available online in this space are: https://github.com/Taranshah9/\nMumbaiHacks2024_LlamaMia https://github.com/nalinrajendran/eco-agri-bot https://github.\ncom/jayeshbhandarkar/AgriGenius\nResponse Assessment:\n– Satisfied/Unsatisfied: Satisfied\n– Additional Comments: The table includes both benchmarks and tools, but it still gives a good\noverview. And I meant overview in terms of features, which could be more granular, such as\nspecifying the types of phenotyping tasks or plant trait assessments supported.", "reviewer_4": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts."}, "author_response": "Responses to Reviewers’ Comments\nTo All Reviewers:\nWe sincerely thank you for your time and effort in reviewing our revised manuscript and\nresponse letter. We greatly appreciate the constructive comments you provided, which have\nhelped us to further improve our work. We have addressed all concerns, as detailed in our\npoint-by-point responses below.\nPlease note that all revisions made in the manuscript and supplementary information are\nhighlighted in orange. For better maintenance, we have also transferred our code repository\nto our organisation’s Github page at https://github.com/vios-s/PhenoAssistant/.\nReviewer #1 (Remarks to the Author):\n[... Author responses to Reviewer 1 ...]\nReviewer #2 (Remarks to the Author):\n[... Author responses to Reviewer 2 ...]\nReviewer #3 (Remarks to the Author):\n[... Author responses to Reviewer 3 ...]\nReviewer #4 (Remarks to the Author):\n[... Author responses to Reviewer 4 ...]"}, {"round": 3, "reviewer_comments": {"reviewer_3": "I am happy with the careful responses from the authors, no further comments."}, "author_response": "Responses to Reviewers’ Comments\nTo Editor and All Reviewers:\nWe sincerely thank you for your time and effort in reviewing our revised manuscript and\nresponse letter. We appreciate your constructive comments, which have helped us to\nimprove our work. We are happy that all reviewers were satisfied with our responses."}]

Decision letter