# Towards LLM Agents for Model-Based Engineering: A Case in Transformation Selection

Zakaria Hachm, Théo Le Calvar, Hugo Bruneliere, Massimo Tisi

► To cite this version:

Zakaria Hachm, Théo Le Calvar, Hugo Bruneliere, Massimo Tisi. Towards LLM Agents for Model-Based Engineering: A Case in Transformation Selection. SAM 2025 - 17th System Analysis and Modelling conference, Oct 2025, Grand Rapids, MI, United States. 10.1109/MODELS-C68889.2025.00061. hal-05228415

HAL Id: hal-05228415

https://hal.science/hal-05228415v1

Submitted on 28 Aug 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Towards LLM Agents for Model-Based Engineering: A Case in Transformation Selection

Zakaria Hachm, Théo Le Calvar, Hugo Bruneliere, Massimo Tisi

IMT Atlantique, LS2N (UMR CNRS 6004)

Nantes, France

firstname.lastname@imt-atlantique.fr

**Abstract**—In Model-Based Engineering (MBE), practitioners frequently face the challenge of selecting appropriate tools from a large number of options. This requires both deep domain-specific knowledge and technical expertise. LLM-based agents are software components that depend on Large Language Models (LLMs) to autonomously select and apply software tools to perform specific tasks. Although LLMs have already been applied to support various MBE activities, considering LLM-based agents to autonomously assist users of MBE tools remains underexplored. This is particularly challenging in industrial MBE environments where only medium-sized on-premise LLMs can be used due to company policies related to security or data privacy (for instance). To investigate the potential of LLM-based agents for MBE, we start with model-to-model transformation as a core MBE technique. Currently, off-the-shelf agents such as Microsoft Copilot can invoke a transformation engine (e.g., ATL) when the task is explicitly described. However, these agents struggle to select the correct transformation when they only have limited contextual information, especially when coupled with medium-size LLMs. To overcome this, we propose an approach based on complementary mechanisms. First, we build a model transformation server and an LLM agent with dedicated tools for each transformation available on the server. Second, to enable the agent to efficiently select transformations, we rely on a tool retrieval technique based on a tool relevance score computed by an LLM. We evaluate our LLM agent on a transformation dataset we also contribute to the community. Our comparative study shows that the newly proposed LLM agent responds more accurately to user instructions.

**Index Terms**—Model-Based Engineering (MBE), Model-to-Model Transformation, Large Language Models (LLMs), Agents, ATL Transformation Language

## I. INTRODUCTION

In Model-Based Engineering (MBE), practitioners often face the challenge of identifying relevant tools from a large range of options. Such a task usually requires a deep domain-specific knowledge, as well as technical expertise. Model transformation [1], as a core MBE tool, is a good illustration of this. When the number of available (versions of) transformations increases, it becomes challenging to select the appropriate one or to compose the right chain of transformations. This requires mastering transformation patterns and syntax which can be difficult for both novice and experienced users [2].

LLM-based agents [3] are software systems that leverage Large Language Models (LLMs) to autonomously select and apply software tools to perform given tasks [4]. These agents combine the reasoning capabilities of LLMs with the ability to invoke specialized tools for complex tasks (e.g.,

web search [5], code execution [6]). In practice, they plan, execute, and verify workflows using external systems to ensure accuracy and precision [7], [8].

Such LLM agents are promising in the context of MBE, where practitioners often face complex, repetitive, and knowledge-intensive tasks that could benefit from automation and natural language-based interfaces [9]. Although LLMs have already been applied to support some MBE activities, the use of LLM agents to autonomously select and apply MBE tools remains underexplored. This is notably the case of model transformation that we study in this paper: engineers can express transformation scenarios using complex text in natural language, several transformations may share similar source/target metamodels while having different purposes, etc.

In addition, industrial settings often come with strong security and privacy constraints regarding the use of LLMs. For this reason, companies may require the use of medium-size LLMs operating on-premise rather than more powerful, larger cloud-based ones. We are currently facing such constraints in the MATISSE project [10] where we want to develop LLM agents that support the MBE of Digital Twins for the industry.

In an initial experiment, we make the ATL model transformation engine [11] available to LLM agents by building an ATL server that implements the Model Context Protocol (MCP) [12]. In this way, an off-the-shelf agent such as Microsoft Copilot can autonomously launch the execution of a transformation on this server when this transformation is explicitly described in a user prompt. However, such an agent struggles to select the correct transformation when it must infer the task from more contextual or partial information. This is more particularly the case when having to use only medium-size LLMs (e.g. Llama 3.2).

To overcome these limitations, we propose a *transformation agent* using two complementary mechanisms. We first provide a model transformation server that exposes available transformations as separate tools together with automatically generated descriptions to guide tool selection and application [13]. However, when the number of transformations increases, the accuracy drops to zero as the descriptions of all the transformation tools exceed the context window of the LLM. We address this scalability issue by employing a tool filtering technique, as a common practice in tool retrieval [4]. We use an LLM to produce tool relevance scores that measure semantic relationships between user instructions
and transformation descriptions. As a result, we can present only highly relevant tools to the agent.

We evaluate our transformation agent using a dataset containing 1,200 transformation instructions paired with ground-truth API calls. The method used to produce this dataset is possibly adaptable to other synthetic dataset generation processes. Through our evaluation, we want to address the following research questions.

* **RQ1:** How effectively can the transformation agent select the appropriate tools from a given tool set, and invoke them with the appropriate parameters?

* **RQ2:** How does the performance of the transformation agent scale when dealing with an increasing number of available transformations?

* **RQ3:** How does the complexity of the instructions, in terms of the required number of reasoning steps, impact the accuracy of the transformation agent?

Our comparative study shows that the transformation agent outperforms baseline agents in responding appropriately to user instructions, achieving higher accuracy when selecting from a larger set of transformations.

In summary, the main contributions of the paper are:

* **An LLM-based transformation agent:** An agent optimized for model transformation selection and application, based on 1) dedicated tools per transformation combined with 2) tool filtering via relevance scores.

* **A dataset for transformation selection:** A synthetically generated dataset containing 1200 instructions paired with ground truth API calls, along with a reusable methodology for generating other evaluation datasets.

* **An MCP server for ATL.** An open source ATL server (with 55 fully operational model transformations) that implements MCP, and so is directly usable by any LLM-based agent supporting MCP (e.g., MS Copilot).

This paper is organized as follows. Section II provides our background and motivation through a running example. Section III introduces our transformation agent and its selection approach based on tool filtering. Section IV presents the ATL MCP server and the transformation agent dataset. Section V describes the evaluation setup and analyzes the obtained results. Section VI reviews the related work, while Section VII concludes with a discussion on future work.

## II. BACKGROUND AND MOTIVATION

To illustrate our work, we consider a practical example based on an existing transformation. `UMLActivityDiagram2MSProject` converts a UML activity diagram into a file in the Microsoft Project format. UML activity diagrams are graphical representations used to model workflows and business processes, showing the sequence of activities and their dependencies [14]. Project is a Microsoft tool for representing and managing tasks, timelines, and dependencies in the context of project planning. Thus, the objective is to automatically convert a workflow described as a UML activity diagram into a Microsoft Project schedule.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Input UML activity diagram.</figcaption></figure>

Figure 1 shows a sample input UML activity diagram. The workflow begins at the initial node, proceeds to action node S1, splits in a fork pseudonode to execute S2 and S3 in parallel, synchronizes in a join pseudonode, continues to S4, and ends at the final node.

```xml
rule ActivityNode {
    from
        s : UML!ActivityNode (
            s.oclIsKindOf(UML!FinalNode)
            or s.oclIsKindOf(UML!Action)
        )
    to
        t : MSProject!Task (
            UID <- thisModule Id(),
            name <- s.name,
            predecessors <- s.getPredecessors(),
            duration <- 1 --days
            startDate <- ...
        )
}
```

<caption>Listing 1: Excerpt from the `UMLActivityDiagram2MSProject` transformation implemented in ATL.</caption>

Listing 1 shows an ATL rule from the `UMLActivityDiagram2MSProject` transformation. The `NodeVertex` rule transforms UML `ActionNode` and `FinalNode` elements into `MSProject` `Task` elements. It assigns a unique identifier using the `getId()` helper and computes task dependencies through the `getPredecessors()` helper. The latter navigates through the fork and join pseudonodes to establish proper predecessor relationships based on the workflow structure.

<table><caption>TABLE I: Output Microsoft Project diagram.</caption><thead><tr><td>ID</td><td>Task name</td><td>Duration</td><td>Start</td><td>Finish</td><td>Pred.</td></tr></thead><tbody><tr><td>1</td><td>S3</td><td>1 Day</td><td>Tue 04/01/25</td><td>Wed 05/01/25</td><td>S1</td></tr><tr><td>2</td><td>S1</td><td>1 Day</td><td>Mon 03/01/25</td><td>Tue 04/01/25</td><td>-</td></tr><tr><td>3</td><td>S2</td><td>1 Day</td><td>Tue 04/01/25</td><td>Wed 05/01/25</td><td>S1</td></tr><tr><td>4</td><td>S4</td><td>1 Day</td><td>Wed 05/01/25</td><td>Thu 06/01/25</td><td>S2, S3</td></tr><tr><td>5</td><td>Final</td><td>1 Day</td><td>Thu 06/01/25</td><td>Fri 07/01/25</td><td>S4</td></tr></tbody></table>

Table I shows the resulting `MSProject` schedule with the proper timeline. It has a parallel task scheduling where S2 and S3 can execute concurrently after S1. S4 is properly scheduled to begin only after both parallel branches have completed. Each task receives a unique identifier, a default duration (1 day), and start/end dates that respect logical workflow dependencies.

In the context of our work, we consider a scenario where a user wants to transform an activity diagram into a Microsoft Project schedule but is unaware of the existing transformation(s). The user may use an LLM-based agent a straightforward prompt such as “Can you transform this activity diagram”
into a Microsoft Project representation in a table?". We provide this particular prompt to an instance of Microsoft Copilot, integrated with Llama 3.2 and connected to an MCP server that hosts a transformation tool capable of handling this task (see Section V-B for more details on this server). We repeat the same experiment 10 times.

<table><caption>TABLE II: Tool usage behavior of Llama 3.2 on 10 iterations of the same prompt.</caption><thead><tr><td>Behavior Type</td><td>Llama 3.2</td></tr></thead><tbody><tr><td>No tool called (LLM inference)</td><td>70%</td></tr><tr><td>Wrong tool + wrong parameters</td><td>30%</td></tr><tr><td>Correct tool + wrong parameters</td><td>0%</td></tr><tr><td>Correct tool + correct parameters</td><td>0%</td></tr></tbody></table>

As shown in Table II, the behavior of the Copilot agent when asked to perform a transformation can vary. In most cases, the agent relies solely on LLM inference to compute the result, without invoking any transformation tool. This results in outputs different from the expected ones (cf. Table I). In other cases, it selects a transformation based on prior knowledge from its training data, but not necessarily the one best suited to the task. Even when the prompt and tool descriptions explicitly include enough context, the agent may still fail, typically, by calling the tool with incorrect parameters.

This example illustrates the limitations of the Copilot agent coupled with a medium-sized LLM (i.e., Llama 3.2) in the context of transformation selection. It highlights the need for a more robust strategy for selecting and applying transformation tools, by taking advantage of a more precise and contextual understanding of the task to be performed.

### III. AN LLM-BASED TRANSFORMATION AGENT

#### A. Dedicated Tools per Transformation

To address these limitations and enable transformation selection, we propose an approach that provides the LLM agent with explicit contextual information about each available transformation. We create dedicated tools for each transformation with explicit descriptions, so that the LLM can directly match user intent with the available transformations. We notably include in the description the name of the source and target metamodels of the transformation. In this way, the LLM can use them to select the transformation if needed, rather than inferring guesses about transformation input/output formats.

We automatically generate two distinct tools for each transformation: one for retrieving metadata about the transformation, and one for applying the transformation to a given source model. For our `ActivityDiagram2MSProject` example, we generate the `getInfo_ActivityDiagram2MSProject_tool` that provides associated metadata (cf. Listing 5), and the `apply_ActivityDiagram2MSProject_tool` that takes an `ActivityDiagram` model and produces a corresponding `MSProject` model. We provide each of the tools with a concise description that explicitly mentions the source and target metamodels of the transformation.

```python
getInfo_ActivityDiagram2MSProject_tool: This tool
displays the details of the transformation
ActivityDiagram2MSProject, transforming the
ActivityDiagram model into a MSProject model.
```

```python
apply_ActivityDiagram2MSProject_tool: Input
metamodel: ActivityDiagram, Output metamodel:
MSProject. This tool takes an ActivityDiagram
model as input and produces a MSProject model.
Input should ONLY be the file path to be
transformed.
```

#### Listing 2: Description of the tools for ActivityDiagram2MSProject.

Listing 2 shows the description of the two tools for ActivityDiagram2MSProject. The explicit metamodel names in the tool descriptions enable the LLM to identify the appropriate transformation when the user instruction contains indications on the desired source and target metamodels.

```python
tools = []
for i, name in enumerate(transformation_names):
    tools.extend([
        GetInfoOnTransformation(name),
        ApplyTransformation(name)
    ])
agent_executor = create_react_agent(
    llm,
    tools,
    state_modify=SYSTEM PromPT)
```

#### Listing 3: Tool dynamic creation in a ReAct agent.

As shown in Listing 3, we create the transformation agent in `LangGraph` by providing a `ReAct` agent [15] with the list of all available (information retrieval and application) tools. We also add a prompt template, partially shown in Listing 4, to enforce the behavior of the agent. This prompt template uses well-known techniques such as Chain-of-Thought [16], few-shot prompting [17], and role-based prompting [3]. It is expanded at runtime to a prompt containing the full list of tool descriptions. The goal is to explicitly guide the LLM in selecting and using the most appropriate transformation tools.

You are a transformation handler agent. Follow these rules strictly:

##### 1. LISTING TRANSFORMATIONS:

SINGLE transformation: Use list_transformation_<transformation_name>_tool
Input: NONE (leave input blank)
Example: To get, list or show the details of "Class2Relational", use the tool "get_Class2Relational_tool" without any input.

...

Here are your tools: {tools}

...

#### Listing 4: Excerpt of the prompt template of our Transformation agent.

The expansion of the prompt template includes the description of the tools and their interactions.
tion of two dedicated tools for each transformation. A primary limitation of this approach is that, as the number of available transformations increases, the expanded prompt grows proportionally and quickly reaches the limits of the LLM context window. The experiment we perform confirms this limitation in practice (cf. Section V). In fact, the transformation agent starts hallucinating and thus generating inconsistent responses when having to choose among 12 transformations and more.

## B. Transformation Selection by Relevance Score

To address this scalability limitation, we adopt the established community technique named tool filtering [4]. This approach has the benefit of presenting a small and focused subset of the available tools (i.e., the most relevant ones) to the agent. We use an LLM for inferring relevance scores for each tool w.r.t. current instructions. This scoring technique takes advantage of the documented ability of the LLMs to evaluate and rank options, and to act effectively as judges [18].

<figure><img src="image_3.png" /><figcaption>Fig. 2: Overview of the proposed approach for tool selection in the LLM-based transformation agent.</figcaption></figure>

Figure 2 provides an overview of the tool selection approach. We start by the *Get user input* action to obtain the *User request*. The *Analyze request* action breaks down this *User request* in order to extract the source *Metamodel name*, e.g., in the case where the user provided a file path. The LLM can potentially rely on this information to determine the most suitable transformation to use. We then iterate on the list of all available *Tools* (actually

proceeding in batches of 6 tools per iteration)$^1$. For each tool, we simply *Get descriptions*. As a result, we are able to provide the tool *Descriptions* and *Metamodel name* together with the *User request* and *Scoring prompt template* ready for the *Generate prompt action*. Each generated *Scoring prompt* is transferred to an *LLM call*, where the LLM assigns a relevance score ranging from 0 to 5 for each tool, in order to obtain *Scored tools*. This iterative process continues until all tools have been evaluated. Once completed, we *Select relevant tools* with the highest relevance scores from the *Scored tools*. These *Selected tools* are then used along with an *LLM* and the *Agent prompt* to build the *Transformation agent* at the end of the process.

## IV. DATASET GENERATION

To demonstrate the performance of the transformation agent, notably in terms of accuracy and scalability, we design and build a dedicated dataset. This also addresses a current limitation of MBE in terms of existing evaluation datasets [19]. The process we follow is inspired by the ones used in the literature to produce datasets for tool selection by LLM agents [20]. In the end, the resulting dataset and the associated ATL transformation server are made publicly available to the community [21]. Interestingly, they are possibly adaptable in the context of other research work.

### A. An ATL MCP Server

The ATL Zoo is an open source library of model-to-model ATL transformations [22]. However, it does not come with a single unified repository structure: the source and target metamodels, corresponding input and output models, and the ATL transformations are sometimes stored in different folders. In some cases, files required for transformation execution are missing. As a consequence, we reorganize these transformations and associated resources into a more systemically structured repository. Each transformation is now linked to a specific configuration file called *config.json*, as shown in Listing 5. This configuration file specifies the paths to the required metamodels, sample models, and ATL file. It also indicates the ATL compiler to be used and if the transformation is currently enabled. As a result, we design, implement, and make available to the community a more reliable and functional version of the ATL Zoo. It currently contains 55 fully operational transformations that cover diverse domains with distinct input and output metamodel types, each with all the necessary dependencies.

Moreover, we integrate these transformations into a dedicated ATL server that implements MCP. In this way, it can be directly plugged into LLM-based agents that support the protocol. The server acts as a central hub that manages and executes ATL transformations. It exposes RESTful API endpoints (e.g., GET and POST), and uses the structured ATL Zoo repository to locate transformation resources and perform

<footnote>$^1$Note that we determined this number by evaluating multiple batch sizes on different smaller datasets.</footnote>
1 # Level 1 - Single-Task - Transformation name mentioned
2 Seed(
3 instruction="Show me the configuration settings for the $transformationName transformation",
4 tool_calls="get_$transformationName_tool",
5 level=1),
6 # Level 2 - Single-Task - Metamodels mentioned
7 Seed instruction="Transform the $input_meta-model_name model $path into a $output_meta-model_name model",
8 level=2,
9 tool_calls="apply_$transformationName_tool ($path)"
10 # Level 3 - Single-Task - Target metamodel mentioned
11 Seed instruction="Transform this model $path to
12 $output_meta-model_name model",
13 tool_calls="apply_$transformationName_tool ($path)",
14 level=3)
15 # Level 1 - Multi-Task - Transformation names mentioned
16 Seed instruction="First show me the configuration of $transformation_name1 and then transform $path2 using $transformation_name2 transformation",
17 tool_calls="get_$transformationName1_tool, apply_$transformationName2_tool ($path2)",
18 level=1)

<caption>Listing 6: Single and multi-tasks example seeds.</caption>

output models for all available transformations. The pre-
viously Verified templates pass through a Values
injection step in which the retrieved information is in-
jected into them. The injected values, the structure of the
resulting instructions, and the expected API calls are also
verified. This approach leverages LLMs to create quality data
without the need to manually collect, clean, and annotate
massive datasets.

1 {"level": 1,
2 "instruction": "I need to see the default
configuration values for the
UMLActivityDiagram2MSProject transformation.",
3 "relevant APIs": [{
4 "api_name": "UMLActivityDiagram2MSProject.get_tool",
5 "arguments": ""}]
}

<caption>Listing 7: An example from the ATL agent dataset.</caption>

As a result, the ATL agent dataset is produced and
composed of 1,200 instructions (600 for single-task, 600 for
multi-task). It is based on real-word scenarios and associated
with the API calls we expect from their execution. As an
illustration, Listing 7 shows one example from the finally
generated ATL agent dataset.

V. EXPERIMENTAL EVALUATION

A. *Experimentation Setup*

Figure 4 presents the experimental setup we imple-
mented to evaluate our transformation agent. For each
item from the Dataset, the corresponding Instruction
and the list of expected tool calls (List<ToolName,
List<Argument>>) are first collected. The agent is then

<figure><img src="image_4.png" /><figcaption>Fig. 4: Overview of the evaluation process for the proposed agents.</figcaption></figure>

invoked with this Instruction to obtain the Agent
response that provides an inferred list of tool calls. The
latter is then matched with the list of expected tool calls for this
particular Instruction (as explained in Section IV, our dataset
contains the ground truth tool calls for each instruction). If
the called tools and all their arguments are equal in the two
lists, the agent receives a score of 1 for that particular
item of the Dataset. Otherwise, the score is set to 0. The
general process is repeated for all items of the Dataset.
This allows us to calculate a single Final score for the
evaluated agent as the average of the individual scores. This
"success" score indicates the ability of the evaluated agent to
produce accurate results with various instructions (from simple
to more complex). This score has already been used in other
similar contexts. For example, it is called Final response
evaluation where performance is assessed based on the
agent's ability to select the appropriate tools and provide the
correct arguments [8].

We compare the performance of our **Transformation agent** with three other LLM based agents. To assess the impact of the two mechanisms we propose in Section III (i.e., dedicated tool per transformation, and tool filtering by relevance score), we include two versions of the agent that do not have these mechanisms. Finally, we include in the comparison a state-of-the-art mechanism for generic tool filtering based on Retrieval-Augmented Generation (RAG). The resulting agents are:

*   **No dedicated tools per transformation.** This baseline agent uses the two generic transformations: `get_transformation_by_name` as an information retrieval tool to obtain configuration details for transformations using their names, and `apply_transformation_tool` as an execution tool to apply transformations on given input files. It also uses a `get_all_transformations_tool` that allows the agent to retrieve all available transformation names and their descriptions. This additional tool ensures that the agent can be aware of the transformations and their names for eventual calls. Otherwise, the agent would have no way to know which transformation names to use when calling a tool. The agent relies on prompt descriptions to guide the LLM in transformation selection
and application. This agent acts as our simple baseline to
evaluate the benefits of more sophisticated context-aware
and tool selection mechanisms.

* **No tool filtering.** This version of the transformation agent disables any tool filtering mechanism. The agent has dedicated tools for each one of the transformations, and tool selection is performed by a single LLM inference.

* **Tool filtering by RAG.** This version of the transformation agent uses RAG for tool filtering, as one of the popular approaches currently [26]. Instead of relying on relevance scores, this version combines semantic retrieval with tool selection. It maintains a vector database of transformation tools, where each tool and its description are embedded using semantic embeddings. When a user provides an instruction, the agent performs a similarity search to retrieve the most relevant tools from the vector store. As a result, the selected tools are then injected into the LLM for execution.

The experimental setup used for the evaluation was designed to address the research questions introduced in Section I. The dataset generation process, considering different levels of complexity and different types of tasks, aims to answer RQ1 (effectiveness of tool selection with the correct parameters). To address RQ2 (scalability when tool sets grow in size), we perform the evaluation several times by varying the number of tools available (20, 24, 30, 50, 70, 90, and 110). Finally, to answer RQ3 (impact of instruction complexity, measured by the number of required LLM reasoning steps), we separate the experiments by instruction complexity and number of tasks, and compare the average results. For our current evaluation, we rely on Llama 3.2:3B to ensure consistency in all our experiments. The source code of the four agents we used and the associated evaluation data are available online [21].

## B. Evaluation Results

Figure 5 and Table III summarize the results of the evaluation of all the agents on the dataset described in Section IV-B. We also follow the setup and process proposed in Section V-A. Figure 5 shows the average score for each agent (Y-axes), w.r.t. the number of tools available (X-axes). Note that the top row of charts is for single-task scenarios, while the bottom row of charts is for multi-task scenarios. Horizontally, the level of complexity increases from left to right (from 1 – Easy to 3 – Hard). Table III shows global average scores for each agent on each kind of instructions (grouped by complexity level + number of tasks).

In general, the results we obtain reveal interesting performance trends. For all agents, performance decreases or remains stable when the number of available transformation tools increases. This is expected since more tools make the selection process more difficult. For all four agents, we also observe a decrease in transformation accuracy as the level of complexity of the provided instructions increases, and as the task type changes from single-task to multi-task. This is expected because more complexity requires more reasoning steps to select and apply the transformation correctly. Thus,

this naturally increases the probability of reasoning errors at
each step. Moreover, multi-task scenarios introduce additional
complexity by requiring the agents to coordinate multiple
transformation operations. Again, this creates more opportu-
nities for errors in tool selection and parameter specification.
As a consequence, we can experimentally confirm that the
performance of the agents is influenced by the complexity of
the instructions and tasks.

Looking more into the details, each version of the agent
exhibits specific performance trends:

1) *No dedicated tools per transformation:* This agent performs consistently in all scenarios and maintains fixed scores when increasing the number of transformations. This is due to the fixed number of tools it exploits compared to the other agents, i.e., only 2 generic tools (to get transformation metadata and apply a given transformation). Although this simple agent performs well on straightforward tasks that do not require reasoning steps (level 1), it generally achieves low performance on more complex tasks (levels 2 and 3). The main reason is that its tool descriptions include only general information and thus lack specific detail about each transformation (e.g., their source and target metamodels). While the agent is often able to select the correct transformation based on knowledge in its training set, it struggles in matching complex instructions with the corresponding transformations.

2) *No tool filtering:* This agent demonstrates solid initial performance with a low number of tools, in particular around 24 tools. However, its performance quickly drops to near zero when considering more than 30 tools. This behavior is consistent for all levels of complexity, indicating a fundamental limitation in the agent architecture rather than a task-specific issue. This substantial decrease illustrates the limitations of the LLM context window when incorporating numerous tools. The size of such window depends on the used LLM (for us Llama 3.2). Once the context window is exceeded, the LLM generally fails to use the injected tools, leading to hallucinations as it attempts to fill the gaps with incorrect or fabricated data.

3) *Tool filtering by RAG:* The RAG agent demonstrates mixed performance across different complexity levels and task types. This reflects both the strengths and limitations of semantic retrieval for the selection of transformation tools. In level 1 scenarios, the RAG agent achieves strong performance with 88.5% accuracy for single-task scenarios, making it competitive with our transformation agent (92.6%). This success can be attributed to the fact that the metamodel name is specified in simple instructions, thus RAG can match it with transformation descriptions in the vector store. However, the performance of the RAG agent degrades as the complexity of instructions increases. While it maintains relatively stable performance in level 2 scenarios (65% single-task, 28.2% multi-task), it experiences significant degradation in Level 3 scenarios (26.5% single-task, 18.5% multi-task). Notably, the RAG agent consistently underperforms in multi-task scenarios across all complexity levels. This suggests fundamental limitations in matching different parts of the prompt in order to retrieve sequential operations.
<table><caption>TABLE III: Performance comparison (average scores)</caption><thead><tr><th>Level of Complexity</th><th>Tool filtering by RAG</th><th>No dedicated tools per transformation</th><th>No tool filtering</th><th>Transformation Agent</th></tr></thead><tbody><tr><td colspan="5" style="text-align: center; font-weight: bold;">Single-task (Average)</td></tr><tr><td>Level 1 (Easy)</td><td>88.5%</td><td>26.4%</td><td><strong>94.1%</strong></td><td>92.6%</td></tr><tr><td>Level 2 (Medium)</td><td>65%</td><td>17.9%</td><td>28.1%</td><td><strong>70.3%</strong></td></tr><tr><td>Level 3 (Hard)</td><td>26.5%</td><td>13.5%</td><td>0.4%</td><td><strong>51.9%</strong></td></tr><tr><td colspan="5" style="text-align: center; font-weight: bold;">Multi-task (Average)</td></tr><tr><td>Level 1 (Easy)</td><td>28.7%</td><td>21.1%</td><td><strong>89.8%</strong></td><td>76.3%</td></tr><tr><td>Level 2 (Medium)</td><td>28.2%</td><td>14.8%</td><td>23.5%</td><td><strong>54.9%</strong></td></tr><tr><td>Level 3 (Hard)</td><td>18.5%</td><td>12.9%</td><td>6.5%</td><td><strong>41.0%</strong></td></tr><tr><td><strong>Global Average</strong></td><td>42.6%</td><td>17.8%</td><td>40.4%</td><td><strong>64.5%</strong></td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Fig. 5: Performance by level and task type</figcaption></figure>

4) *Transformation Agent*: Our proposed agent demonstrates more resilient performance as the number of available tools increases, and it maintains relatively high scores even with 110 tools in most scenarios. The performance advantage of the transformation agent becomes even more significant when dealing with complex instructions, particularly in the context of multi-task scenarios. In that case, the tool filtering process we apply allows a better balance between tool availability and context window constraints. As a result, the transformation agent outperforms the three other agents in the most complex

scenarios. This is notably the case in the situations that involve the largest number of transformations.

### C. Answer to Research Questions

The evaluation results presented provide interesting insights on the research questions introduced in Section I.

Regarding RQ1 (effectiveness of tool selection and call with correct parameters), the transformation agent globally demonstrates a higher average capability in selecting appropriate tools and invoking correct parameters. It also outperforms
state-of-the-art RAG retrieval. We still observe that the LLM
may fail in several distinct ways: by not calling any tool or
selecting the wrong tool with incorrect arguments, the wrong
tool with correct arguments, or the correct tool with incorrect
arguments.

Regarding RQ2 (scalability when tool sets grow in size), the
results show that the transformation agent generally maintains
its accuracy when the number of tools grows, up to our current
maximum number of 110 tools.

Regarding RQ3 (impact of instruction complexity), the
transformation agent outperforms others at most levels and
for most types of tasks. For the simplest instruction (level
1), it shows only a slightly lower accuracy than the baseline
agent. However, in both single-task and multi-task settings, its
accuracy drops by approximately 22% from Level 1 to Level 2
(single-task: 92.6% to 70.3%; multi-task: 76.3% to 54.9%) and
by about 18% from Level 2 to Level 3 (single-task: 70.3% to
51.9%; multi-task: 54.9% to 41.0%). This clearly highlights
the significant impact of increasing the complexity of the
provided instructions, at least in our model transformation con-
text. The performance of the transformation agent is primarily
limited by its reliance on semantic similarity for tool selection.
Indeed, this works well for simple cases but struggles with
complex scenarios that require precise tool coordination. The
consistent underperformance of the transformation agent in
multi-task operations (28.7% vs 76.3% for the enhanced agent
in Level 1) indicates that semantic retrieval alone is insufficient
for coordinating complex transformation workflows where
multiple transformations share similar semantic profiles.

D. *Threats to validity*

Our dataset generation approach faces some limitations that
can affect validity. The manually created seeds may not capture
the full spectrum of user interaction patterns in real-world
ATL usage. This potentially introduces bias that can affect the
coverage of all possible real-world cases. These seeds, while
designed by actual ATL transformation practitioners, represent
a limited sample and may not reflect the diversity of complex
industrial MBE scenarios. Similarly, our complexity levels
(1 to 3) and task categorizations (single-task vs. multi-task)
provide a useful structured evaluation framework. However,
they may not contain all the variations that practitioners can
encounter in the context of real-world instructions.

Technical limitations can also be a source of threats to
validity. The batch size of 6 tools used in relevance scor-
ing was empirically determined on smaller datasets. It may
require further adjustment to support different configurations
depending on the actual LLM context window. More broadly,
other limitations imposed by the LLM context window can
still be faced depending on the LLM one decides to use. The
LangChain community highlighted this issue in a recent report
that indicates that some LLMs exhibit a low pass rate on
various agent tasks [27].

VI. RELATED WORK

In the context of our work, we explored the current state of
the art at the intersection of LLM-based agents using tools,
and LLM in/for MBE techniques and tools.

A. *LLM-based Agents Using Tools*

LLMs already demonstrated capabilities in natural language
understanding and generation, which quickly led to their
practical integration into autonomous agents. These agents
leverage LLMs to perform complex tasks and reason over
existing information, but also to interact with external tools.
A recent survey explores the construction of LLM-based
autonomous agents and proposes a unified framework that
encompasses various components such as planning, memory,
tool use, and evaluation [8]. This survey highlights the LLM
capabilities mentioned before and, most importantly in our
context, discusses challenges related to domain adaptation
and the corresponding tool integration. The work presented
in this paper, with its focus on improving the “transformation-
awareness” of LLM-based agents, goes in the direction of
addressing such challenges.

Another recent work in the field of autonomous agents
focuses on domain-specific tool-augmented LLMs that target
various software engineering challenges [28]. This work in-
troduces CodeAgent, a system that augments code generation
by incorporating tool-integrated agents specifically designed
for repository-level coding tasks. As in our case, integrating
tools allowed them to improve the performance of their agents
by 18.1% to 250% on their CodeAgentBench benchmark. In
the same vein, ChatDev is a virtual chat-powered software
development company that exploits LLMs throughout their
entire development process [29]. They propose to treat dif-
ferent activities separately (e.g., designing, coding, testing,
documenting), and to support them with specialized software
agents that collaborate to address particular tasks. We draw
inspiration from such an approach to provide specialized
agents for model transformation.

In parallel, more generic agent frameworks and domain-
agnostic methodologies have been developed. For example,
EasyTool transforms diverse and lengthy tool documentation
into more unified and concise tool instructions that are applica-
ble across domains [30]. Another work addresses tool retrieval,
as an important challenge [4], with an approach named COLT
(Collaborative Learning-based Tool retrieval) [31]. It captures
both semantic similarities and collaborative information of
tools through bipartite graphs. The TPTU framework proposes
to improve general task planning and tool usage in LLM-
based agents [32], [33]. T-Eval introduced the decomposition
and evaluation of tool utilization in multiple dimensions [34].
Toolformer provides a self-supervised model trained to decide
which APIs to call in a domain-independent way [35].

Such general-purpose agent frameworks face particular
challenges when applied to specialized technical domains [36].
This is notably the case in our context where very specialized
tasks such as model transformations require more specific
solutions. For example, generic frameworks can be constrained
by tool documentation within specific token limits, cannot
process instructions exceeding the LLM context window, and
can overlook inter-dependencies between tools [30]. In our
case, the large number of considered transformations, related
tools, and their descriptions are elements that prevent to
directly reuse such frameworks. Moreover, these frameworks
are often evaluated on widely known languages like SQL
and Python [32]. Thus, they are lacking capabilities needed
to ensure the support for less widespread domain-specific
languages (such as ATL for model transformation as in our
case) that would require more initial semantic learning.

Finally, for the design and generation of our reusable
dataset, we draw inspiration from the work around ToolLLM
that enables LLMs to master more than 16,000 real-world
APIs [20]. We attentively looked at their established good
practices while tailoring our dataset to address the research
questions introduced in Section I.

B. LLMs in/for MBE

The already existing applications of LLMs in/for MBE activities, techniques, and tools show both promises and limitations. In particular, while autonomous LLM-based agents with tool integration capabilities have been developed in various domains (cf. Section VI-A), MBE remains today largely unexplored in this regard. This situation has been highlighted in a very recent survey that studies the state of the art in terms of the different uses of LLMs in MBE [9]. This study examines current applications and proposes a forward-looking research agenda to improve current support. In particular, they propose to overcome the limitations observed when using LLMs alone in the context of modeling tasks by working on more specialized LLM agents that leverage external tools. Similar plans for LLM agents appear in the context of business process modeling activities [37]. The work we present in this paper is a concrete step in the direction of building relevant LLM-based agents for MBE.

Several approaches further highlight the current limitations
of LLMs in/for MBE. For example, an existing approach
evaluates the UML modeling capabilities of ChatGPT by
comparing them with its code generation capabilities [38].
As a result, it identifies significant limitations including syn-
tactic and semantic deficiencies, inconsistent responses, and
scalability. These limitations are aligned with the challenges
we initially encountered during our first experiments in the
context of model transformation. In the end, this confirmed
our decision to incorporate external tools rather than rely only
on direct calls to the LLM for transformation tasks. Another
work addressed the limitation of the LLM context window by
introducing PathOCL, a GPT-4-based approach that employs
a fragmentation technique to support the generation of OCL
queries [39]. In our model transformation context, we faced
similar issues that we also overcome by relying on comple-
mentary external tools.

As explained above, none of the identified approaches
specifically addresses LLMs for model transformation [9].
A recent work compares model transformations generated

by LLMs with existing transformations in dedicated lan-
guages [2]. They highlight the current limitations of LLMs
when it comes to automatically generating and correctly
executing complex transformations. The work presented in this
paper aims to bypass these limitations by combining LLMs
with dedicated transformation tools.

VII. CONCLUSION AND FUTURE WORK

In this paper, we present our work on the design, develop-
ment, and evaluation of a model transformation agent with tool
filtering capabilities. We provide a working implementation
of this model transformation agent that is dedicated to ATL
model transformations. We then compare it with three other
different transformation agents: 1) a base agent with only
minimal tools, 2) a RAG-based agent with support for tool
retrieval, 3) a transformation agent with dedicated tools for
each transformation but without support for tool filtering. As
part of this effort, we also contribute a reusable generation
process for datasets to evaluate model transformation agents,
together with an actual dataset for ATL model transformation
agents based on a new open-source ATL MCP Server. Thanks
to this, we evaluate the behavior of all these agents on 1,200
instructions covering a significant number and variety of trans-
formations. As a result, we observed that our transformation
agent outperforms the other agents in terms of accuracy to
select the appropriate tools with the right parameters. This is
more particularly the case for non-trivial transformation tasks.
We also demonstrate that this model transformation agent can
scale up to at least 110 tools (corresponding to 55 different
model transformations), and potentially more.

In terms of future work, and to better support the complexity
of some model transformation scenarios, we could extend our
current tool descriptions in different manners. One way to do
that is to provide more detailed explanations of the transfor-
mation logic (i.e., the transformation rules and constraints),
while paying attention not to exceed the context window of
the LLM. In addition, we could also increase the number of
available transformations to further assess the applicability of
our approach in large-scale scenarios. Our current ATL MCP
server, already containing 55 ATL transformations and cover-
ing a diverse range of scenarios, could be enriched with other
different model transformations in this respect. Moreover, we
foresee interesting research aspects to study when combining
our approach with emerging techniques for collaboration and
orchestration involving multiple agents. In this way, we could
complement our current support for model transformation with
other agents specialized in different MBE tasks. For example,
we could consider more complex task chains that integrate
the automatic definition and building of model views [40] or
the use of automated modeling recommendations [41] together
with model transformations.

ACKNOWLEDGMENT

This research work has been funded by the Key Digital
Technologies (KDT) Joint Undertaking through the project
MATISSE, grant agreement No. 101140216.
REFERENCES

[1] N. Kahani, M. Bagherzadeh, J. R. Cordy, J. Dingel, and D. Varró, "Survey and classification of model transformation tools," *Software & Systems Modeling*, vol. 18, no. 4, pp. 2361-2397, 2019.

[2] T. Buchmann, "Prompting Bidirectional Model Transformations-The Good, The Bad and The Ugly," in *Proceedings Of The ACM/IEEE 27th International Conference On Model Driven Engineering Languages And Systems*, 2024, pp. 550-555.

[3] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, "React: Synergizing reasoning and acting in language models, 2023," URL https://arxiv.org/abs/2210.03629, 2023.

[4] C. Qu, S. Dai, X. Wei, H. Cai, S. Wang, D. Yin, J. Xu, and J.-R. Wen, "Tool learning with large language models: A survey," *Frontiers of Computer Science*, vol. 19, no. 8, p. 198343, 2025.

[5] S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried et al., "Webarena: A realistic web environment for building autonomous agents," *arXiv preprint arXiv:2307.13854*, 2023.

[6] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, "Swe-agent: Agent-computer interfaces enable automated software engineering," *Advances in Neural Information Processing Systems*, vol. 37, pp. 50528-50652, 2024.

[7] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, "Self-consistency improves chain of thought reasoning in language models," *arXiv preprint arXiv:2203.11171*, 2022.

[8] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin et al., "A survey on large language model based autonomous agents," *Frontiers of Computer Science*, vol. 18, no. 6, p. 186345, 2024.

[9] J. Di Rocco, D. Di Ruscio, C. Di Sipio, P. T. Nguyen, and R. Rubei, "On the use of large language models in model-driven engineering," *Software and Systems Modeling*, pp. 1-26, 2025.

[10] Bucaioni, Alessio and Eramo, Romina and Berardinelli, Luca and Bruneliere, Hugo and Combemale, Benoit and Khelladi, Djamel Eddine and Muttillo, Vittoriano and Sadovykh, Andrey and Wimmer, Manuel, "Multi-partner project: A model-driven engineering framework for federated digital twins of industrial systems (matisse)," in *2025 Design, Automation & Test in Europe Conference (DATE)*, 2025, pp. 1-6.

[11] F. Jouault and I. Kurtev, "Transforming models with ATL," in *International Conference on Model Driven Engineering Languages and Systems*. Springer, 2005, pp. 128-138.

[12] Anthropic, "Model context protocol," 2024. [Online]. Available: https://www.anthropic.com/news/model-context-protocol

[13] I. Milev, M. Balunović, M. Baader, and M. Vechev, "ToolFuzz-Automated Agent Tool Testing," *arXiv preprint arXiv:2503.04479*, 2025.

[14] U. Diagrams, "Activity Diagrams," 2007. [Online]. Available: https://www.ulm-diagrams.org/activity-diagrams.html

[15] LangGraph, "How to use the pre-built ReAct agent," 2024. [Online]. Available: https://langchain-ai.github.io/langgraph/how-tos/create-react-agent/

[16] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., "Chain-of-thought prompting elicits reasoning in large language models," *Advances in neural information processing systems*, vol. 35, pp. 24824-24837, 2022.

[17] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language models are few-shot learners," *Advances in neural information processing systems*, vol. 33, pp. 1877-1901, 2020.

[18] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al., "Judging llm-as-a-judge with mt-bench and chatbot arena," *Advances in neural information processing systems*, vol. 36, pp. 46595-46623, 2023.

[19] J. Kienzle, G. Mussbacher, B. Combemale, L. Bastin, N. Bencomo, J.-M. Bruel, C. Becker, S. Betz, R. Chitchyan, B. H. Cheng et al., "Toward model-driven sustainability evaluation," *Communications of the ACM*, vol. 63, no. 3, pp. 80-91, 2020.

[20] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian et al., "Toolllm: Facilitating large language models to master 16000+ real-world apis," *arXiv preprint arXiv:2307.16789*, 2023.

[21] "Code artifact archive," 2025. [Online]. Available: https://zenodo.org/records/16418428

[22] "ATL zoo transformation examples." [Online]. Available: https://eclipse.dev/atl/atltransformations/

[23] C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang, "Wizardlm: Empowering large language models to follow complex instructions," *arXiv preprint arXiv:2304.12244*, 2023.

[24] L. Long, R. Wang, R. Xiao, J. Zhao, X. Ding, G. Chen, and H. Wang, "On llms-driven synthetic data generation, curation, and evaluation: A survey," *arXiv preprint arXiv:2406.15126*, 2024.

[25] X. Guo and Y. Chen, "Generative ai for synthetic data generation: Methods, challenges and the future," *arXiv preprint arXiv:2403.04190*, 2024.

[26] T. Gan and Q. Sun, "Rag-mcp: Mitigating prompt bloat in llm tool selection via retrieval-augmented generation," *arXiv preprint arXiv:2505.03275*, 2025.

[27] LangChain, "Benchmarking Single Agent Performance," 2025. [Online]. Available: https://blog.langchain.dev/react-agent-benchmarking/

[28] K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin, "Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges," *arXiv preprint arXiv:2401.07339*, 2024.

[29] C. Qian, X. Cong, C. Yang, W. Chen, Y. Su, J. Xu, Z. Liu, and M. Sun, "Communicative agents for software development," *arXiv preprint arXiv:2307.07924*, vol. 6, no. 3, 2023.

[30] S. Yuan, K. Song, J. Chen, X. Tan, Y. Shen, R. Kan, D. Li, and D. Yang, "Easytool: Enhancing llm-based agents with concise tool instruction," *arXiv preprint arXiv:2401.06201*, 2024.

[31] C. Qu, S. Dai, X. Wei, H. Cai, S. Wang, D. Yin, J. Xu, and J.-R. Wen, "Towards completeness-oriented tool retrieval for large language models," in *Proceedings of the 33rd ACM International Conference on Information and Knowledge Management*, 2024, pp. 1930-1940.

[32] Y. Kong, J. Ruan, Y. Chen, B. Zhang, T. Bao, S. Shi, G. Du, X. Hu, H. Mao, Z. Li et al., "Tptu-v2: Boosting task planning and tool usage of large language model-based agents in real-world systems," *arXiv preprint arXiv:2311.11315*, 2023.

[33] J. Ruan, Y. Chen, B. Zhang, Z. Xu, T. Bao, H. Mao, Z. Li, X. Zeng, R. Zhao et al., "Tptu: Task planning and tool usage of large language model-based ai agents," in *NeurIPS 2023 Foundation Models for Decision Making Workshop*, 2023.

[34] Z. Chen, W. Du, W. Zhang, K. Liu, J. Liu, M. Zheng, J. Zhuo, S. Zhang, D. Lin, K. Chen et al., "T-eval: Evaluating the tool utilization capability of large language models step by step," *arXiv preprint arXiv:2312.14033*, 2023.

[35] T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, "Toolformer: Language models can teach themselves to use tools," *Advances in Neural Information Processing Systems*, vol. 36, pp. 68539-68551, 2023.

[36] A. Ye, Q. Ma, J. Chen, M. Li, T. Li, F. Liu, S. Mai, M. Lu, H. Bao, and Y. You, "SOP-Agent: Empower General Purpose AI Agent with Domain-Specific SOPs," *arXiv preprint arXiv:2501.09316*, 2025.

[37] L. A. Sarr, P. K. Ayite, A.-M. Barthe-Delanoë, D. Bork, G. Macé-Raméte, and F. Benaben, "Towards the Integration of Conversational Agents Through a Social Media Platform to Enhance the Agility of BPM," in *Working Conference on Virtual Enterprises*. Springer, 2024, pp. 36-48.

[38] J. Cámara, J. Troya, L. Burgueño, and A. Vallecillo, "On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML," *Software and Systems Modeling*, vol. 22, no. 3, pp. 781-793, 2023.

[39] S. Abukhalaf, M. Hamdaqa, and F. Khomh, "PathOCL: Path-based prompt augmentation for OCL generation with GPT-4," in *Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering*, 2024, pp. 108-118.

[40] J. W. Pontes Miranda, H. Bruneliere, M. Tisi, and G. Sunyé, "Towards an In-Context LLM-Based Approach for Automating the Definition of Model Views," in *Proceedings of the 17th ACM SIGPLAN International Conference on Software Language Engineering*, 2024, pp. 29-42.

[41] M. Weyssow, H. Sahraoui, and E. Syriani, "Recommending metamodel concepts during modeling activities with pre-trained language models," *Software and Systems Modeling*, vol. 21, no. 3, pp. 1071-1089, 2022.