DOI: 10.5281/zenodo.15703811
title: Benchmarking LLM for MTBE

description: FILES OVERVIEW:

The experiment involves multiple transformation tasks and configurations across five LLMs. All input/output files follow a structured naming convention:

[filename]_[model]_[n]_examples.[filetype]These files contain the output model generated by each LLM under a given scenario and number of example pairs.

[filename].[filetype]These files are the example model to be input.

mappingrules[inputType][outputType].texThese files store the mapping rules written in abstract machine-readable xmi format.

[inputType/outputType]metamodel.texThese files contain metamodels for either desired input or output models.








Filename Example
Description




VehicleDefinitions_gpt_2_examples
Output by GPT-4.5 for SysML-to-AAS with 2 example pairs


Port Examples.aas
An AAS model used as part of an example pair for SysML-to-AAS


mappingrulesSysMLAAS.tex
Mapping rules for scenario SysML-to-AAS


AASmetamodel.tex
AAS metamodel used for scenario SysML-to-AAS






 




KEY POINTERS WHEN RE-DOING THE EXPERIMENTS:





Prompt Composition: Prompts must follow the format:Prompt = Mapping Rules (M) + n Example Pairs (En) + Metamodels (MM)




Semantic Completeness: Ignore small formatting issues. Prioritize semantic and structural correctness.




Model-Specific Behavior: Be aware that LLMs like GPT-4.5 are sensitive to prompt length and example order.




Manual Review: Since some outputs include hallucinations or missing fields, results must be manually reviewed for meaningful validation.



STEPS TO REPLICATE THE PROCESS:

1. Prompt Generation

Prepare prompt for each scenario (RDBMS-to-UML, UML-to-Java, SysML-to-AAS) and configuration (1pairs, 2pairs, 4pairs), each including:





Textual mapping rules (M)




Example pairs (E)




Metamodel descriptions (MM)




New source model for transformation



You should have one prompt per configuration per scenario.

2. Running LLMs

For each of the five tested LLMs:





Copy the entire prompt text into the LLM’s web interface (ChatGPT, Claude, Gemini, etc.).




Paste manually—do not modify the formatting or insert extra instructions.




Specify the task in the system prompt if available, e.g.,"You are a transformation assistant that learns from example models and generates new target models."



Models Tested:





ChatGPT-4.5




DeepSeek V3




DeepHermes 3 LLaMA 3 8B




QwQ 32B




OlympicCoder 32B



Repeat the process for:





3 scenarios (RDBMS-to-UML, UML-to-Java, SysML-to-AAS) 




3 configurations (1, 2, 4 example pairs) = 9 prompts per model



3. Save the Output





After the model responds, copy the generated model into a .[outputType] file named using the format:





[filename]_[model]_[n]pairs.[outputType]






Make no edits to the output at this point—preserve the raw generation.



3. Manual Evaluation





Compare LLM-generated models against ground-truth.




Use two validation metrics:





% Correct: Cr / (LOC + Miss)




% Weighted Success: (Cr − InCr + Ad − 2×Miss) / LOC






Use Excel/Google Sheets for structured tracking:





Columns: Model, Scenario, #Pairs, LOC, Cr, InCr, Ad, Miss, %Correct, %WSuccess





4. Repeat for All Models and Configurations

Total combinations to run and evaluate:





3 Scenarios × 3 Configs × 5 LLMs = 45 runs



Each output must be individually reviewed and scored.