DOI: 10.5281/zenodo.18488802
title: UML-to-Java Model Transformation with Large Language Models

description: This repository contains the replication package for the research work:

An empirical evaluation of Large Language Models (LLMs) for UML-to-Java model transformation in a zero-shot setting.

The study investigates whether modern Large Language Models can serve as out-of-the-box model transformers, translating UML class diagrams serialized in XMI/Ecore into structurally correct Java code, without explicit transformation rules, in-context examples, or interactive correction.

Building on the experimental framework originally proposed by Kazai et al., this work extends the evaluation in scale, rigor, and analysis depth, using:





116 UML class diagram models,




three state-of-the-art LLMs (GPT-5.2, DeepSeek v3, Gemini 2.5),




a fixed zero-shot prompt, and




a two-iteration regeneration protocol.



The repository implements the entire experimental pipeline, including oracle generation, LLM execution, automated comparison, error filtering, and result aggregation.

Research Context and Goal

Model transformation is a cornerstone of Model-Driven Engineering (MDE), but traditional rule-based transformations (e.g., ATL, QVT) introduce significant accidental complexity and maintenance overhead.

This research evaluates whether LLMs can:





lower the entry barrier to MDE,




act as direct UML-to-Java transformers, and




preserve structural fidelity across increasing model complexity.



The central research question addressed by this repository is:

To what extent can pretrained LLMs perform UML-to-Java model transformation correctly and consistently, without explicit transformation rules, and how does this capability degrade with model complexity?

Experimental Design (Implemented in This Repository)

The pipeline follows a constructive and experimental research design, consisting of the following phases:





Dataset collection and screening




Oracle Java generation




Zero-shot LLM-based transformation




Automated comparison and manual filtering




Iterative regeneration (maximum two iterations)




Result aggregation and error analysis



Iteration Policy (Important)

Although the original pipeline supported up to three iterations, empirical analysis showed that:

The third iteration produced no additional error-free transformations.

Therefore, this repository intentionally supports only two iterations, which:





capture all observed improvements,




avoid artificial inflation of results, and




keep the evaluation controlled and comparable.



All scripts, folder structures, and result reporting reflect this two-iteration design choice.

Prerequisites

The pipeline was executed and validated on Windows, using the following tools:





Windows OS




AutoHotKey (v1.1 and v2.0)




Beyond Compare




Visual Studio Code





Language Support for Java™ by Red Hat (v1.28.1)






Python




Google Chrome




ChatGPT (web interface)


DeepSeek (web interface)




Gemini (web interface)


The use of AutoHotKey requires that LLMs be accessed via a browser and that user interaction not be interrupted during execution.

Installation and Configuration

Java Formatting Configuration

Enable automatic Java formatting in VS Code:











 








 





Settings → Command Palette → Preferences: Configure Language Specific Settings

→ Java → Text Editor → Formatting → Format On Save










Path Configuration

You must specify local executable paths in the following files:

VS Code path





formatJava.ahk (lines 4, 11)




formatJava_SecondIteration.ahk (line 3)



Python path





firstIteration.bat




secondIteration.bat




specificExecution.bat



(Line numbers are documented inside each file.)

Beyond Compare

Place a shortcut to Beyond Compare in the repository root directory.

Prompt Used for LLM Transformation

All transformations use the same fixed zero-shot prompt:



















The following is an XMI file based on a UML diagram.

Please write corresponding Java code without adding any explanatory text.

Do not implement getters/setters and do not add comments.

Generate classes as if they were in separate files.

Start and end the response with [[STARTEND]] as a Java comment.










No examples, hints, or oracle excerpts are provided.

Execution

Demo Run

To verify correct installation, run:



















specificExecution.bat










 

and specify:



















demo










 

After execution, results are stored in:



















./allXMI/works/demo/










 

Folder contents:





oracle/ – Java reference generated via Modelio or ATL+Python




gpt/ – XMI input and LLM-generated Java output




comparison/ – Beyond Compare reports and error logs



Iteration Workflow (Two Iterations Only)

First Iteration





Executed via firstIteration.bat




Produces initial LLM output




Compares against oracle




Classifies results as error-free or erroneous



Second Iteration





Executed via secondIteration.bat




Re-runs only failed cases




Uses the same prompt (no corrections injected)




Produces final cumulative results



There is no third iteration in this repository, by design.

Repository Structure



allXMI/



└── works/

        └── done/

                └── "UML model_s_name"/

                         ├── 2nd ite/ (if needed)

                         ├── result/

                         ├── oracle/

                         └── comparison/

 

 



Key subfolders for each model:





oracle/ – Reference Java code generated by Modelio or ATL+Python




result/ – LLM outputs




comparison/ – Comparison reports and error logs



Evaluation Method

Evaluation combines automated tooling and manual inspection, following established practice:





Modelio / ATL+Python → oracle generation




Python scripts → file normalization




AutoHotKey → automation




Beyond Compare → structural differencing




Manual filtering → removal of cosmetic differences



A transformation is classified as error-free only if no structural discrepancies remain after filtering.

Replication Notes





The dataset, prompts, and iteration logic are fixed




Results are reproducible if the file structure is preserved




For the correct setup, refer to the demo folder as a template



Built With





AutoHotKey




Beyond Compare




Modelio 4.1




Python




Visual Studio Code




Java Language Support (Red Hat)




ChatGPT (web interface)


Gemini (web interface)




DeepSeek (web interface)