DOI: 10.5281/zenodo.17343408
title: Benchmarking LLM over roundtrip evolution of Ecore and SQL

description: FILES OVERVIEW

This repository evaluates round-trip Model Transformation-by-Example (MTBE) between Ecore (S) and SQL-XMI (T). Each experiment starts from a paired model (S, T), applies a single operator delta Δ to one side, then performs:





Inverse step: Δ(T) → S′




Forward step: S′ → T′



We validate by comparing S′ vs Δ(S) and T′ vs Δ(T).



KEY POINTERS WHEN RE-DOING THE EXPERIMENTS

Prompt Composition (same formula):Prompt = Example Pair(s) (E) +  Δ-variant input (IM) + Stage Task

Semantic Completeness: Ignore minor formatting/whitespace. Prioritize structural and referential correctness (classes/tables, attributes/columns, types, multiplicities/keys, containment vs reference, FKs).

Model-Specific Behavior: LLMs are sensitive to prompt length and example order. Keep the same example pair(s) across both stages of a run.

Manual Review: Outputs may contain hallucinations or omissions. Always perform loadability and semantic checks before scoring.



STEPS TO REPLICATE THE PROCESS

1. Prompt Generation

Prepare two prompts per (pairId, op, k):

A) Inverse (Δ(T) → S′)Include (in this order):





Example Pair(s) (Eₙ) — S_[pairId].ecore + T_[pairId].sql.xmi (k = 1, 2, or 4)




Input Artifact — Delta_[op]_T_[pairId].sql.xmi




Task - “Provide only the Ecore/XMI serialization of S’, metamodel-conformant. If ∆(T ) contains elements not covered by the example mapping, briefly flag them in XML comments, but still produce a valid S’.”



B) Forward (S′ → T′)Same structure, but the Input Artifact is the S′ you just produced.Task - “Generate only SQL-XMI for T′ (no prose/comments). Ensure metamodel conformance and Δ-awareness.”



Keep E fixed across both stages to isolate round-trip effects.


2. Running LLMs

For each of the four tested LLMs:





GPT-5




DeepSeek V3




Qwen-3-Next-80B-A3B




Gemini 2.5



Execution tips:





Optional system prompt:“You are a transformation assistant that learns from example models and produces structurally valid models only.”



3. Manual Evaluation

Compare:





Inverse: *_inverse_*.ecore (S′) vs Delta_[op]_S_[pairId].ecore (Δ(S))




Forward: *_forward_*_sql.xmi (T′) vs Delta_[op]_T_[pairId]_sql.xmi (Δ(T))



Record per stage in a sheet with columns:

LOC, Cr, InCr, Ad, Miss, %Correct, %WSuccess, DeltaDetected, Loadable

Metrics (same style ):





% Correct = Cr / (LOC + Miss − Ad)




% Weighted Success = (Cr−InCr−Miss−Ad[harmful]−2[∆ undetected] / LOC



Delta checks:





S′ ≡ Δ(S) and T′ ≡ Δ(T) (after canonical whitespace/ID normalization)




Semantic/structural equivalence (entities, properties, types, multiplicities, FKs, containment vs reference), even if IDs/order differ from the original files.



Loadability sanity:





Ensure XML namespaces (xmi:version, xmlns:*, nsURI) match the metamodel declarations.



4. Repeat for All Combinations

Let:





B = number of base pairs (here 10)




D = number of delta operators (here 10)




L = number of LLMs (here 4)



Then you’ll have:





Runs: B × D × L, each with two saved outputs (inverse + forward)




Stage evaluations: 2 × B × D × L (here 800)