DOI: 10.5281/zenodo.17799431
title: Replication Package for the Paper: Evaluating Large Language Models for Detecting Architectural Decision Violations

description: This replication package provides how we evaluate LLMs for detecting ADR Violations. The packge leverages the dataset of ADR files from Buchgeher et al. (ORIGINAL_README.md) to prepare the project's codebase and selected ADR files for the analysis with Large Language Models, and the LLM Detection Process.

Note: All the hyperlinks referring to the shared files only work in the local version (downloading the relpication package). The only version will not work as the browser does not find the files.

License

All generated data is provided under DATA_LICENSE Creative Commons 4.0 Attribution License.

All scripts are provided under the Script_LICENSE MIT License.

Contents

This repository consists of the following files:





README.md: The README file describes the project and how to run the scripts to get the data and do the analysis




INSTALL.md: Installation and configurations instructions




requirements.txt: Library requirements for the Python environment




Figures





jaccard_C_heatmap.pdf: Heatmap diagram of pairwise jaccard coefficients (C)




jaccard_overall_heatmap.pdf: Heatmap diagram of pairwise jaccard coefficients (overall)




llm_process.pdf: Diagram of the general LLM Processing Implementation






Data_collection





ADR-dataset: The original dataset files (renamed from repositories)




.env: Environment variables for the bash environment




ORIGINAL_README.md: The original README file of the dataset repository




adr_all.csv: Data on all ADR files from the original dataset reported in a single CSV




adr_selected_projects.csv: Data on ADR files only from projects selected based on TLoC and the number of commits




adr_template.csv: Data on ADR files from the selected projects for which the template was identified in the original dataset




project_stats.csv: Data on TLoC and number of commits of each project






Scripts





01_data_collection





01_clone_repos.sh: Script to clone locally the repositories from the dataset




02_create_codebase.sh: Script to merge source code files from repositories into a single JSON file




03_project_stats.sh: Script to analyze TLoC and the amount of commits of the repositories




04_list_adr_files.py: Script to process the dataset of ADR files to list all files, select ADR files from projects based on TLoC and commits, as well as select ADR files based on the parsed template




05_create_adr.sh: Script to move the selected ADR files to the projects' codebases for analysis with LLMs






02_llms





adr_judge.py: Script to using LRM to detect the ADR violations




adr_validate.py: Script to using 3 LLMs to validate the ADR judgement results from LRM




sbatch.sh: Script to submit the work to the supercomputer






03_data_analysis





rq1analysis.py: Script to calculate the overall agreement (Fleiss' Kappa, Agreement by category, and pairwise Jaccard coefficients)




rq2analysis.py: Script to calculate the sklearn metrics








Results





llms





judge_marco.csv: Raw results generated from LRM (Marco-o1)




judge_marco_statuscorrection.csv: Results through the status filtering




status_category.csv: Status extraction and mapping




validate_mistral.csv: Raw results (validation) generated from Mistral Nemo




validate_llama.csv: Raw results (validation) generated from Llama




validate_qwen.csv: Raw results (validation) generated from Qwen




human_validation.csv: Human validation process for the sample size results






RQ1





agreement_analysis.csv: Input data sent to the rq1analysis.py to calculate the overall agreement




rq1data.csv: Generated data (Overall Inter-LLMs Agreement and Agreement by Label with Class Frequency)






RQ2





performance_analysis.csv: Input data sent to the rq2analysis.py to calculate the sklearn metrics




rq2data.csv: Generated data (table that present all the needed sklearn metrics)






RQ3





rq3.1data.csv: Results for the ADR category (limitation analysis)




rq3.2data.csv: Results for the error labels generated by LLMs (error label analysis)








LLM_Input



Abelaa--pcmt: One repo example as the LLM input




Replication of the results

This section describes the steps necessary to replicate the project, ADR selection, and ADR violation detection using LLMs steps from our work.

Setting up the environment

Follow the instructions in INSTALL.md to set up the environment and define environment variables.

Cloning the repositories

Use the script 01_clone_repos.sh to attempt to clone all repositories from the dataset. The scripts clones all repositories to the directory given by the $REPO_DIR environment variable. Cloning is performed over https, and the repository is skipped if the URL is not found (project has been deleted) or if authentication is requested (project has been switched to private).

For each repository, the owner and the project name are concatenated using -- to create the unique string identifying the project to be used as the directory, i.e. for the project URL https://github.com/OWNER/REPO the directory will be OWNER--REPO. This convention is used all throughout this repository and the scripts.

Creating the codebase files

Use the script 02_create_codebase.sh to merge the source code of the cloned projects into JSON files. The script goes through all the projects in the $REPO_DIR, identifies all the source code files by common extensions, checks if the files are text, non-empty and in UTF-8 encoding, and merges them into a single JSON file. The files are written to the codebase directory given by $CODEBASE_DIR environment variable for each project.

Analyzing project statistics

Use the script 03_project_stats.sh to compute the TLoC and the number of commits of each project. The script count the number of code lines from codebase files found in $CODEBASE_DIR and uses git to check the number of commits in $REPO_DIR. The data is saved to the CSV file given in $STATS_FILE environment variable.

Selecting the ADR files

Use the script 04_list_adr_files.py to parse the original dataset and select the ADR files.

The script performs the following:



Iterate over all files from the original ADR_dataset and write information on each repository owner, name and URL, ADR file, its parsed template and acceptance status into the CSV file adr_all.csv

Select the cloned projects based on the TLoC (4th quartile) and number of commits (2nd quartile and above) given in project_stats.csv, and save data about only their ADR files to the CSV file adr_selected_projects.csv

Further select only the ADR files that have their template parsed and save them to the CSV file adr_template.csv


The file adr_template.csv will thus contain all ADR files selected for analysis from the original dataset.

Adding ADR files to the codebase

Use the script 05_create_adr.sh to add the selected ADR to the codebases.

The script takes all ADR files given in adr_template.csv, attempts to find them in the corresponding cloned repository in $REPO_DIR and copy to the corresponding codebase in $CODEBASE_DIR. If a file is not found it is skipped.

Final data selection structure

After all scripts are successfully executed, the $CODEBASE_DIR has the following structure:



$CODEBASE_DIR/



OWNER1--REPO1/



OWNER1--REPO1--codebase.json

ADR1.md

ADR2.md

...



OWNER2--REPO2/



OWNER2--REPO2--codebase.json

ADR1.md

ADR2.md

...



.../




Files in this structure can then be given to LLMs.

We provide one example (one repo) in the LLM_Input folder, Abelaa--pcmt, which shows the above structure, and can be used to test the script as the input.

LRM Running (supercomputer)

After we get the above data with the described structure, we use it as the input for running LRM (Marco-o1).

We use adr_judge.py and sbatch.sh, submit the work to the supercomputer, with the following command structure:

sbatch sbatch.sh "MODEL_NAME" PORT OUTPUT_FILE


Then we get the results judge_marco.csv. Then we analyze the status situation of each project, try to map the status and remove the projects whose status is "rejected". Finally, we get the results judge_marco_statuscorrection.csv. Now the LRM Running process is completed.

Three LLMs Validation Running (supercomputer)

After we get the results from LRM, we continue using 3 LLMs to validate the results. We use adr_validate.py and sbatch.sh, submit the work to the supercomputer with the same command structure.

Then we get the validation results from 3 LLMs: validate_mistral.csv, validate_llama.csv, and validate_qwen.csv.

Non-supercomputer (Local Demo, No SLURM)

This section is intended for that do not have access to the supercomputer. It runs a minimal demo on a single machine.

Demo input data

LLM_Input folder, Abelaa--pcmt

The demo data includes:



A small set of ADRs (e.g., adr-001.md ... adr-007.md)

One codebase snapshot in JSON format (e.g., Abelaa--pcmt--codebase.json)


Start an OpenAI-compatible LLM endpoint (vLLM)

This project expects an OpenAI-compatible /v1/chat/completions endpoint.

Example: start a local vLLM server (single GPU)

python -m vllm.entrypoints.openai.api_server \
  --model AIDC-AI/Marco-o1 \
  --port 8000 \
  --tensor-parallel-size 1


Sanity check:

curl http://localhost:8000/v1/models 


Run LRM (Marco-o1) locally

Once the vLLM server is running, you can run the LRM script using:

python Scripts/02_llms/adr_judge.py \
  --model_name "AIDC-AI/Marco-o1" \
  --api_url "http://localhost:8000/v1/chat/completions" \
  --output_file Results/llms/judge_marco_demo.csv


Expected output:





A CSV file containing one or more rows per extracted rule.




The output should include at least the fields: Rule, Judgement, and Reason.



Notes on multi-model setup (4 models)

The paper evaluates 4 models:





LRM (Marco-o1)




3 validator models (Mistral / Qwen / Llama)



vLLM serves one model per server instance.Therefore, multi-model validation can be done in two ways:





Sequential mode (recommended for demo): stop vLLM and restart it with another model.




Multi-port mode: start 4 vLLM servers on different ports.



The HPC scripts use the same principle (one model per server job).

Human Validation and Data Analysis (RQs)

We selected the sample size for human validation.

Then we do the data analysis based on the defined RQs.

All the needed script, data, and results are shown in the "Scripts - data analysis" and "Results - RQ1/RQ2/RQ3" folders.