DOI: 10.5281/zenodo.18628435
title: Replication Package: "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation"

description: Replication Package: "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation"



Journal: Science of Computer Programming

Year: 2026

Authors: Eneko Pizarro, Maider Azanza, and Beatriz Perez


This repository contains the replication package for the paper "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation".

Repository structure

1) Test suite (resources under test)

These are the classes and functions used for the analysis and metric collection in the paper. The table below lists each tested item, its origin, and the evaluation focus.




Function/Method
Function Type
Evaluation Focus
Source


assemble
Rental Assembler
Complex object interactions, database mocking, multi-layered architecture


https://github.com/BSPQ18-19/BSPQ19-S1.git



isPrime
Prime Checker
Algorithmic logic, exception handling, high cyclomatic complexity, multiple error paths
Software Engineering II (course), EHU/UPV


addUser
User Management
Input validation, null handling, database mocking, CRUD operations
https://github.com/jononekin/Forum-assertJ.git


getBonus
Bonus Calculator
Nested conditional business rules, external dependencies
https://github.com/jononekin/Forum-assertJ.git


isIPV4Valid
IPV4 Validator
Utility function integration, helper function interaction
https://www.geeksforgeeks.org/program-to-validate-an-ip-address/


isStrobogrammic
Strobogrammatic Number
Data structure manipulation (map/dictionary), low leakage risk
https://www.w3resource.com/java-exercises/basic/java-basic-exercise-186.php


palindrome
Palindrome Detector
Multi-function integration, coordinated test generation
https://code-exercises.com/programming/hard/4/longest-palindrome-in-word


isLeapYear
Leap Year Checker
Boundary conditions, multiple branching paths
Software Engineering II (course), EHU/UPV




 

2) Aggregated results

Folder: Aggregated_Results_by_Function_Oct.2025

Comprehensive results for all methods and LLMs tested. Data were collected between March 2024 and October 2025. The results include the metrics described in the evaluation framework section of the paper.

Page index:



Weights: Metric descriptions, calculation and evaluation methods, and aggregation weights.

Journal-Summary Published: Aggregated results as reported in the paper.

Comparison: GitHub test vs Custom Agent: Aggregated results comparing test generation methods (GitHub Copilot `/test` vs Custom Agent prompt).


3) Prompt artifacts



Prompt_for_Integration_Testing-2025: Final prompt used to generate the test cases.

Custom_Agent_Prompt_for_Integration_Testing.md: Custom Agent personalization structured prompt used in the measurement method.


4) Sources



sources.7z: Source code classes and functions from which the test classes shown in the paper were generated.


5) Expert test cases

Folder: Expert_Test_Cases

Curated, human-authored test suites used as reference baselines for the evaluation.

6) Evaluation

Folder: Evaluation

Evaluation artifacts including:



Evaluation_Consent_Form.pdf: Participant consent form used in the evaluation study.

Evaluation_Information_Sheet.pdf`: Information sheet provided to evaluation participants.


7) Results and measurements

Folder: Results_and_Measurements

Raw and processed outputs from all measurement campaigns, organized by collection window and target function.

8) Related project

Tetrics PoC: https://github.com/lksnext-ai-lab/tetrics.git

Tetrics is a research prototype implementing a domain-independent, continuous evaluation framework for LLM-powered development tools.