Incremental Java Unit Test Generation via Large Language Models Based on Pull Request Changes (Replication Package
Creators: name: Çakar, Berk
name: Akar, Atakan
name: Tüzün, Eray
Publisher: name: figshare
Year: 2024
Subjects: classificationCode: 461201
subject: Automated software engineering
classificationCode: 461202
subject: Empirical software engineering
classificationCode: 461207
subject: Software quality, processes and metrics

description: <b>Automated Unit Test Generation via LLMs based on Pull Request Changes (Replication Package)</b><br><br>This repository contains the replication package for the paper “Automated Unit Test Generation via LLMs based on Pull Request Changes”.<br><br><b>Included Files:</b><br><b>Source code:</b><br><br>This folder contains the Python code files used in the project. These scripts likely include the main logic for processing data, generating results, and conducting the experiments.<br><br><b>Contents of this file:</b><b>requirements.txt</b><b>:</b> Contains a list of Python libraries and their versions that are required to run the project.<b>env.py</b><b>:</b> File stores essential configuration variables like API keys, endpoints, and deployment details required for the project's integration with services such as OpenAI, Azure, and GitHub, and the placeholders (“” and ‘’) must be filled in with your own information.<b>fetch_prs.py</b><b>:</b> Fetches the pull requests and analyzes GitHub pull requests with features to filter only merged and non-bot pull requests and generates structured JSON data.<b>attach_diffs_to_prs.py</b><b>: </b>Calculates the diff between the code before and after the pull request and appends to a new file for further analysis.<b>generate_tests.py</b><b>:</b> It creates prompts for the LLMs using a `new_prompt.md` template, incorporating information like differences. The generated test content is appended to the file called `prs_with_generated_tests`.<b>new_prompt.md</b><b>:</b> This file is a prompt template that guides the LLM in generating unit tests based on pull request changes. It organizes inputs like old code version, diff file, and existing test cases if available.<b>validate_and_repair_test.py</b><b>:</b> Repairs small errors in generated test files, like missing curly braces or misplaced imports. It uses predefined heuristics to make minimal changes. The repaired test content is updated in the JSON data and saved to the `prs_with_repaired_tests` directory.<b>run_tests.py</b><b>:</b> Automates testing for pull requests by running both tests that are written by developers and LLMs separately. The results, including success status, logs, and code coverage data, are saved in JSON files under the `prs_with_test_results` directory.<b>analyze_test_results.py</b><b>:</b> Evaluates the effectiveness of developer-written and LLM-generated unit tests for Java projects by analyzing test results and calculating metrics like test coverage improvements. It compares the coverage changes introduced by each type of test and identifies unique contributions from generated tests. The findings are summarized and stored in analysis files for further review.<b>plot.py</b><b>:</b> Generates plots, such as pie, bar, scatter, and box plots, for visualizing the test coverage increases and build success rates across projects, saving results in a directory called `plots`.<b>util.py</b><b>:</b> A file that contains utility functions that assist in analyzing Java files and code diffs. It includes helper functions to be used such as calculating the number of the changed and counting occurrences of specific patterns in a string.<b>models.py</b><b>:</b> This code defines several Python classes (models) designed for managing and analyzing pull request changes and test results. The `FileDiffAssociation` model represents file changes in a pull request, linking them to associated test files and their contents, while `TestResult` is a model for storing detailed test execution outcomes, including coverage metrics.<br><b>Dataset:</b><b>repos:</b> The "repos” directory contains the cloned GitHub repositories of the projects where the experiments are applied.<b>prs_with_test_results: </b>This folder contains unit tests generated by different models for various pull requests along with detailed data and logs of their execution and evaluation outcomes.<br><b>Results:</b>The dataset includes results organized into three main categories for each of the four large language models (Gemini 1.0 Pro, Gemini 1.5 Flash, GPT-4, and GPT-4o). These categories represent different aspects of the evaluation process.<br><b>Model-Specific Analysis:</b>These files evaluate the performance of individual models in generating unit tests, focusing on metrics like the number of generated tests, build success rates, and coverage improvements.<b>Files:</b>analysis_results_gemini-1.0-pro.csvanalysis_results_gemini-1.5-flash.csvanalysis_results_gpt-4.csvanalysis_results_gpt-4o.csv<br><b>Column Names and Descriptions:</b><br><b>no_of_prs</b><b>: </b>Total number of pull requests (PRs) analyzed for this model.<b>no_of_initial_tests</b><b>:</b> Number of initial tests present before the PR changes were introduced.<b>no_of_generated_tests</b><b>:</b> Number of unit tests generated by Gemini 1.0 Pro.<b>no_of_test_classes_created</b><b>:</b> Number of new test classes created as a result of test generation.<b>no_of_test_classes_expanded</b><b>:</b> Number of existing test classes expanded with new test cases.<b>no_of_unique_tests_generated</b><b>:</b> Count of unique test cases generated, avoiding duplication of functionality.<b>no_of_test_classes_failed_to_generate</b><b>:</b> Number of test classes that the model attempted but failed to generate.<b>no_of_test_classes_generated</b><b>:</b> Total number of test classes successfully generated.<b>no_of_successful_builds</b><b>: </b>Number of builds where the generated tests compiled and executed successfully.<b>no_of_failed_builds</b><b>:</b> Number of builds that failed due to issues in the generated tests.<b>no_of_initial_tests_run</b><b>:</b> Count of initial tests run before introducing the generated tests.<b>no_of_initial_tests_failed</b><b>:</b> Number of initial tests that failed during execution.<b>no_of_initial_tests_error</b><b>:</b> Number of initial tests that encountered runtime errors.<b>no_of_initial_tests_skipped</b><b>:</b> Number of initial tests that were skipped during execution.<b>no_of_generated_tests_run</b><b>:</b> Number of generated tests executed during the evaluation process.<b>no_of_generated_tests_failed</b><b>:</b> Number of generated tests that failed due to assertion issues.<b>no_of_generated_tests_error</b><b>: </b>Number of generated tests that encountered runtime errors during execution.<b>no_of_generated_tests_skipped</b><b>:</b> Number of generated tests skipped during execution.<b>no_of_experiments_done_files</b><b>: </b>Total number of file-level experiments performed (each file change in a PR counts as one experiment).<b>no_of_experiments_done</b><b>:</b> Total number of experiments conducted at the PR level.<b>no_of_experiments_failed</b><b>:</b> Number of experiments that failed due to issues with the model-generated outputs.<b>no_of_prs_instruction_coverage_increase</b><b>:</b> Number of PRs where instruction coverage increased due to generated tests.<b>no_of_prs_branch_coverage_increase</b><b>:</b> Number of PRs where branch coverage increased due to generated tests.<b>no_of_prs_line_coverage_increase</b><b>:</b> Number of PRs where line coverage increased due to generated tests.no_of_prs_complexity_coverage_increase: Number of PRs where complexity coverage increased due to generated tests.<b>no_of_prs_method_coverage_increase</b><b>:</b> Number of PRs where method coverage increased due to generated tests.<b>average_change_in_instruction_coverage</b><b>:</b> Average improvement in instruction coverage across all analyzed PRs.<b>average_change_in_branch_coverage</b><b>: </b>Average improvement in branch coverage across all analyzed PRs.<b>average_change_in_line_coverage</b><b>:</b> Average improvement in line coverage across all analyzed PRs.<b>average_change_in_complexity_coverage</b><b>:</b> Average improvement in complexity coverage across all analyzed PRs.<b>average_change_in_method_coverage</b><b>:</b> Average improvement in method coverage across all analyzed PRs.<b>project_name</b><b>:</b> Name of the project from which the PRs were analyzed.<br><br><b>Human-Written Test Comparisons:</b>This category compares the effectiveness of human-written tests with the ones that are generated by the LLMs.<b>Files:</b>analysis_results_human_written_tests_gemini-1.0-pro.csvanalysis_results_human_written_tests_gemini-1.5-flash.csvanalysis_results_human_written_tests_gpt-4.csvanalysis_results_human_written_tests_gpt-4o.csv<br><b>Column Names and Descriptions:</b><br><b>human_written_tests_found_in_prs:</b> Count of the pull requests in which the human-written tests were found.<b>human_written_tests_count: </b>Total count of human-written tests found in the considered PRs, reflecting the volume of manually created tests by developers.<b>no_of_prs_instruction_coverage_increase:</b> Number of PRs where instruction coverage increased due to human-written tests, showing the capability of manual tests to improve instruction execution.<b>no_of_prs_branch_coverage_increase:</b> Number of PRs where branch coverage increased due to human-written tests, showing how well manual tests explore decision-making paths.<b>no_of_prs_line_coverage_increase: </b>Number of PRs where line coverage increased due to human-written tests, reflecting the additional lines of code executed.<b>no_of_prs_complexity_coverage_increase:</b> Number of PRs where complexity coverage increased due to human-written tests, highlighting their ability to cover complex code structures.<b>no_of_prs_method_coverage_increase:</b> Number of PRs where method coverage increased due to human-written tests, indicating the extent to which methods are exercised by manual tests.<b>average_change_in_instruction_coverage:</b> Average improvement in instruction coverage across all PRs due to human-written tests, showing their overall impact on instruction execution.<b>average_change_in_branch_coverage:</b> Average improvement in branch coverage across all PRs due to human-written tests, highlighting their influence on decision-making path coverage.<b>average_change_in_line_coverage:</b> Average improvement in line coverage across all PRs due to human-written tests, reflecting the effectiveness of manual tests in executing more lines of code.<b>average_change_in_complexity_coverage: </b>Average improvement in complexity coverage across all PRs.<b>average_change_in_method_coverage: </b>Average improvement in method coverage across all PRs due to human-written tests, showing their effect on the testing of different methods.<b>project_name:</b> Name of the project from which the PRs were analyzed.<br><br><b>Most Possible Change Analysis:</b>These files explore the hypothetical maximum improvements in coverage metrics, assuming that the generated tests successfully cover all the changes in the classes under test.<b>Files:</b>analysis_results_most_possible_change_gemini-1.0-pro.csvanalysis_results_most_possible_change_gemini-1.5-flash.csvanalysis_results_most_possible_change_gpt-4.csvanalysis_results_most_possible_change_gpt-4o.csv<b>Column Names and Descriptions (Data for Table V):</b><b>average_most_possible_change_in_instruction_coverage</b><b>:</b> The average potential increase in instruction coverage that could be achieved if all possible changes in the PRs were optimally covered by the generated tests.<b>average_most_possible_change_in_branch_coverage</b><b>:</b> The average potential increase in branch coverage that could be achieved. Indicates how much more decision-making paths could have been accessed if the generated tests were more comprehensive.<b>average_most_possible_change_in_line_coverage:</b> The average potential increase in line coverage across PRs. Reflects how many additional lines of code could be covered if the tests addressed all possible scenarios.<b>average_most_possible_change_in_complexity_coverage</b><b>:</b> The average potential increase in complexity coverage. Demonstrates how much more intricate code structures could have been tested if the generated tests fully exploited their potential.<b>average_most_possible_change_in_method_coverage</b><b>:</b> The average potential increase in method coverage. Indicates how many more methods could have been tested if the generated tests maximized their reach.<b>project_name</b><b>:</b> The name of the project from which the PRs were analyzed.<br> <br><b>Plots:</b><b>figure-2.pdf</b><b>:</b> High-resolution version of Figure 2 in the paper. (Build success rate of generated unit test classes)<b>figure-3.pdf</b><b>:</b> High-resolution version of Figure 3 in the paper. (Build success rates per project)<b>figure-4.png</b><b>:</b> High-resolution version of Figure 4 in the paper. (Proportion of PRs where generated tests increased line coverage of changed classes)<b>figure-5.pdf</b><b>: </b>High-resolution version of Figure 5 in the paper. (Distribution of line coverage increases for PRs with improved coverage)<br><br><b>Running the Experiment Code:</b>Step 1: Set Up the Virtual Environment with Necessary LibrariesSet up a Python virtual environment. Navigate to the extracted directory and run the following commands:<br>python3 -m venv .venv<br>source .venv/bin/activate<br>pip install -r requirements.txt<br><br>Step 2: Entering API InformationBefore running the scripts, ensure all required environment variable information, such as API keys, are correctly entered with your own information. Update the `env.py` file with the necessary keys. Ensure that the required AZURE_OPENAI_API_KEY key is entered according to Azure's OpenAI setup. Unlike the standard OpenAI API, you will need the Azure-specific API key from the Azure portal. <br>Step 3: Running ScriptsUse `fetch_prs.py` to retrieve pull requests:python3 generate_tests_new.pyRun `attach_diffs_to_prs.py` to append diffs on pull requests:<br>python3 attach_diffs_to_prs.py<br>Run `generate_tests.py` to generate test cases based on the pull request changes:<br>python3 generate_tests.py<br>Run `validate_and_repair_test.py` to handle small errors in the generated tests:<br>python3 validate_and_repair_test.py<br>Use `run_tests.py` to execute developer-written and generated tests:<br>python3 run_tests.py<br>Run `analyze_test_results.py` to calculate metrics and summarize findings:<br>python3 analyze_test_results.py<br>Generate the plots:<br>python3 plot.py<br>