FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair
Creators: name: Fatima, Sakina
Publisher: name: figshare
Year: 2025
Subjects: classificationCode: 461201
subject: Automated software engineering
classificationCode: 461208
subject: Software testing, verification and validation
classificationCode: 461103
subject: Deep learning

description: This is the replication package associated with the paper: 'FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test for Code Repair'<br>### RequirementsThis is a list of all required python packages:-imbalanced_learn==0.8.1-numpy==1.19.5-pandas==1.3.3-transformers==4.10.2-torch==1.5.0-scikit_learn==0.24.2-openai==v0.28.1<br>#Automated tool for labelling dataset with flaky test fix categories<br>This is a step-by-step guideline for automatically labelling dataset with flaky test fix categories<br><br>### Input Files:This is a an input file that is required to accomplish this step:* Data/IdoFT_dataset_filtered.csvhttps://figshare.com/s/47f0fb6207ac3f9e2351<br>### Output Files:* Results/IdoFT_dataset_filtered.csv<br><br>### Replicating the experiment<br>This experiment can be executed using the following commands after navigating to the `Code\` folder:<br>```consolebash Automated_labelling_tool.sh```<br>It will generate the dataset required to run our prediction models to predict the category of the fix, given a flaky test code<br>---<br># Prediction models for fix categories using the test case code<br>This is the guideline for replicating the experiments we used to evaluate our prediction models i.e. CodeBERT and UniXcoder (fine-tuned with Few Shot Learning and Feed Forward Neural Network independently) for classifying tests with different fix categories. <br>### Input Files:This is a list of input files that are required to perform the binary classification for each fix category:* Data/change_assertion.csv*Data/change_condition.csv*Data/change_data_format.csv*Data/change_data_structure.csv*Data/handle_exception.csv*Data/reorder_data.csv*Data/reset_variable.csv*Data/call_static_method.csv*Data/reorder_parameters.csv*Data/misc.csv<br>### Replicating the experiment<br>To run experiment with our prediction model, navigate to the `Code\` folder and run the following commands:<br>```consolebash codemodel_with_fnn.shbash codemodel_with_fsl.sh```<br># Generate the repaired flaky tests using GPT 3.5 Turbo:<br>Input Files:This is a an input file that is required to accomplish this step:<br>### Experiments on the 181 tests using prompts with and without labels:* Data/Dataset_for_GPT.csv<br><br>### Experiments on the tests using prompts with, without labels and in-context learning:<br>###For Change Assertion:* Data/change_assertion_input_FSP.csv###For Change Condition:* Data/change_condition_input_FSP.csv###For Change DataStructure:* Data/change_dataStructure_input_FSP.csv<br>To run this experiment, navigate to the `Code\` folder and run the following commands:<br>```consolebash gpt3.5_experiments.sh```# Execute a sample of GPT-reapired flaky tests:<br>Input Files:This is a an input file that is required to accomplish this step:* Data/sampleTests_For_Execution.csv<br>To execute the 35 GPT-repaired flaky tests:<br>-First Clone the Github project (From the 'PR Link' column in the sampleTests_For_Execution.csv file)-Checkout on the the commit of the given PR link (if merged, checkout on the master branch)-Navigate to the project folder and run command: ```console*mvn clean test -Dtest=[Test Class Name]#[Test Method Name] -DfailIfNoTests=false```