Replication Package of the Paper: "REACT: A Retrieval-Augmented Framework for Enhancing Commit Message Generation"
Creators: name: Anonymous
Publisher: name: Zenodo
Year: 2024

description: Replication Package of the Paper: "REACT: A Retrieval-Augmented Framework for Enhancing Commit Message Generation"

This package includes all data and scripts for reproduction. All datasets, experimental results and model checkpoints can be found in the /data/ folder. 

Directory Structure

Below is an overview of the project's directory structure and a brief description of each file and folder:

REACT/ # Root directory├── data/│ ├── checkpoints/│ ├── dataset/│ ├── experimental_results/│ ├── tokenizers/│ └── source.csv├── scripts/├── README.md└── requirements.txt/data/checkpoints/: This folder contains all PLMs fine-tuned checkpoints, can be directly used for inference and evaluation./data/dataset/: This folder contains the original dataset and augmented dataset. The augmented dataset has prefix `react`./data/experimental_results/: This folder contains all experimental results generated by PLMs, LLMs and baselines, the metric scores can be calculated using `eval.py`. Subfolder `directly` contains the results directly applying PLMs and LLMs on CMG, while subfolder `react` contains the results generated by REACT./data/tokenizers/: This folder contains all tokenizers of PLMs./data/source.csv: The source database for exemplar retrieval, which is derived from the [CommitBench](https://zenodo.org/records/10497442).

Environment setup

Python >= 3.11 is required.For general use:pip install -r requirements.txtFor those who want to run data_construct.py: You need to manually install PyLucene that is not pip installable. For those who want to run ollama_infer.py for LLMs inference: Firstly, manually install ollama and pull the LLMs you want to run from ollama library, then pip install ollama.

 

Data construction

cd scriptspython data_construct.py --origin_dataset ../data/dataset/test.csv

The script will retrieve the most similar diff-message pair from the source database for each row in the dataset, and save the augmented dataset into `dataset` folder.

Fine-tuning of PLMs

python finetune.py \--model Salesforce/codet5-base \--tokenizer ../data/tokenizers/codet5_react_tokenizer \--model_type t5 \--max_source_len 1024 \--dataset ../data/datasetThis script is used to fine-tune CodeT5, CodeT5+, and PLBART with the augmented dataset. For UniXCoder, please refer to its official repository.The tokenizers saved in data/tokenizers have three special tokens added ([QUERY], [DIFF], [MSG]) that will be fully trained in the fine-tuning process.

Inference of PLMs

 

python inference.py \--model_path ../data/checkpoints/codet5_react \--tokenizer_path ../data/tokenizers/codet5_react_tokenizer \--model_type t5 \--batch_size 64 \--dataset ../data/dataset/react_test.csvThis script is used to generate commit messages using the fine-tuned PLMs in batch way. The generated commit messages will be saved in `data/experimental_results/` folder.

Inference of LLMs

python ollama_infer.pyWe provide a simple script to run inference on LLMs using ollama framework. Before running it, you need to serve ollama service in the background first.

Evaluating results

python eval.py --result_file ../data/experimental_results/react/codet5_react.csvThe script will calculate BLEU, Rouge-L, METEOR scores using the implementation of huggingface's `evaluate` library, with all parameters set to default. You can replace the `result_file` with the path to your own result file.