An Empirical Study of Interrogative Comments Posed by Review Comment Generators in Gerrit (complimentary material and replication package)
Creators: name: Anonymous, Anonymous
Publisher: name: Zenodo
Year: 2024

description: This record includes the complementary file for the paper "An Empirical Study of Interrogative Comments Posed by Review Comment Generators in Gerrit." It contains all the files and code to replicate our experiment and the generated dataset. 

# Note**Please have a look at the `Install.md` file for the installation instructions.**

# What is this replication package?This package is to help researchers/practitioners to replicate our study "Interrogative Comments Posed by Automatic Code Reviewers—An Empirical Study of the Gerrit Community." This package has both **code** and **dataset** used in this study, as well as all the materials required to replicate our code, analysis, tables and figures.This package contains a step by step scripts of files/notebooks to first mine the data from Gerrit project, and then replicate our empirical study.

# Replicate this studyIn order to replicate this study, you can run run the files indicated with step_X_ in their names. Please follow the instructions in the notebooks or on top of the python files to replicate the results. Please note, that in some cases, such as when running the mining algorithm, the scripts calls external APIs and you may need to provide your credentials or use a proxy in order not to be banned by the API provider.

# Important Note about replicationAs some of the simulations and analyses take time, we also provided our intermediary results along with instructions and scripts on how to redo the experiments. If one would like to replicate the paper with the result of our simulation, they can only run the parts with "*" next to their time. As for analysis, we put pointers on how to fast-track the run using the data from the replication package. These items will replicate the drawings and table information based on the experiments we have in our paper. 

 # Overview of the processIn short, you need to run the following steps to replicate this study to its fullest:1. Extract the content of this replication package to a folder (Explained in the `Install.md` file)2. Install the necessary packages (Explained in the `Install.md` file)3. For each step, run the scripts inside the folders in the order of their names ("step_x_").    1. Run all the notebooks with the prefix "Step_0_" to "Step_5_" to reproduce the results of the paper.     2. Wait for the run to Complete. Some scripts may take some times in total.4. The results of our labelling and Kappa scores are available in the `RQ2_analysis` folder. 

# Available files in the replication packageThis replication package consists of two folders and a docker-compose file.- `step_1_Data_Extraction`: The scripts and tools used to mine the curated dataset from the Gerrit project. Also, we identify data duplication between our dataset and fine-tuning dataset in this step.- `step_2_Task_Specific_Comment_Generation`: The scripts and tools used to generate comments using the task-specific models.- `step_3_LLM_Comment_Generation`: The scripts and tools used to generate comments using the LLM models.- `step_4_RQ1_analysis`: The scripts and tools used to analyze the results of RQ1.- `step_5_RQ2_analysis`: The scripts and tools used to analyze the results of RQ2.- `step_6_follow_up_analysis`: The scripts and tools used to analyze the results of the follow-up analysis.

 

# PrerequisiteIn order to run the containers, please follow the steps in the `Install.md` file.