EMSE PTM4Tag+: Tag Recommendation of Stack Overflow Posts with Pre-trained Models Replication PackageEMSE-- PTM4Tag+: Tag Recommendation of Stack Overflow Posts with Pre-trained Models
Creators: name: He, Junda
Publisher: name: figshare
Year: 2024
Subjects: classificationCode: 461202
subject: Empirical software engineering

description: <strong># Replication Package for PTM4Tag+: Tag Recommendation of Stack Overflow Posts with Pre-trained Models</strong><br>
<br>

This is the codebase for the EMSE submission: "PTM4Tag+: Tag Recommendation of Stack Overflow Posts with Pre-trained Models"<br>
<br>

<strong>## Data</strong><br>
<br>
The data used in thia paper can be download from this [link](https://zenodo.org/record/5604548#.YXoG7NZBw1I). <br>
<br>

```shell<br>
 # Download train data<br>
 wget https://zenodo.org/record/5604548/files/train.tar.gz<br>
 # Download test data<br>
 wget https://zenodo.org/record/5604548/files/test.tar.gz<br>
 # tag file<br>
 wget https://zenodo.org/record/5604548/files/_1_commonTags.csv<br>
 # unzip file, change the filepath to your actual filepath<br>
 tar -xvf file.tar.gz <br>
<br>
```<br>
<br>

<strong>## Requirements</strong><br>
Our experiments were conducted under Ubuntu 18.04. We have made a ready-to-use docker image for this experiment.<br>
<br>
```<br>
docker pull dennishe9707/ptm4tag:emse<br>
```<br>
<br>
Then, assuming you have Nvidia GPUs, you can create a container using this docker image. An example:<br>
<br>
```<br>
docker run --name=ptm4tag_plus --gpus all -it --mount type=bind,src=/media/codebases,dst=/workspace dennishe9707/ptm4tag:emse<br>
```<br>
<br>
<strong>## Overview</strong><br>
<br>
```<br>
.<br>
├── README.md<br>
├── preprocess<br>
│   ├── 01_process_to_tensor.py<br>
│   └── example.sh<br>
└── src<br>
    ├── data_structure<br>
    │   ├── __init__.py<br>
    │   └── question.py<br>
    ├── model<br>
    │   ├── __init__.py<br>
    │   ├── loss.py<br>
    │   └── model.py<br>
    ├── rq1<br>
    │   ├── test_triplet.py<br>
    │   ├── train.py<br>
    │   └── train_triplet.py<br>
    ├── rq2<br>
    │   ├── 02_1_train_no_title.bash<br>
    │   ├── 02_2_train_no_text.bash<br>
    │   ├── 02_3_train_no_code.bash<br>
    │   ├── __init__.py<br>
    │   ├── test_triplet_csv.py<br>
    │   ├── test_triplet_no_code.py<br>
    │   ├── test_triplet_no_text.py<br>
    │   ├── test_triplet_no_title.py<br>
    │   ├── train.py<br>
    │   ├── train_no_code.py<br>
    │   ├── train_no_text.py<br>
    │   └── train_no_title.py<br>
    └── util<br>
        ├── __init__.py<br>
        ├── data_util.py<br>
        ├── eval_util.py<br>
        └── util.py<br>
```<br>
<br>
`preprocess/`: Contains scripts related to the initial processing (tokenization) of the dataset.<br>
<br>

`data_structure/`: Defines the data structure of the Stack Overflow posts used in the dataset.<br>
question.py: Contains the Question class definition, representing a Stack Overflow question.<br>
<br>

`model/`: Contains the implementation for the PTM4Tag+ framework.<br>
<br>
`rq1/` and `rq2/`: Directories for the code related to specific research questions.<br>
<br>
`test_triplet.py`: Script for testing the PTM4Tag+ model.<br>
`train_triplet.py`: Script for training the PTM4Tag model.<br>
<br>
`util/`: Utility functions supporting data handling, evaluation, and other common tasks.<br>
<br>
<strong># Tokenization</strong><br>
<br>
Before training the model, we need to tokenize the dataset to convert the raw text into a format that's compatible with the pre-trained models. Tokenization involves splitting the text into tokens (words or subwords), which can then be converted to numerical IDs. For more details, you can refer to this [tutorial](https://huggingface.co/learn/nlp-course/en/chapter2/4).<br>
<br>

For this project, we have utilize various pre-trained models, we provided an example using the [BERTOverflow model](https://huggingface.co/jeniya/BERTOverflow), which is optimized for programming-related text found on Stack Overflow.<br>
<br>
Step-by-Step Guide for Tokenization:<br>
<br>
Prepare your dataset: Download the data. We assume the data files are in the format of pickle files, and we assume your dataset is under the `processed_train` directory.<br>
<br>
We have included an example script, process/example.sh, to demonstrate how to run the tokenization process. Refer to this script as a template for the following steps.<br>
<br>
```<br>
cd /preprocesss<br>
<br>
. example.sh<br>
```<br>
<br>
Here's the command for executing the tokenization script: Run the 01_process_to_tensor.py script with the necessary arguments to tokenize your dataset. <br>
<br>
```<br>
python 01_process_to_tensor.py \<br>
<br>
    --input_dir ../data/processed_train \<br>
    --out_dir ../data/bertoverflow/ \<br>
    --model_type jeniya/BERTOverflow \<br>
    --title_max 50 \<br>
    --text_max 50 \<br>
    --code_max 50<br>
```<br>
<br>
--input_dir: The directory containing your raw dataset.<br>
--out_dir: The directory where the tokenized dataset will be saved.<br>
--model_type: Specifies the pre-trained model used for tokenization. For this example, we use jeniya/BERTOverflow.<br>
--title_max, --text_max, --code_max: Maximum length for the title, text, and code segments of the posts, respectively. Posts longer than these values will be truncated.<br>
<br>

<strong>## PTM4Tag+ Training and Testing</strong><br>
<br>
<strong>### RQ1</strong><br>
navigate to `/src/rq1/`<br>
<br>

<strong>#### Model Train</strong><br>
<br>
```python<br>
CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch \<br>
 --nproc_per_node=2 train_triplet.py \<br>
 --data_folder ../../data/bertoverflow \<br>
 --output_dir ../../data/results \<br>
 --per_gpu_train_batch_size 64 \<br>
 --logging_steps 100 \<br>
 --gradient_accumulation_steps 4 \<br>
 --num_train_epochs 3 \<br>
 --code_bert jeniya/BERTOverflow \<br>
 --learning_rate 7e-5<br>
```<br>
<br>
Key Arguments:<br>
`--data_folder`: Specifies the directory containing the pre-processed data formatted for training. <br>
<br>
`--output_dir`: The output directory where the model checkpoints and predictions will be written.<br>
`--vocab_file`: Specifies the path containing the tags. <br>
--per_gpu_train_batch_size 64: Sets the batch size to 64 for each GPU. This means each GPU will process 64 examples in each training step.<br>
<br>
`--code_bert`: Indicates the pre-trained model to be used for initialization before training. You can the correct model name from the huggingface website. <br>
<br>

<strong>#### Model Test</strong><br>
```python<br>
<br>
python -u test_triplet.py \<br>
 --data_dir ../../data/test \<br>
 --model_path /path/to/your/model_checkpoint.pt<br>
 --test_batch_size 64 \<br>
 --mlb_latest<br>
```<br>
<br>
Make sure to replace /path/to/your/model_checkpoint.pt with the actual file path to your trained model checkpoint.<br>
<br>

<strong>### RQ2</strong><br>
<br>
In RQ2, we conduct the ablation study on the effect of different post components. <br>
Navigate to `src/rq2/`<br>
<br>

```bash<br>
# remove title<br>
. 02_1_train_no_title.bash<br>
# remove description<br>
. 02_2_train_no_text.bash<br>
# remove code<br>
. 02_3_train_no_code.bash<br>
```<br>
<br>
Testing<br>
```python<br>
CUDA_VISIBLE_DEVICES=6,7 python -u test_triplet_no_code.py \<br>
 --data_dir ../../data/test_tensor \<br>
 --test_batch_size 500 \<br>
 --code_bert microsoft/codebert-base \<br>
 --model_path tbert.pt \<br>
 --mlb_latest 2&gt;&amp;1| tee ./logs/test_trinity_no_code0926.log<br>
 <br>
```