# Improving VulRepair's Perfect Prediction by Leveraging the LION Optimizer
## 3 Methodology
### Replication of Other Models  
Following the replication of M1, we proceeded with the replication of the models in [7]. Models 1 and 2 were trained from scratch, allowing us to closely examine their training processes. For Models 3 to 10, we utilized the pretrained models supplied by Fu et al. The goal was not to train, enhance, or benchmark CodeBERT and VRepair, but rather to gain a comprehensive understanding of Fu et al.'s research.  
Model 2 (M2) is the CodeBERT platform that utilizes a BPE tokenizer, programming language (PL) pretraining, natural language (NL) pretraining, and a BERT architecture. In this case, we trained the model using current libraries and used SalesForce/codet5-base, as in M1, and Google Colab. We also used the deprecated AdamW optimizer, as our intention was not to improve these models. We did not use the Fu et al. 'MikeyMike' tokenizer. We obtained a 34.47% perfect prediction after training and testing. Fu et al. reported a 31% PP [22]. Our results were better by 3.5%, but we did not use the exact methods of [PERSON] et al. We did not conduct further testing to pinpoint the improvements, nor did we use their trained model.  
Model 3 (M3) is VulRepair without PL pretraining compared to M1, which includes it. This model demonstrates the importance of PL pretraining. In this instance, we did not train the model but used the one provided by [PERSON] et al. We tested the model and obtained a 29.89% PP. We compared this to [PERSON] et al.'s 30% PP.  
Model 4 (M4) is VulRepair with no PL or NL pretraining. This model demonstrates that pretraining yields a better PP. We did not train this model from scratch but tested it. We obtained a 6.21% PP compared to 6% in the [PERSON] et al. model [22].  
Model 5 (M5) is CodeBERT, or M2, without PL. This was used to compare VulRepair and CodeBERT when no PL pretraining was used. We tested the model but did not train it from scratch. We obtained a 29.19% PP compared to [PERSON] et al.'s reported results of 29% [22].  
Model 6 (M6) is CodeBERT without pretraining. Like M5, it was used to compare with VulRepair's results. Here, we tested it but did not train it from scratch, as this was not our primary focus. We obtained a 0.64% PP compared to [PERSON] et al.'s reported results of 1%.  
Model 7 (M7) is VulRepair with a word-level tokenizer that replaces its subword tokenizer. This demonstrates the importance of the tokenizer. We replicated this through testing. Since it was not our focus, we did not train it from scratch. We obtained a 35.17% PP as expected, consistent with the [PERSON] et al. model's results of 35%.  
Figure 1: We successfully replicated VulRepair’s 44% PP (M1) and other models, as explained in Section 3.2.  
Model 8 (M8) is VRepair, or the vanilla transformer, with a subword tokenizer instead of a word-level tokenizer. This model was compared to both VulRepair and CodeBERT. We tested this model but did not train it from scratch, obtaining a 34.17% PP as expected, consistent with [PERSON] et al.'s results of 34% [22].  
Model 9 (M9) is CodeBERT with a word-level tokenizer instead of a subword tokenizer. This was compared to VulRepair's results, demonstrating the importance of tokenization. We tested the model but did not train it from scratch. We obtained a 16.65% PP compared to 17% in the [PERSON] et al. model [22].  
Model 10 (M10) is VulRepair with no pretraining and a word-level tokenizer instead of a subword tokenizer. This shows that VulRepair requires NL and PL pretraining and subword tokenization to achieve superior results compared to CodeBERT and VRepair. Without training from scratch, we tested the model, achieving a PP of 0.64%, as expected, compared to [PERSON] et al.'s model results of 1%.