# Improving VulRepair's Perfect Prediction by Leveraging the LION Optimizer
## 3 Methodology
### Improving VulRepair: ImpVulRepair
#### 3.3.6 Encoder and Decoder Block Sizes  
The T5 transformer is an encoder-decoder model, or a text-to-text transfer transformer, used in various natural language processing (NLP) tasks that converts text input into text output. Encoders are used to understand and extract relevant information from text input and then convert the input sequence into a vector. The decoder inputs this vector and predicts an output sequence. As noted by [PERSON] et al., the T5 architecture is limited to 512 tokens, so anything larger is truncated. This means that the truncated portion is not processed by the machine learning model and its usefulness is lost. Similarly, VulRepair performs best if the patch to the vulnerability contains fewer than 20 tokens. [PERSON] et al. stated that researchers should explore techniques that handle these larger token sizes for the vulnerability function and its repair [7]. We note that the model itself is provided in binary form, so we did not try to adjust the neural network layers.  
However, according to the GitHub site of [PERSON] et al., the encoder block size is set to 512, the maximum source length, and the decoder block size is set to 256, the maximum target length [22]. Since these values were provided, we adjusted them and used them for training and testing. We treated them as hyperparameters and observed their performance. We found that doubling the block size to 1024 for the encoder and 512 for the decoder produced stable results after adjusting other hyperparameters.  
When we increased the size of the block, the training time increased. For example, it took 17 h to train for 57 epochs when we set the encoder to 1024 and the decoder to 512. However, it took 9 h to train the model for 75 epochs when the encoder was set at 512 and the decoder at 256. This is not surprising since padding can significantly slow training and should only extend to the longest example in a batch [37]. In addition, when we doubled the block sizes, the GPU and Google Drive disconnected at 57 epochs. In this case, we did not achieve stable results but proceeded with testing the model, achieving a perfect prediction of 40%. This is lower than the PP of Fu's VulRepair of 44%. We adjusted the hyperparameters to achieve stability in subsequent tests.  
We also found that anything over 1536 for the encoder and 512 for the decoder block sizes immediately produced an out-of-error response. We tried different combinations, such as 1280/512 for the encoder/decoder, respectively, during training. Some combinations allowed the model to train but did not test well. Memory errors, GPU and file disconnections, and screen freezing could occur.  
We also tried to eliminate out-of-memory errors by reducing the beam size. For example, we increased the encoder block size to 1280 and the decoder block size to 512 during training. We decreased the beams from 50 to 40. ImpVulRepair achieved a perfect prediction of 52.7%.  
Although we could have further adjusted different combinations of hyperparameters to increase the perfect prediction percentage, we limited the training to 1024 for the encoder and 512 for the decoder. Although we tested a beam size of 40, we did not perform an exhaustive search to determine the optimal relationship between the encoder, decoder, and beam size. We could achieve stable testing by setting the beam size to 50 depending on other hyperparameter settings.