# Machine Learning-Based Code Auto-Completion Implementation for Firmware Developers
## 4 Results and Discussion  
### Sensitivity Analysis  
Sensitivity analysis with respect to the GPT-2 diversity parameters is performed to enhance the level of understanding of the relationship between prediction accuracy and the diversity parameters. The GPT-2 model has three different diversity parameters implemented in the sampling process. The Boltzmann temperature is one of the GPT-2 model diversity parameters that control randomness in the sampling process. Figure 11 shows the MCS results with respect to the Boltzmann temperature. Lower and upper bounds are specified with 0.1 and 0.9, respectively. A black dot represents one experiment case generated by the MLP-based surrogate model with three different input variables randomly sampled from the uniform distributions with respect to the GPT-2 model's diversity parameters. As can be seen from Figure 11, it seems that decreasing Boltzmann temperature (i.e., \(x\)-axis) keeps the model to generate a high prediction score (i.e., \(y\)-axis), while increasing Boltzmann temperature causes the model to tend to frequently have a low prediction score. Based on these observations, one may claim that the model with lower Boltzmann temperature value, named deterministic design in this paper, would be the best option to predict the most likely  
Figure 10: Notional sketch of the MCS process flow diagram. \begin{table}
\begin{tabular}{c c} \hline \hline
**Case** & **Score** \\ \hline An option with the highest probability matches the actual code & 100 \\ There is an option among the possible candidates, which matches to the actual code & 50 \\ There is no available option that matches the actual code & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Score metric for the MLP-based surrogate modeling. next code element as the deterministic design strives to minimize the degree of surprise in model output. However, it is too early to draw such a conclusion, because the model with a higher Boltzmann temperature value, named probabilistic design in this paper, would provide a list of reasonable next code elements in which one could select it manually to increase prediction accuracy. Details will be discussed in the section of model evaluation. The _Top_k_ is another GPT-2 model diversity parameter that controls the number of sampling words to be considered. For example, the most likely word is only considered if the _Top_k_ is equal to onem thus resulting in deterministic design. The deterministic design can successfully eliminate rather weird candidates; however, one may claim that better results would be achieved if the algorithm considers sampling words more than one. Figure 12 shows the MCS results with respect to the _Top_k_ parameter. Lower and upper bounds are specified with 40 and 100, respectively. As can be seen, it appears that the _Top_k_ parameter value does not have a significant impact on the prediction score, indicating that the _Top_k_ parameter may not entirely contribute to the model output diversity. Figure 11: MCS results for diversity parameter 1 (i.e., Boltzmann temperature). Figure 12: MCS results for diversity parameter 2 (i.e., _Top_k_). The _Top_p_, one of the GPT-2 model diversity parameters, considers sampling words from the largest possible set of words whose cumulative probability exceeds a user-defined number. Instead of sampling only from the most likely \(K\) words, the _Top_p_ parameter provides an option that dynamically controls the size of the set of sampling words to be considered. For example, if the _Top_p_ is equal to 0.9, the algorithm computes cumulative probability distribution (CDF) and cuts off the words as soon as the CDF exceeds 90 percents. Figure 13 shows the MCS results with respect to the _Top_p_ parameter. Lower and upper bounds are specified with 0.1 and 0.9, respectively. As the _Top_p_ parameter value increases, it results in more randomness in terms of the prediction score. On the other hand, as the _Top_p_ parameter value decreases, it leads to less randomness with regard to the prediction score.