# Machine Learning-Based Code Auto-Completion Implementation for Firmware Developers
## 2 Literature Review
### Research Gap  
Although many research groups have deployed a transformer-based language model for code auto-completion functionality, it is important to note that they have trained the model, with open source codes mostly coming from GitHub. For example, the Deep TabNine [17] is trained on around 2 million files from GitHub. This indicates that the software may not be applicable if one needs to predict very unique code patterns and style. Therefore, a need to establish a domain-specific language model has been identified based on the following reasons: (1) firmware codes implement very specific sets of features for the hardware and (2) firmware codes typically comply with unique coding styles and patterns optimized for embedded environments.  
In fact, some companies (e.g., TabNine) advertise that they offer a GPU-based cloud service that enables users to create a custom model by fine-tuning the model with their input data. However, a security-sensitive company such as Samsung Electronics may not want to utilize the commercial tools given that there is a risk of leaked source codes. In addition to the security issue, a company has to pay for the license fee because the tools are not free to use the service. Thus, a need to develop in-house codes for code auto-completion functionality is identified.  
As we seek to develop a domain-specific (i.e., solid state drive (SSD) firmware development) language model by using the GPT-2 model, it naturally leads us to consider how to determine diversity parameters (i.e., _Top_k_, _Top_p_, and Boltzmann temperature) of the model. Given that there has not been any analysis done on the optimal diversity parameter values especially on the SSD firmware development domain, the following research question can be constructed: \"How can we determine the GPT-2 diversity parameter values properly for the SSD firmware development domain\"?  
To answer the question, in this paper, we propose a hybrid approach that harnesses the synergy between ML techniques and advanced design methods (e.g., design of experiment, surrogate modeling, and Monte Carlo simulation) [19] to enhance the level of understanding of the relationship between  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Approach** & **Example Method** & **Advantage** & **Limitation** \\ \hline \multirow{3}{*}{Integrated Development Environment (IDE)} & \multirow{3}{*}{Eclipse’s content assist feature} & It provides a list of type-compatible names for the next tokens immediately & It is generally organized alphabetically, which is not very effective \\  & & It solves a drawback of & \\  & & IDE’s ineffectiveness by optimizing/ranking & It is difficult to be scaled to large programs \\  & & the list & \\ \hline \multirow{3}{*}{Machine learning-based Language Models} & \multirow{3}{*}{Recurrent Neural Network (RNN)} & It is typically better than statistical language models in predictions & It is not effective to capture long-term dependencies \\  & & It overcomes the issue & \\ \cline{1-1} Transformer-based Language Models & \multirow{3}{*}{Generative Pre-trained Transformer (GPT)} & It overcomes the issue related to long-term dependencies by introducing attention mechanisms & It is computationally expensive for those who use a personal computer \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparative table of the related works.  
the GPT-2 model diversity parameters and code auto-completion functionality in the SSD firmware development domain. Figure 4 notionally illustrates the process of the hybrid approach used for this research.