## 4 Materials and Methods
### Proto-MAML  
#### 4.3.1. BERT Preprocessing and Tokenization  
To adapt BERT with \(\mathcal{S}\) and \(\mathcal{Q}\) to the Proto-MAML architecture, it is essential to ensure that the inputs comply with the specific requirements and dimensions of the model. BERT encompasses a wide range of pre-trained models varying in parameter size, tokenization type, and number of transformer layers. For this project, the BERT Uncased variant [50] was selected (which will hereafter be referred to simply as BERT). This version stands out for incorporating 12 transformer layers, thus optimizing the informational capacity of input sequences. Additionally, it includes 768 hidden layer dimensions, preserving a deep representation of semantic, lexical, and syntactic features, alongside 12 attention heads that enable precise refinement of the relevance of each word in these representations. Due to its case insensitivity, BERT is particularly suitable regarding the volatile and evolving syntax of source code. Furthermore, its compact vocabulary and ability to adapt to multiple domains greatly enhance the understanding of natural language questions, source code as a technical abstraction, and answers that integrate both aspects. Figure 4 provides a schematic visualization of the data flow through a general BERT model in a QA environment. The sets \(\mathcal{S}\) and \(\mathcal{Q}\) are first tokenized into word units and transformed into different input embeddings, which are then processed by the transformer layers to generate output embeddings used in predictions for \(A^{\prime}_{V}\), which are composed of \(A_{V[\text{START}]}\) and \(A_{V[\text{END}]}\). Before being adapted by BERT, the tuples \(Q_{V}\) and \(C_{V}\) within \(\mathcal{S}\) and \(\mathcal{Q}\) must undergo a preprocessing procedure to eliminate noisy textual content and out-of-context vocabulary, which could lead to incorrect or poorly interpretable outputs. The specific steps for preparing the inputs are listed below:  
* _Handling Comments_: Comments within \(C_{V}\) are identified using language-specific delimiters (e.g., // or /* */ for PHP). Depending on their relevance, these comments are either retained to enrich the context or removed to eliminate superfluous information. * _Normalization:_ Non-essential whitespace, tabulations, and special symbols are standardized. Natural language text is converted to lowercase, ensuring alignment with the case-insensitive characteristics of the BERT tokenizer. * _Irrelevant Character Filtering:_ Unnecessary symbols, such as excessive punctuation or repeated line breaks, are removed from \(C_{V}\). However, tokens integral to the source code, such as function calls, variable names, and SQL queries, are preserved to maintain contextual integrity. * _Retention of Stopwords:_ Stopwords are deliberately retained within \(Q_{V}\) and \(C_{V}\), diverging from traditional preprocessing techniques. Preserving these elements is essential to maintain the semantic and syntactic coherence between questions and contexts, particularly for the accurate identification of answer spans within \(C_{V}\). After preprocessing, the BERT tokenizer (denoted as \(\tau\)) tokenizes sentences into unigrams using the WordPiece algorithm, generating subwords for the elements \(Q_{A}\) and \(C_{V}\) present in \(\mathcal{S}\) and \(\mathcal{Q}\) for each task \(\mathcal{T}_{i}\), and forming a unified sequence as shown in Equation (5). \[[\texttt{CLS}]+Q_{V}+[\texttt{SEP}]+C_{V}+[\texttt{SEP}]. \tag{5}\]  
In this sequence, \([\texttt{CLS}]\) acts as a classification marker, while \([\texttt{SEP}]\) separates the question from the context. To illustrate this process, the PHP code described in Listing 1 is used as an example, which describes an input that is vulnerable to SQL injection. Figure 4: Schematic representation of the data flow through BERT in a QA environment.