# Evolutionary Approaches for Adversarial Attacks on Neural Source Code Classifiers
## 4 Experimental Settings
### Mutation Operators  
The proposed mutation operators are designed to introduce semantic-preserving modifications to existing functions, so as to alter the code of a program without altering its semantics, that is, the behavior of the program in terms of relation between input and output. In general, there exist many possible semantic-preserving transformations. Here, we consider two of them, one that is based on a consistent renaming of the variable identifiers, and the other one that relies on the introduction of dead code in the functions.  
First operator  
The \(M_{1}\) mutation operator consists of modifying the identifiers within the source code of the functions so that, in each function \(F\), each occurrence of the identifier ID\(i\) is consistently replaced with the same new identifier ID\(i^{\prime}\). To this end, two different vocabularies of words are used, and the algorithm can generate new variable names during the mutation process by selecting words from these vocabularies. They are meant to follow two different policies:  
1. The first vocabulary is composed by \(\approx\) 450K generic English words ([[https://github.com/dwyl/english-words](https://github.com/dwyl/english-words)]([https://github.com/dwyl/english-words](https://github.com/dwyl/english-words)), accessed on July 2023). It is used with the underlying idea of not bringing any domain-specific bias into the pool of possible identifiers. We will refer to this configuration as \(M_{1}^{E}\).
2. The second, smaller, vocabulary is instead composed of \(\approx\)30 words or expressions that are somehow related to known vulnerabilities, shown in Table A1 (in Appendix A). We denote this experimental configuration with \(M_{1}^{S}\).  
Algorithm 3 shows the functioning of the first mutation operator \(M_{1}\).  
```
1:input: individual \(p\), genome \(g_{p}\)\(\triangleright\) original genome
2:output: individual \(p\), genome \(g_{p}\)\(\triangleright\) mutated genome
3:for all\(id\in g_{p}\)do
4: with probability \(mProb\)do\(\triangleright\) mutation probability chosen
5:\(new1d\leftarrow\) IDfromVocab()\(\triangleright\) random identifier
6:if\(new1d\
ot\in g_{p}\)then\(\triangleright\) check for consistency behavior
7:\(g_{p}[i]\gets new1d\)\(\triangleright\) update genome with new identifier
8:else
9:while\(new1d\in g_{p}\)do
10:\(new1d\leftarrow\) IDfromVocab()
11:endwhile
12:\(g_{p}[i]\gets new1d\)
13:endif
14:updateIndiv1(\(p,g_{p}\))
15:done
16:endfor
17:return\(p,g_{p}\)
```  
**Algorithm 3** First mutation operator \(M_{1}\)  
In the \(M_{1}^{E}\) experiment, each run took approximately 21 h to complete. This indicates the time required for the ES algorithm to iterate through the specified number of genera tions, perform the fitness evaluations, and apply the mutations to the individuals in the population. For the \(M_{1}^{5}\) experiment, instead, each run took approximately 10 h to complete.  
With regard to the domain-specific set of words (\(M_{1}^{5}\)), in the context of black box attacks where explicit knowledge of the neural model is limited, an attempt is made to deceive the neural network using a limited vocabulary. This method aims to explore the possibility of influencing the model's predictions by replacing words (identifiers) with others having different semantics. This choice comes because the model might misclassify code snippets containing trigger words that might recall vulnerability names. A simplified example of input and output is shown in Figures 3 and 4.