# Complex Cases of Source Code Authorship Identification Using a Hybrid Deep Neural Network
## 7 Experiment Setup and Results
The second experiment was devoted to assessing the accuracy of determining the unity of the generative model using two samples of source code (Figure 6). Two source code samples generated by the same language model were used in this experiment. The study involved 50 Java source code samples split into pairs for each generative model, respectively. These pairs were alternately fed into the input of the model trained for 5, 10, and 20 authors, one of which was the target generative model. The average accuracy was 94% for GPT-2, 92% for GPT-3, and 91% for RuGPT-3. Figure 5: Difference between human and generative models. Figure 6: The unity of the generative model for pairs of source code samples. The third and final experiment on artificially generated source codes was aimed at evaluating model accuracy in distinguishing authorship between three different generative models (Figure 7). In this case, the experiment was set up on three generative models that participated in the training: GPT-2, GPT-3, and RuGPT-3. The purpose of HNN was to determine whether the anonymous source code, which was artificially generated, belonged to one of the language models. The experiment was conducted in five programming languages: Java, C++, Python, JavaScript, and Go. The average accuracy of determining the authorship of a source code generated by a language model was 94%. The last experiment focused on repository contributors' commits (Figure 8). The dataset was generated from the commit samples (fragments modified in a common source code file by a particular user) of individual users. To identify methodological recommendations for identifying the authors, which develop commercial software and use version control systems, we decided to experiment with a different number of files--5, 10, and 20, respectively. According to the results, at least 10 source code files per programmer should be used to obtain a highly accurate tool for making decisions about authorship. If this condition is not met, the accuracy may be insufficient for efficient problem solving. The most important case is the commit base data, due to the prevalence of commercial software development. Based on this fact, we decided to apply for comparison two other generally well-known models--fastText [34] and BERT [35]. Our decision is driven by their high efficiency in solving problems in related fields [36; 37]. Bert-base-multilingual-cased was used as a pre-trained version of BERT. Model tuning was performed according to the recommendations of the model's creators: learning rate = 3 \(\times\) 10\({}^{-5}\), warmup proportion = 0.1, train batch size = 16 and 5 epochs. For fastText training, we used experimentally selected parameters: learning rate = 0.1, rate of updates for the learning rate = 100, size of the context window = 5, number of negative samples = 5, loss function--SoftMax, and 50 epochs. Figure 9 shows a graph of BERT accuracies depending on the number of authors and samples for training, and a similar graph for fastText is shown in Figure 10.  
Figure 7: Difference between generative models. Figure 8: Authorship identification accuracy based on commits. Figure 9: Authorship identification accuracy based on commits (BERT). Non-parametric [PERSON] and [PERSON] post hoc tests were used to check if there is a statistically significant difference between HNN, BERT, and fastText. The tests were applied to the results of the models' cross-validation for 5, 10, and 20 authors. The number of training samples per author is 20. A null hypothesis is that a difference between the results of the models is random. An alternative hypothesis is that a difference is statistically significant. According to the results of the calculations, the \(p\)-value was 0.000079 for five authors, 0.000056 for 10, and 0.000093 for 20. The null hypothesis was rejected since the \(p\)-value did not exceed 0.05 in any case. [PERSON]'s statistical test confirmed the significance of the difference between the results.