# Complex Cases of Source Code Authorship Identification Using a Hybrid Deep Neural Network
## 7 Experiment Setup and Results  
A generalized experimental methodology is demonstrated as a framework diagram in Figure 2. A detailed description of each phase can be found below in this section. A dataset that was appropriate to each complex case was generated to train and evaluate the author's HNN model. The code samples meet the following criteria:  
1. The length of the source code must be at least 30 symbols and no more than 3000 symbols. 2. No more than 30 samples of source code per author are selected. 3. Sample selection is random. According to dataset criteria, the minimum number of code lines for training per author is 90, and the maximum is 90,000. These limits were chosen based on the statistics of the collected dataset. The number of epochs for training HNN is five iterations of 10 epochs, for a total of 50 epochs. A callback procedure named Early Stopping, which automatically interrupts the training, was applied in this study. Practice shows that 50 epochs are sufficient to obtain a minimum error rate and high accuracy. A larger number of epochs leads to an increase in the value loss and overfitting. A smaller number of epochs does not enable high accuracy to be achieved. A 10-block cross-validation method was used to evaluate the model's accuracy. The principle of this algorithm comprises 10 iterations, each of which includes sequential training of the model on nine blocks of data and evaluating it on the one remaining block. In this study, the following experimental cases were considered:  
* Mixed data, including two programming languages (language pairs);
* Mixed data, including three or more languages;
* Distinction of authorship between man and generative models;
* Determining the unity of the generative model for pairs of samples;
* Distinction of authorship between generative models;
* Data obtained from contributors' commits. According to the results, the average classification accuracy for authors who use two programming languages is 87% (Figure 3). At the same time, it should be noted that the accuracy of identifying the author based on language pairs with similarities in syntax exceeds the accuracy of language pairs with no similarities. This finding is especially correct for JavaScript-C++, Swift-Java, and JavaScript-Python language pairs. The average accuracy for authors developing in three or more programming languages is 76% (Figure 4). Since datasets are totally mixed and can include more than ten programming languages per author, it is impossible to evaluate the importance level of each language. It should be noted that this case is particularly of research interest since it is extremely rare in practice. Figure 2: A generalized experimental methodology. The results achieved on the mixed datasets prove the high efficiency of the author's technique. The obtained accuracy of 76% for datasets containing three or more programming languages, and 87% for datasets consisting of language pairs, exceed the accuracy of some technique analogs [32; 33] that were obtained for simple identification cases. Three complex cases were addressed using artificially generated source code. The first and the simplest case was finding a difference between human and generative models (Figure 5). In this case, the evaluation was performed as follows: for each experiment (5, 10, and 20 authors), one of the authors was a generative model. One part of the experimental samples was written by humans, and another was generated by models of the GPT family: GPT-2, GPT-3, RuGPT-3 (from Sber AI). All samples were written in the most popular programming language--Java. The average accuracy for GPT-2 was 97%, and 94% for GPT-3 and RuGPT-3. The difference in accuracy can be explained by the fact that the third generation models show better generative ability than the second generation. Consequently, the separability of the author's model also decreases. Figure 4: Classification accuracy for different language combinations. Figure 3: Classification accuracy for language pairs.