# Image-Based Feature Representation for Insider Threat Classification
## 4 Implementation
### Experimental Results  
The evaluation is performed using the CMU CERT v4.2 data. The features are represented as 1D feature vectors and images, as described in Section 3. Random sampling is performed on the data to overcome the class imbalance problem. Various sampling ratios are used to perform the experiments. We tested on various train-test split ratios. The dataset is split to training and testing before feeding it to the deep learning model. Accuracy, precision, recall, and F-score are used to compare the results.  
DNN is used for classification of 1D feature vectors. The DNN consists of 3 hidden layers and a final a classification layer which uses the sigmoid function. A dropout layer is added before the classification layer to reduce the effect of overfitting. Table 5 gives the overall performance of the DNN model for different train-test split of the dataset on the feature vectors. In the table, A, P, F, and R represent accuracy, precision, F1-score, and recall, respectively.  
DCNN is used for classification of grayscale images. The weights used in the original pre-trained models are applied but with varying hyper-parameters. Table 6 gives the overall performance of the model for different train-test split of the dataset on the grayscale images. The results show that VGG19 performs better than the other two models.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Non-Malicious Instances** & **Malicious Instances** & **Undersampling Ratio** \\ \hline
24,150 & 966 & 25 \\
19,320 & 966 & 20 \\
14,490 & 966 & 15 \\
9660 & 966 & 10 \\
4830 & 966 & 5 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Sampling ratio.  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{
\begin{tabular}{c} **Sample** \\ **Ratio** \\ \end{tabular} } & \multicolumn{6}{c}{**Training \% (70)**} & \multicolumn{6}{c}{**Training \% (80)**} \\ \cline{3-10}  & & **A** & **P** & **F** & **R** & **A** & **P** & **F** & **R** \\ \hline \multirow{4}{*}{**DNN**} & 5 & 90.86 & 76.04 & 73.37 & 70.88 & 91.72 & 79.62 & 72.25 & 66.12 \\ \cline{2-10}  & 10 & 93.46 & 64.32 & 64.81 & 65.31 & 93.56 & 70.24 & 63.27 & 57.57 \\ \cline{1-1} \cline{2-10}  & 20 & 95.57 & 53.65 & 52.76 & 59.87 & 95.88 & 67.34 & 57.39 & 45.49 \\ \cline{1-1} \cline{2-10}  & 25 & 96.24 & 55.10 & 22.69 & 14.29 & **96.34** & **62.96** & **36.30** & **25.50** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance evaluation on the benchmark dataset using 1D feature vectors.  
Tables 5 and 6 show the performance achieved for the 1D feature vectors and the grayscale images. Though the accuracy of the DNN on the 1D feature vectors is almost same as that of DCNN on image-based analysis, the precision and recall deteriorate for DNN. DCNN gives much improved precision and recall. Hence, the image-based analysis proved to be more promising. The proposed approach is compared with state-of-the-art approaches. Random forest method proposed by [PERSON] [13] gave 94% accuracy with randomization and 90% accuracy with normal Random Forest. Several works use LSTM [14; 16] for feature learning, followed by classification algorithms, like CNN. Another approach used deep learning for feature learning and then applied one class SVM [15] for classification. Our proposed approach gave an improved AUC = 97.38 when compared to the method using LSTM [16] gave an AUC = 94.49. Since various papers use different metrics for performance evaluation, we chose the papers with accuracy for the comparison, and those use the CERT CMU v4.2. The difference in the number of features and the representation used, methods applied for class imbalance handling, performance metrics applied, etc., affect the results of each method. Still, we tried to compare the proposed approach with the existing deep learning methods and the CMU CERT dataset. Table 7 shows the performance of the proposed method compared to existing approaches.  
In the next section, we provide the discussion of results and conclude the paper with possible directions for future work.