# Early Ransomware Detection with Deep Learning Models
## 5 Experimental Evaluation
### Results  
#### 5.5.1 Baselines  
Baseline results for traditional models (RF and SVM) on PARSEC-500 and PARSEC-5000 datasets appear in Table 2. For these models, we used 1-hot encoding of text features and window size W = 1, and the Ops list of data features. The RF model showcases remarkable efficiency with low test times on PARSEC-5000 dataset, indicating its scalability. It also provides much better scores on this dataset than SVM. The SVM model, despite the adjustment of the maximal iterations number that we performed, incurs significantly higher test times. This result implies that SVM is not suitable for early detection, because the system must respond quickly to a threat.  
We evaluated our main baseline, MLP, on all feature sets, all text representations, and all window sizes (W \(=1,3,5,7\)). Table 3 contains the best results for all feature sets on the PARSEC-500 and PARSEC-5000 datasets (full results are shown in Appendix A.5 of the Appendix).  
MLP achieved higher scores than traditional baselines on all datasets, showing that neural networks are more suitable for our domain. We observed, however, that adding more API call features did not necessarily improve the results. The MLP model exhibited much lower test times compared to the RF model in all cases, indicating that it is more suitable for the task of early detection. The results also reveal that the best results were achieved for W = 7. However, not all feature sets and text representations (such as OpResDurDet) were feasible for training the MLP model in a reasonable time.  
\begin{table}
\begin{tabular}{l l l l l l|r r} \hline \hline  & \multicolumn{6}{c}{PARSEC-500} \\ \hline model & text repr & FLIST & sensitivity & specificity & F1 & test time (s) & W \\ \hline RF & 1-hot & Op & 0.9930 & 0.6484 & 0.8518 & 2.07 & 1 \\ SVM & 1-hot & Op & 0.9930 & 0.6484 & 0.8518 & 42.49 & 1 \\ RF & 1-hot & OpRes & 0.9897 & 0.6826 & 0.8624 & 2.85 & 1 \\ SVM & 1-hot & OpRes & 0.9897 & 0.6826 & 0.8624 & 48.30 & 1 \\ \hline \multicolumn{6}{c}{PARSEC-5000} \\ \hline model & text repr & FLIST & sensitivity & specificity & F1 & test time (s) & W \\ \hline RF & 1-hot & Op & 0.9208 & 0.6658 & 0.8119 & 0.79 & 1 \\ SVM & 1-hot & Op & 0.4128 & 0.8197 & 0.5160 & 444.72 & 1 \\ RF & 1-hot & OpRes & 0.9144 & 0.6719 & 0.8108 & 0.92 & 1 \\ SVM & 1-hot & OpRes & 0.4064 & 0.8256 & 0.5119 & 532.08 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 2: SVM and RF F1 scores on PARSEC-500 and PARSEC-5000 datasets (the best scores are marked in gray).  
\begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline  & \multicolumn{6}{c}{PARSEC-500} \\ \hline model & text repr & FLIST & sensitivity & specificity & F1 & test time (s) & W \\ \hline MLP & BERT SE & Op & 0.9935 & 0.9664 & 0.9802 & 0.03 & 7 \\ MLP & BERT SE & OpRes & 0.9967 & 0.9372 & 0.9679 & 0.13 & 7 \\ MLP & BERT SE & OpResDur & 0.9805 & 0.9372 & 0.9597 & 0.09 & 7 \\ MLP & BERT SE & OpResDurDet & 0.9827 & 0.9317 & 0.9583 & 0.47 & 7 \\ \hline \multicolumn{6}{c}{PARSEC-5000} \\ \hline model & text repr & FLIST & sensitivity & specificity & F1 & test time (s) & W \\ \hline MLP & 1-hot & Op & 0.9989 & 0.9801 & 0.9896 & 0.04 & 7 \\ MLP & 1-hot & OpRes & 0.9999 & 0.9849 & 0.9925 & 0.05 & 7 \\ MLP & FastText & OpResDur & 0.9583 & 0.9490 & 0.9539 & 0.38 & 7 \\ \hline \hline \end{tabular}
\end{table}
Table 3: MLP scores on PARSEC-500 and PARSEC-5000 datasets (the best scores are marked in gray).