# Power-Based Side-Channel Attacks on Program Control Flow with Machine Learning Models
## 4 Results  
For each firmware, with the exception of firmware where a single execution is performed (\(w=1\)), classification accuracy values approaching 100% are observed for the random forest classifiers. Accuracy values of 98%+ are seen for decision trees and logistic regression. The KNN classifier is the outlier with an observed lower bound on accuracy values that was slightly greater than 80%. Firmware with only a single execution of the addition operation (\(w=1\)) was the exception to these metrics of accuracy. The resulting skew in overall performance is illustrated in Figure 4 while the exact performance of all four classifiers on each of the folds of testing is illustrated in Figure 5. The highest observed accuracy for this firmware (\(w=1\)) was associated with a single fold of validation and the KNN classifier; it was only able to achieve a maximum accuracy of 3.26%. While this is nearly an order of magnitude more accurate than randomly guessing the class, it is far from a desirable accuracy. Preprocessing, ensemble classification, and hyperparameter optimization would be relevant approaches to addressing this concern if the classification of this firmware were the primary goal; however, as the goal is characterization of the bounds of potential classification, this is left for future work. Therefore, we can conclude that when the amount of work performed in each branch of control flow (i.e., when the amount of work performed by each state) is low, \"off-the-shelf\" machine learning models will struggle to determine the underlying program execution flow. In stark contrast to the classification accuracy of traces obtained from this firmware (\(w=1\)), as the \(w\) value of the firmware increases, the performance capabilities of simple machine learning classifiers are well suited to the classification task set before them. For firmware with \(w\) values of 2, 4, and 8, a random forest classifier was able to correctly recover the program execution flow in all five folds of cross-validation with 100% accuracy. For algorithm specific and cross-fold specific performances on firmware with two executions of addition operations (\(w=2\)), see Figure 6. While all other classifiers achieved high levels of accuracy on average (99%+) over the same firmware, only the random forest classifier achieved this level of performance. Across all classifiers, a decrease in overall accuracy was observed moving from firmware with 8 executions (\(w=8\)) to firmware with 16 (\(w=16\)). In the case of random forest classifiers, this is a decrease from an average accuracy of 100% to 99.92%. Both the decision tree classifiers and logistic regression had their accuracy decrease as the number of operations performed by the firmware increased, but both remained over 98% classification accuracy on average. The KNN classifier saw vastly diminishing performance in the move to firmware with more operations, only achieving a maximum accuracy of 81.4% with an average accuracy of 80.8%. Similar decreases in performance were observed when moving to the next level of firmware (\(w=32\)). A visual presentation of the average performance across these various firmware with more than two operations is available in Figure 7.  
Figure 4: A figure which demonstrates the variance, or lack of variance, exhibited by each classification algorithm depending upon the number of operations performed by the firmware. The y-axis is split to emphasize the difference between \(w=1\) and the other \(w\geq 2\). Figure 5: A heatmap demonstrating the accuracy of the testing phase for firmware with a single operation in each state (\(w=1\)). Most notable is that, while results are better than random guessing, classification accuracy of 2% is extremely different than the 98%+ accuracy achieved for all other tested firmware.