Explainable machine learning in cybersecurity: A survey

International Journal of Intelligent SystemsVolume 37, Issue 12 p. 12305-12334 RESEARCH ARTICLEOpen Access Explainable machine learning in cybersecurity: A survey Feixue Yan, Corresponding Author Feixue Yan [email protected] orcid.org/0000-0002-1334-5644 School of Science, Computing and Engineering Technologies, Swinburne University of Technology, Melbourne, Victoria, Australia Distributed Systems Security, CSIRO's Data61, Sydney, New South Wales, Australia Correspondence Feixue Yan, School of Science, Computing and Engineering Technologies, Swinburne University of Technology, Melbourne 3122, Victoria, Australia. Email: [email protected]Search for more papers by this authorSheng Wen, Sheng Wen School of Science, Computing and Engineering Technologies, Swinburne University of Technology, Melbourne, Victoria, AustraliaSearch for more papers by this authorSurya Nepal, Surya Nepal Distributed Systems Security, CSIRO's Data61, Sydney, New South Wales, AustraliaSearch for more papers by this authorCecile Paris, Cecile Paris Knowledge Discovery and Management, CSIRO's Data61, Sydney, New South Wales, AustraliaSearch for more papers by this authorYang Xiang, Yang Xiang School of Science, Computing and Engineering Technologies, Swinburne University of Technology, Melbourne, Victoria, AustraliaSearch for more papers by this author Feixue Yan, Corresponding Author Feixue Yan [email protected] orcid.org/0000-0002-1334-5644 School of Science, Computing and Engineering Technologies, Swinburne University of Technology, Melbourne, Victoria, Australia Distributed Systems Security, CSIRO's Data61, Sydney, New South Wales, Australia Correspondence Feixue Yan, School of Science, Computing and Engineering Technologies, Swinburne University of Technology, Melbourne 3122, Victoria, Australia. Email: [email protected]Search for more papers by this authorSheng Wen, Sheng Wen School of Science, Computing and Engineering Technologies, Swinburne University of Technology, Melbourne, Victoria, AustraliaSearch for more papers by this authorSurya Nepal, Surya Nepal Distributed Systems Security, CSIRO's Data61, Sydney, New South Wales, AustraliaSearch for more papers by this authorCecile Paris, Cecile Paris Knowledge Discovery and Management, CSIRO's Data61, Sydney, New South Wales, AustraliaSearch for more papers by this authorYang Xiang, Yang Xiang School of Science, Computing and Engineering Technologies, Swinburne University of Technology, Melbourne, Victoria, AustraliaSearch for more papers by this author First published: 03 November 2022 https://doi.org/10.1002/int.23088AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Abstract Machine learning (ML) techniques are increasingly important in cybersecurity, as they can quickly analyse and identify different types of threats from millions of events. In spite of the increasing number of possible applications of ML, successful adoption of ML models in cybersecurity still highly relies on the explainability of those models that are used for making predictions. Explanations that support ML model outputs are crucial in cybersecurity-oriented ML applications because people need to get more information from the model than just binary output for analysis. The explainable models help ML developers solve the “trust” problem for a security application prediction in a faithful way: validating model behaviours, diagnosing misclassifications and sometimes automatically patching errors in the target models. Therefore, explainable ML for cybersecurity has become a necessary and important research branch. In this paper, we present the topic of explainable ML in cybersecurity through two general types of explanations: (1) ante hoc explanation, and (2) post hoc explanation, with their methodologies. We systematically review and categorise the state-of-the-art research, and provide comparative studies to help researchers find the optimal solutions to specific problems. We further list open issues in this field to facilitate future studies. This survey will benefit diverse groups of readers from both academia and industries, who want to effectively use ML to solve cybersecurity challenges. 1 INTRODUCTION Machine learning (ML) refers to a series of methods that data processing systems use to make and improve predictions.1 It is fundamental to recent advances in cybersecurity technology, and it has been successfully integrated into security-relevant systems, for example, malware detection,2, 3 intrusion detection4, 5 and vulnerability discovery.6, 7 Such ML processes in these systems map input data to predicted labels. Predictors can be linear models or neural networks with a set of weights.8 Without the need to explicitly program them and output results, ML techniques provide security applications with the capability to automatically learn based on past algorithms.9 Unfortunately, due to the lack of explanation, there are various problems in the security field: Problem 1: Digital applications suffer from cyberthreats, while the users—operators and designers—do not know where the threat comes from. The decision-making processes of the detection models are generally not interpretable to users.10 Problem 2: A gap exists between predictions and explanations; an ML model predicts without explanation and attempts to come to a conclusion without explanation. Unless explained, ML merely makes guesses.9 Problem 3: In contrast to other application domains, cybersecurity poses particular challenges to the need for explanation methods.11 The users tend to trust to use the security systems when explanations are given and corresponded with predictions rather than the system with only calculated predictions. Problem 4: A large number of learnt models in security applications contain prejudices and human biases.12 Data-based decision-making models can inherit these biases, and then the prediction from such models becomes unfair or even wrong. Explanations in security applications might demonstrate the underlying causalities in the model. To meet the needs of explainable ML, there is more and more intensive research on improving the capability of providing explanations of ML methods. It has been proposed that various explainable methods can help target users understand how they work. These methods decide if one should trust a prediction. We demonstrate “what the explanations in cybersecurity give”: validating learnt model behaviours2 for ML developers; identifying classification errors;13 and patching errors for the learnt model for safety.14 Moreover, the different explanations between various types of detection classifiers enable security developers to better design the security system structures. With more attention on their applications in cybersecurity, people have been thinking about questions: (1) which explainable methods are suitable for different security systems and (2) what the requirements do these systems need to meet? Explanations provide trust in ML methods, that exemplify the importance of explainable ML.15, 16 In recent years, several directions for explaining security systems have been explored. The cybersecurity experts have started to develop models for understanding ML in cybersecurity, for example, Android malware detection,2, 17, 18 portable document format (PDF) malware detection11, 14, 19 and discovering vulnerabilities in source code.6, 11 These explainable methods can help trace deep learning's prediction back to the input in security to help understand the decision-making process. In this survey, we introduce the procedure of how to build an explainable method and two general types of various explainable ML methods deployed in cybersecurity: ante hoc explanations and post hoc explanations. Further, we present each explanation approach under two categories in detail. We present the cons and pros of these approaches to help select methods on various real-world data sets. This study paves the way for implementing and trusting robust neural models accompanied by explanations for their predictions.20 Comparative studies are provided based on rigorous evaluation metrics to help researchers find the optimal explainable approaches for their work. Finally, we systematically collect and analyse open issues with an event timeline. This will help readers understand explanatory developments, milestones, practical implications and critical problems. The organisation of this paper is as follows. First, an overview of two types of explainable processing systems' methodologies is given in Section 2. We systematically review and categorise the technical background of each explainable learning method from state-of-the-art in Sections 3 and 4. Comparative studies are provided according to rigorous evaluation metrics in Section 5. The explanation summary for open issues is described in Section 6. Furthermore, Section 7 presents a discussion for future research. Finally, our conclusions are described in Section 8. 2 EXPLANATIONS IN CYBERSECURITY Role of explanations in cybersecurity A number of papers examined different types of security threats and tried to identify the causes,7, 21, 22 but there was no significant reduction in threats based on the results. Explanations play an important role in guiding users through the decision–making process and identifying the causes in the security domain. ML explanations in security systems provide people with a better understanding: (1) what is the security system model; (2) which features of an instance are malicious and why can it be attacked; (3) how to use a system securely.23 Explanations for ML have been explored with remarkable success in several application fields, for example, natural language processing (NLP)24 and speech recognition,25 while their applications have not received much attention in cybersecurity so far. Explainable ML in security has been applied in the aspects shown in Figure 1. This survey deals with the following types of applications in the security domain: 1. Malware detection: There are two kinds of malware detection systems that employ explanations: (1) Android malware detection. It uses ML models to identify Android malware by building features from application's codes and manifests, such as request permissions and suspicious API calls. Recent research on Android malware detection systems employing deep learning.26 Typical Android malware detection systems include Drebin2 and deep android malware detection (DAMD).3 (2) PDF malware detection. The PDF detection systems, for example, the Mimicus+ system, are designed to detect malicious PDF documents. Training is based on PDF document features, including document structure data.14 2. Intrusion detection: The intrusion detection, such as the Network Intrusion Detection System (NIDS), enables system operators to detect their systems' network security breaches. The data set is derived from NetFlow traces captured by the real network.19 One of the NIDSs applies Adversarial Auto Encoder4 to do task for Anomaly Detection. The NIDS encounters unpredictable attacks when building a detection model.27 3. Binary analysis: Binary reverse engineering hardens the security of software with a well-trained Recurrent Neural Networks (RNNs) model. Binary analysis helps security analysts do manual work such as detecting the function start.14 4. Vulnerability discovery: Recent attacks involve diverse vulnerabilities (e.g., cross-domain vulnerability28). There has been some research conducted on the application of ML techniques to assist in the detection of vulnerabilities.29 VulDeePecker is a discovery system using an RNN to analyse vulnerabilities in C/C++ code.6 The security of these systems refers to the security properties of their stakeholders (e.g., the developer, the client, the analyst and the defender). The security properties include confidence, trust and transparency. All stakeholders want security, which is the primary reason why explainable ML is vital to security systems. Explaining ML is used to explain predictions in these systems, such as for individual prediction of a black-box ML model and local boundary of the model in both a linear way and a nonlinear way. Figure 1Open in figure viewerPowerPoint Explainable ML for cybersecurity. Four types of applications in the security domain have applied explainable ML. ML, machine learning; NLP, natural language processing. Explanation setup in cybersecurity Explainable methods for ML in cybersecurity can be distinguished by whether the explanations are achieved by a self-explanatory learning model (ante hoc) or by applying methods to analyse an already trained model (post hoc). Ante hoc and post hoc explanations in cybersecurity use their respective design systems. The workflows of building explanations to users are shown in Figures 2 and 3. 1. Ante hoc explanation setup: As shown in the ante hoc explanation workflow in Figure 2, the explanatory models incorporate the modules that generate explanations into their architecture to provide explanations for their predictions. These models use different ML processes to train predictions associated with explainable models. The functions of these models have the capacity of predicting and are capable to explain why it makes that prediction as well. The explainable output displays additional information that can give insights into why such a prediction was made. Ante hoc explanations restrict the complexity of the model with the process of explainable modelling. It develops inherently more explainable models. Ante hoc explanations are applied to intrinsically interpretable modelling. 2. Post hoc explanation setup: Post hoc explanations refer to applying an explainable method to a target model, while this target model mostly is a black-box model. Post hoc explanations focus on interpretable methods (feature importance and partial dependency graph) after training.30 Post hoc explainable methods are used to extract explanations to describe predeveloped models, and they can also be seen as proxy models or surrogate models. Post hoc explanations can be deployed to explainable classifications.31 The post hoc explainable methods have a workflow with two interconnected modules shown in Figure 3: (1) the prediction module is dedicated to making predictions for the task at hand, and (2) the explanation module is dedicated to making predictions. The process of making an interpretation is a multi-task learning (MTL)32 process, which is able to combine different interpretation algorithms to make an interpretation at the same time. The explanation module does not influence the predictions made by the prediction module. Figure 2Open in figure viewerPowerPoint Ante hoc explanation workflow in cybersecurity: learned function outputs predictions and explanations for the input Figure 3Open in figure viewerPowerPoint Post hoc explanation workflow in cybersecurity: the learned classifier outputs predictions, and the joint algorithm acts as an explainable method and outputs explanations results Taxonomy of explanation in cybersecurity The taxonomy of current approaches is shown in Figure 4. We analyse each category of ante hoc and post hoc explanations. The two categories have different classification criteria, based on the structural characteristics of models33 and the target models.34 Figure 4Open in figure viewerPowerPoint Taxonomy of approaches to explainable cybersecurity: ante hoc explanations are classified by the type of explainers models; post hoc explanations are classified by the type of the target model. CAM, class activation mapping; CNN, convolutional neural network; GBM, gradient boosting machine; LEMNA, local explanation method using nonlinear approximation; LIME, local interpretable model-agnostic explanation; LRP, layerwise relevance propagation; NGBoost, natural gradient boosting; RNN, recurrent neural network; SHAP, SHapley Additive exPlanation; SVM, support vector machine; XGBoost, extreme gradient boosting. 1. Ante hoc explanation type: The ante hoc explanation models are algorithmic transparent with providing human-interpretable mechanisms.35 The decomposability of an ante hoc explanation model requires all parts of the model, including a model structure, model parameters, every input to the model, and every dimension feature that allows intuitive explanation. 2. Post hoc explanation type: In the applications of ML in cybersecurity, architectures of RNNs and convolutional neural networks (CNNs) are prevalent.36 These architectures of neural networks suffer from a severe deficiency: The neural networks are opaque, making them hard to interpret for practitioners. These predeveloped models are target the model that needs to be described by explanations. Other ML model or mixed neural network is the third type for target models. CNNs: In CNN, each neuron in a layer receives input from a restricted subsection of the previous layer.37 Layers with neighbourhoods in the inputs data form a strong spatial structure. In cybersecurity, CNN models can be used for malware detection from raw Dalvik bytecode of Android applications.3 RNNs: The network with loops in RNNs connects neurons from the output layer to the input layer. RNNs with a recurrent structure mainly operate sequential data38 and process information across predictions. RNN models have been used for security applications of sequential data (e.g., the discovery of vulnerabilities in software).6 Universal: Some approaches process mixed Neural Networks rather than typical CNNs or RNNs. The universal methods are not limited to the type of target models. This category is universal to CNN, RNN, mixed neural networks and ML models. The details, including security applications, pros and cons, of each explainable approach will be introduced in Sections 3 and 4. Properties of explanations in cybersecurity We look at the explanations' properties. The properties can be used to identify the suitability of an explanation method or to judge how good it is.39, 40 The explanations for predictions usually correlates the feature contributions of a single instance to its classification in a form that human easy understand. Besides, the explanations include a set of data samples. We can detect malware by support vector machine (SVM) models and explain malware by local proxy methods. Alternatively, we can use models equipped with an explanation module (e.g., a gradient boosting model). The properties of explainable approaches are as follows: 1. Global explanation: Global techniques allow us to better understand the entire model, which can include visualisations of the distribution of weights in neural networks and neural network layers propagation. 2. Local explanation: Local techniques focus on a smaller group of data that usually overlooked by global techniques. They are the cases looking at a single instance and providing explanations. 3. Model-agnostic: Model-agnostic explanations can be applied to any trained model. They usually examine the input features and output distribution of data. Recent techniques for model-agnostic explanations are about the black box that needs to be interpreted. 4. Model-specific: These techniques are only suitable for the use of a specific class of models. They generally focus on the algorithm or intermediate structure of the model. The explanations with intrinsically interpretable algorithms are model-specific. 5. Security: Explainers threaten the security of their target systems.41 Due to more attention to the explanation of the Deep Neural Network (DNN) model, the key factors for the classification model's decision can be identified, which can easily construct malicious adversarial samples to attack the classifier of the security system. Security issues, such as cyber attacks in Cyber-Physical Systems (CPSs),42 pose a huge threat to the systems. 6. Transparency: Explainable technologies are used to solve the trust problem for predictions with transparent methods. They help models to become more transparent. We usually use explainable approaches to solve the transparent box design problem. Sometimes, ML models have other important properties, such as robustness, reliability, causality, generality, and scalability. Models could maintain certain levels of performance. They are robust and reliable to small changes in the input data or parameters. They control the classification changes due to perturbations affecting the model predictions (causality). These properties behave differently when dealing with data sets or models30, 43 and are therefore not general enough and will not be discussed in this paper. 3 ANTE HOC EXPLANATION APPROACHES Ante hoc explanation deals with the inner working of models to interpret its results. It mainly refers to whether the model outputs are interpretable. Developing an explainer means it provides users with potential recommendations and even interventions,44 which improve their situation. Meanwhile, explainable ML systems provide comprehensive understanding. It means customers clear the behaviours of the model, how model outputs get translated into actions,45 and how to obtain the outcomes generated by the system. It also enables users to understand and know why a decision is made. These explainers are more interpretable than other ML models.20 For example, it is easier to understand the output of a linear model than the output of a neural network model. However, due to the fact that a mathematically straightforward model is more interpretable,46 these ante hoc explanation models require a more explainable feature engineering than others.47 Basically, before the model is run, human decisions derived from ante hoc explanation come into play. We divide these explainers into three types: linear model, tree-based model, and parametric model. Linear model 3.1.1 Linear regression Security application: Android malware detection and vulnerability discovery Linear regression models predict the instance as weighted feature inputs.48 The linearity relationship makes the explanation of learning models much easy. Computer experts have long used linear regression algorithms to solve quantitative issues. Method: Linear models can establish the dependence relation of target models y $y$ on the features x $x$ . They are models with linear relationships. The function of a linear model is as follows: y = β 0 + β 1 x 1 + β 2 x 2 + ⋯ + β n x n + ε . $y={\beta }_{0}+{\beta }_{1}{x}_{1}+{\beta }_{2}{x}_{2}+\cdots +{\beta }_{n}{x}_{n}+\varepsilon .$ (1) The predicted result of an instance is the weighted features. The β i ${\beta }_{i}$ represents the learnt weights of features. The first weight β 0 $\left({\beta }_{0}\right)$ is not multiplied with any feature. It can be seen as an intercept. The ε $\varepsilon $ follows a Gaussian distribution. It is a difference between predictions and real outcomes, like, an error. Advantages: The model makes decision-making more intuitive for humans as the model has weighted features. They are transparent to estimate the feature weights and get optimal weights theoretically. Plenty of applications have used linear regression models. It shows that linear regression models are accepted for prediction in a lot of places. There are collective experience and expertise at a high level. It has been implemented in software. Linear regression methods can be found in various computing languages, such as Python, R, Java, JavaScript, and so forth. Disadvantages: Since linear regression only represents linear relationships for the features of input, interaction and nonlinear relationships cannot use it. Besides, the predictive performance of linear models is often suboptimal because the learned relationships are restricted and often oversimplify the reality.49 3.1.2 SVM Security application: Android malware detection, intrusion detection and binary analysis SVMs utilise the so-called support vectors from the training samples to look for the decision boundary.43 It finds a hyperplane with a maximum margin around the SVM decision boundary with a set of different kernels.50 Method: SVM in security only use its one method, linear SVM. We get a training data set with n $n$ data points. The hyperplane of SVM can be regarded as the set of data points, where these data points x → $\overrightarrow{x}$ satisfy w → ⋅ x → = − b , $\overrightarrow{w}\cdot \overrightarrow{x}=-b,$ (2) where w → $\overrightarrow{w}$ is the (not necessarily normalised) normal vector of the decision hyperplane and b $b$ is an intercept term. w $w$ and b $b$ pair to the training points x $x$ . The application of SVM in Drebin is shown in Figure 5. By virtue of the simple detection function of the linear SVM, it determines the contribution of each feature.2 The linear SVM provides an explanation of detection from the perspective of malware families. Figure 5Open in figure viewerPowerPoint A separating hyperplane with maximum margin (SVM) on the malware detection system. Constructing an input x $x$ as a vector φ ( x ) $\varphi (x)$ . Mapping the input to a vector space to see its classification (benign or malware) from the hyperplane. SVM, support vector machine. Advantages: SVM is a linear margin-maximising classifier. It works well when there are clear margins of separated classes in various real-world problems. SVM application can tremendously reduce the need for training instances with labelled data in the standard inductive settings.51 It is easy to have the importance of each feature from linear SVM. This is what convinces the users the most because it is the model's own explanation. Disadvantages: In security, classifiers required high accuracy, while linear SVM is not as accurate as nonlinear SVM. DREBIN is an efficient and explainable malware detection using SVM.2 The successful rate reaches 97%. But the SVM is not suitable for large data sets. Besides, if the number of features for each instance exceeds the number of training samples, the performance of the SVM algorithm will decrease.52 Tree-based model 3.2.1 Gradient boosting machine (GBM) Security application: Android malware detection and vulnerability discovery GBM is a basic tree-based method working on boosting technique. Each new tree in GBM can be used to modify the original data set.53 GBM trains the models basically in a gradual and additive manner. Method: In gradient boosting, the loss function can be minimised by gradient descent, which indicates how good the coefficients of the model reflect the data. What is to be optimised determines the information of explanations. The design idea is to iteratively build the model in sequence, with each updated model reducing errors based on the previous model until the loss is negligible.54 Advantages: GBM is a method that is easy to read. The prediction capability is efficient through the use of its clone methods. GBM makes its explanations on predictions easy to control as well. Disadvantages: Gradient boosting models are continuously improved to minimise all errors. This can lead to overfitting by focusing too much on outliers. Also, GBM has to perform grid search and can only test limited values.55 3.2.2 Extreme gradient boosting (XGBoost) Security application: Android malware detection and vulnerability discovery XGBoost is one of the tree-based methods. It is known as regularised version of GBM.56 XGBoost is an efficient and easy-to-use ML algorithm. Compared with other algorithms, it delivers high performance and accuracy. Method: Objective functions in XGBoost contain two parts: (1) training loss and (2) regularisation. To train this model, it needs to define the objective function to measure how well the model fits the training data. One commonly used loss function is the logistic loss. For the regularisation term, it controls the complexity of the model, so that it can help us avoid overfitting. Advantages: XGBoost prevents this model from overfitting with in-built Lasso Regression (L1) and Ridge Regression (L2) regularisation. Users are allowed to run cross-validation at each iteration in the boosting process.57 In this way, it is easy to get the exact optimal boosting iterations number. Besides, it is much faster than GBM because XGBoost utilises the power of parallel processing. It can use multiple central processing unit cores when executing a model. XGBoost supports the following main interfaces: CLI, C++, Python, Julia, R, Java, and so forth. Disadvantages: Although XGBoost is fast to some extent, it does not perform well in large data sets as it is time-consuming for training. Once XGBoost runs failed, it takes quite a while. What is more, people have to manually create label encoding for categorical features before feeding them into the models. 3.2.3 Natural gradient boosting (NGBoost) Security application: Android malware detection and vulnerability discovery NGBoost is a general probabilistic prediction algorithm with explainability.58 This is a method of allowing the prediction of uncertainty estimates, rather than point estimates of typical regression models. Method: By targeting the parameters of the conditional distribution, NGBoost generalises gradient boosting on probabilistic regression. The algorithm has three modular components