dataesr/scientific-paragraphs-categorization huggingface dataset:
With the extensive application of deep learning (DL) algorithms in recent years, e. g. , for detecting Android malware or vulnerable source code, artificial intelligence (AI) and machine learning (ML) are increasingly becoming essential in the development of cybersecurity solutions. However, sharing the same fundamental limitation with other DL application domains, such as computer vision (CV) and natural language processing (NLP), AI-based cybersecurity solutions are incapable of justifying the results (ranging from detection and prediction to reasoning and decision-making) and making them understandable to humans. Consequently, explainable AI (XAI) has emerged as a paramount topic addressing the related challenges of making AI models explainable or interpretable to human users. It is particularly relevant in cybersecurity domain, in that XAI may allow security operators, who are overwhelmed with tens of thousands of security alerts per day (most of which are false positives), to better assess the potential threats and reduce alert fatigue.
Taxonomy AI models are a major actor in the cybersecurity research landscape. However, ensuring the proper use of AI models in a cybersecurity context is an arduous task. Shaukat et al. [111]  provide a broad review on the applications of ML techniques to cybersecurity. Cybersecurity applications of AI encompass network security, computer security, mobile security, etc. In essence, AI and XAI methods have been implemented on various datasets corresponding to the research trends. As such, we believe that discussing the broadest range of applications will give the reader a diversified vision about the landscape of XAI in a cybersecurity context. Therefore, we argue that transparency of and trust in AI also belong to the scope of cybersecurity as they contribute to reducing the potential maliciousness toward the AI model and the system in general.
Hagras  [46]  discussed the link between humanunderstandable information and the flexibility of the data labeling process. Paredes et al. [94]  discussed explanations for cybersecurity and insisted that explanations should be able to capture changes in an attacker's strategy, or to help identify anomalies when they are outlined by detection mechanisms. Kuppa et al. [66]  proposed a taxonomy for XAI concerning its security properties. They also demonstrated a novel black-box attack on explainable models and evaluated it on three datasets. The proposed taxonomy covered three domains: the explanations of predictions made by a model, the security properties associated with models (i. e. , confidentiality, integrity, and privacy), and the threat models used. The authors differentiated confidentiality and privacy by highlighting that the former pertained to the features of data, while the latter pertained to the explanations given to various security actors.
We employed a different approach by considering the intersection of XAI and cybersecurity (Table  1 ). In this survey, we explored both methods for explaining AI-based cybersecurity applications and security analyses of XAI methods. In the first case, the literature we surveyed covers various practical scenarios, mostly supported by cybersecurity datasets (23 papers). In the second case, we identified several properties with respect to the security of XAI that were discussed in the state of the art (27 papers). Our taxonomy differs from those of existing XAI surveys, as most of them considered XAI from an intrinsic perspective. Numerous explanation methods were not attacked; however, the security properties presented in this survey should be relevant for these methods as well. We do not mention these methods in this survey to preserve our cybersecurity perspective.
Partial dependence plot (PDP) arg min x � max L(x, x � , y � , ) f (x) = K ∑ j=1 j ( j. x + j ) where f denotes the partial function, x s represents the input feature that is going to be plotted, and x C consists of other features used in the model. [36]  explains the influence of a feature on the prediction result of an AI model on average. The concept of ALE was introduced to address the main limitation of a PDP: its fidelity level reduces drastically if the features in the AI model are correlated. ALE show the variation of model prediction in a small area where the analyzed input is located. [90]  is an explainable method for generating a surrogate model by approximating the prediction result and the interpretability of the underlying explainability model. First, a dataset is selected (it can be the same dataset that was used to train the underlying model or a new dataset).
Explaining with surrogate models Islam et al. [54]  proposed a semantic approach assigning confidentiality, integrity, and availability (C, I, A) meanings to attacks and features in a dataset. First, they designed a feature generator, where the three most important features for each attack are extracted. New features were then created by assigning a C, I, or A attribute to the ones they extracted. In addition, they computed a coefficient that illustrated the weight of the feature in the overall impact of C, I, or A. Second, they implemented an evaluator that ran multiple attack detection models and measured the impacts of different sets of features, including the previously generated ones. They used the CIC-IDS2017 dataset  [110]. Due to resource limitations, the dataset could not be entirely exploited, thereby limiting the scope of validity of their results.
Similarly,  [123]  proposed a hybrid oracle-explainer approach to develop an explainable intrusion detection system (IDS), which combined an opaque classifier based on an artificial neural network (ANN) and an interpretable module using DT. During the inference phase, the ANN classification decision of a given input is explained by the DT's decision on the same input, or the closest input in terms of the l 2 (Euclidean) norm. Guo et al. [45]  proposed LEMNA, an explanation framework for malware detection applications. LEMNA is the sole explanation method designed specifically for cybersecurity applications. They used a the PDF malware dataset from Smutz et al. [120]. LEMNA attempts to approximate a local area within a DL decision boundary using an interpretable model. This model is designed to handle feature dependency and nonlinear local boundaries.
They experimented with the KDD Cup 1999 dataset (intrusion detection) from the UCI ML archive, revealing that SHAP outperformed LIME on reducing the reconstruction error. SHAP was also used in [106] for explaining and interpreting classification decisions of an ML-based network IDS. Two classifiers-a deep feedforward NN and an RF-were evaluated on several recent intrusion detection databases, namely CIC-IDS2018  [22] , TON_IoT [87], and BoT-IoT  [63]. Two feature sets of each database were considered: one set contained 83 features extracted via CICFlowMeter  [71] , while the other contained 43 features extracted from NetFlow  [107]. The evaluations were focused on finding the most interesting features for each classifier. The results of explainability exhibited some similarities between the two classifiers with respect to the most influential features in different databases.
The authors also noted that the influence and importance of each network feature varied with the dataset. Alenezi et al. [8]  designed an explainable ML framework for malware and malicious URL detection. They implemented three XAI methods based on SHAP: TreeExplainer, KernalExplainer, and DeepExplainer. These XAI methods explained the prediction of common classifiers, such as RF and XGboost. They used a URL dataset, ISCX-URL2016 [81], and an Android malware dataset, CICMalDroid 2020 [80], both published by the Canadian Institute for Cybersecurity. They compared the performance of each method using different setups. The results showed no optimal universal setup for any one scenario. Khan et al. [58]  designed a timely detector for attack vectors on the Internet of Medical Things networks. The model was developed using bidirectional simple recurrent units.
The detector uses the phenomenon of skip connections to solve the vanishing gradient problem and reduce the training time. The study used the TON_IoT dataset [87], which contains several cyberattack classes, such as ransomware, backdoor, denial-of-service, distributed denial-of-service, Man-in-the-Middle, injection, cross-site scripting (XSS), and scanning attack. Their experiment demonstrated that the proposed model outperformed the long short-term memory and gated recurrent unit models at a lower computational cost. They also used LIME to investigate the contribution of each feature in the prediction phase. Zebin et al. proposed an RF-based IDS to detect DNSover-HTTPS (DoH) attacks  [124]. Precisely, they used SHAP to explain the results by determining the most important features. The proposed method was evaluated with the CIRA-CIC-DoHBrw-2020 [86] dataset, where the traffic is described by 29 features.
The explainable variables were the marginal contributions associated with each feature and are calculated using the Shapley-Lorenz decomposition method. The results showed that Shapley-Lorenz values were significantly easier to be interpreted than the Shapley values that were not normalized. Iadarola et al. [53]  introduced a framework for Android malware detection using an explanation method for image recognition. They tried to address the weakness of signaturebased anti-malware detection, which cannot detect zero-day malware. First, a binary executable dataset was converted into images, and then a CNN was used for training. In the explainability steps, they generated heatmaps with Grad-CAM [108] and classified a subset of the test samples to the corresponding class of malware. They evaluated 8,446 Android malware samples from 6 malware families and obtained an average accuracy of 0.
Mahbooba et al. [79] aimed at addressing the trust issue between users and ML models for IDS. They highlighted that most previous studies focused on the accuracy of classifiers without providing any insight into their reasoning or behavior. They used the DT algorithm on the KDD99 dataset  [126]. In summary, their methodology comprised three main steps: feature ranking, DT rule extraction, and comparison with state-of-the-art algorithms. They described the feature on each branch and the threshold value to explain how the tree made decisions. However, their algorithm was vulnerable to overfitting amid noise in the dataset. Wang et al. [133]  introduced TrafficAV, an explainable mobile malware detector. They captured the network traffic of a mobile device and investigated it for suspicious activity.
Explanations for improving the performance of classifiers In this subsection, we discuss recent works about using XAI to improve the performance of classifiers. The majority of latest publications in the field of AI have focused on improving accuracy, detection rate, and F 1 score, while reducing the false alarm rate. Improving the model's performance via parameter tuning in a heuristic manner is computationally taxing. Consequently, XAI is deemed capable of increasing AI's performance in an explainable way. We highlight three use cases: 1. Explainable IDS, 2. Side channel attack detection, and 3. Anomaly detection. Khan et al. [131]  leveraged global explanation on Tree-SHAP to correlate an RF decision for explainable IDS. Their explainable IDS architecture comprised three main modules: (1) RF classifier (RFC) module for security predictions; (2) SHAP module extracting values relative to each feature of the dataset, and representing the importance of each feature in the decision made by the RFC; (3) Credibility assessment module (CAM), which utilizes the prediction and Shapley values to evaluate the confidence of prediction made by the RFC.
In particular, the CAM module evaluates the plausibility of the second-most probable prediction over the confidence expressed by the Shapley values of the most probable prediction computed by the RFC module. In the case of divergence, other classifiers were used to reassess the decision. The CIC-IDS2018 dataset  [22]  was finally used to compare the proposed IDS with other state-of-theart classifiers. Adversarial attacks were also employed to evaluate the robustness of the proposed IDS, which outperformed a native version of the RFC. Gulmezoglu  [43]  proposed an XAI-based framework using side-channel analysis against website fingerprinting attacks. The framework detects side-channel attacks by discovering the most dominant features extracted from a dataset using CNN and RF. During the training phase, they used a self-generated side-channel attack dataset collected from the Google Chrome and Firefox browser developer tools.
After the DL model was trained, LIME and saliency maps were used to examine the most dominant features of the website fingerprinting attack. They also verified the robustness of the framework. After perturbing the data points, a new model was trained and tested with the perturbed dataset. Their experiment revealed a drop of 16% in the attack rate, with five times less performance overhead. Subsequently, they generated adversarial noise to anticipate further attacks. Roshan et al. [102]  proposed the application of KernelS-HAP to reconstruct the errors of an AE to select the best features in an anomaly detection dataset. To understand and interpret the roles of features in improving AE-based anomaly detection, they compared three cases:  (1) ] no feature selection (all features); (2) feature selection using unsupervised feature correlation; (3) feature selection using SHAP.
Explaining errors A few studies have explained, using XAI, why a security model would make a mistake. Marino et al. [82] used adversarial examples to explain the importance of each feature on the decision made by gradient-based classifiers. They examined a set of misclassified samples and found the minimum amount of modification required to rectify the classification. They compared two models, an LR and a multilayer perceptron  [101]. They used the NSL-KDD dataset  [126]. However, the dataset was published in 2009. This limitation does not influence the usability of the proposed method; however, it does challenge the validity of the results for current threats. Although the study allowed determining the key features in the model, it did not provide interpretation guidelines about the meanings of the features.
Fan et al. [32]  used SHAP for feature attribution computation in an Android malware detection system. The main purpose of their study was to maintain model classification performance over time. They stated that the change in performance was difficult to understand when they updated the model. SHAP was used to interpret the output of the model by assigning Shapley values for each feature. The prediction change was analyzed by comparing the pattern changes of feature attribution. First, the feature attribution of each sample before and after the update was collected. Second, the changes in feature attributions were clustered, obtaining the pattern of changes. The experiment showed that the method successfully prevented overfitting and ineffective updates. Liu et al. [76] introduced FAIXID, a framework that provides data cleaning and XAI for IDS.
Unlike other studies, they offered a data cleaning method to address the data quality problem. The cleaning process solved the issue of "the data we want and the data we have. " Their focus was to propose several data cleaning techniques, instead of the explainability method. However, their method was designed specifically for a homogenous set of features-if the dataset has different features, more adjustments will be needed. Farrugia et al. [34]  proposed the usage of XAI in an application for cyber-fraud detection, with the goal of achieving a fully autonomous prescriptive solution for explainable cyber-fraud detection within the iGaming industry. The application of XAI in this context allowed the authors to minimize the adverse effects of incorrect predictions. A private dataset with labeled instances of verified fraudulent players made by the Gaming Innovation Group was used.
However, this manually labeled dataset was also the main limitation. In their method, they trained different models (RF, LGB, DT, and LR) on the dataset; they used stratified 10-fold cross-validation and compared the models using the area under the curve (AUC). Next, they extracted explanations for every individual prediction. Finally, they empirically evaluated data drift and suggested retraining the model every month as the drift rate was approximately one month. They compared each model using the standard metrics (AUC, precision, time, recall, and F1), where RF obtained the most consistent results.
Discussion Despite not belonging to a cybersecurity use case, the merits of the following study deserves the reader's attention. Aguilar et al. [4]  proposed an explainable AE using DTs for anomaly detection that deals with categorical data. Particularly, categorical attributes were one-hot encoded, and each attribute was used to construct a DT, which output the predicted attribute by considering other attributes. A final prediction layer was employed to determine whether the data point was an anomaly. As such, each DT explained one attribute, explaining the final decision through a set of DT rules. Twenty-eight datasets from the UCI ML Repository  [14]  were used for the experiments. Especially, datasets with two balanced classes or multiple imbalanced classes were divided using distribution optimally balanced stratified cross-validation  [136].
The resulting datasets had one majority class and one minority class for anomaly detection. The performance of the AE was compared with six other classifiers in terms of AUC and average precision score. The proposed solution ranked third per median value for AUC, but first for average precision. The strength of their study is the simplicity with which explanations were extracted from the trees while demonstrating them through synthetic examples. However, this system had limited scalability to datasets with less than a thousand attributes or with categorical attributes having tens of different values. As listed in Table  2 , DTs have already been used in cybersecurity tasks, and the performance of this DT-based AE approach is worth investigating.
g. , with LIME). This attack could be detected by exhibiting inconsistencies between explanations; however, the authors showed that these inconsistencies were difficult to obtain in practice, at least when the search was random. Diamnov et al. [27]  proposed a method that could modify a pretrained model to manipulate the output of feature importance explanation methods. They assumed that the explanation methods used in real-world settings could not indicate the fairness of a model. To prove their hypothesis, they chose several datasets from the UCI ML Repository  [14] , which contains data with sensitive features, such as sex or race. They optimized an existing model with a modified loss objective function. After the attack, the feature importance computed by the explanation method was completely modified.
Aivodji et al. [7]  investigated the rationalization problem and the associated risk of fairwashing. Subsequently, they introduced LaundryML, an algorithm that enumerated the optimal model according to fidelity and unfairness. They considered fairwashing when the fidelity of the new model is high while the unfairness is significantly lower. They used two datasets known for their biased nature-Adult Income  [62]  and ProPublica Recidivism  [55]. They used the fidelity and unfairness metrics to evaluate the performance of LaundryML. The results obtained using a real-world dataset demonstrated the feasibility of the proposed approach. Hence, all those attacks on fairness are fairwashing, where the model owner attempts to deceive auditors into believing their model was fair. However, fairwashing is not the only type of attack that can be orchestrated on fairness.
Their experiments were performed on the Dog vs. Cat dataset  [31]  and the UIUC sports event dataset (Event8)  [73]. Two methods of XAI (layered Grad-CAM and guided Grad-CAM) were used to detect adversarial attacks. However, the authors showed that the attack had no discernible impact on the interpretations produced by XAI to someone who is unaware of the attack, as the change was not detectable. Thus, relying on the explainability of results to detect the existence of adversarial attacks is not a rigorous approach. Kuppa et al. [66]  proposed a black-box attack on explainable models and evaluated the attack on three datasets. The attack had two possible targets: either the classifier alone or the classifier and interpreter. The attack focused on gradientbased classifiers (i. e. , NNs). The purpose of the attack was to identify relevant perturbations in an adversarial sample to compromise the system, while maintaining the explanation of the sample.
The authors presented three cybersecurityrelated scenarios where the classifiers were variations of the perceptron. These scenarios targeted a perceptron tasked with detecting malicious PDF documents  [45] , a perceptron for Android malware classification  [12] , and an IDS using adversarial AEs  [67]. They implemented these attacks using the datasets proposed by the original authors for each scenario, which reinforced the validity and reusability of Kuppa et al. 's  [66]  results. Kuppa et al. [65]  proposed a general formalization of security problems in XAI. First, they highlighted the motivation of their work by describing five real-life use cases: i) membership inference attacks (which involved determining if a sample belonged to the training data), ii) model extraction attacks (which aimed at retrieving the actual parameters used in a model), iii) poisoning attacks (which targeted the classification performance by corrupting training data), iv) adversarial examples (which aimed to evade security classifiers), and v) counterfactual explanations (the goal of which was to find the input data points that shared similarities with other entry points but did not yield the same classification results).
Although counterfactual explanations were similar to adversarial examples, they differed in purpose; the latter aimed to evade security classifiers, whereas the former facilitated understanding of a model by providing explanations for it. The authors mathematically defined the attack model for these use cases as well as the explanation methods. They exploited three datasets: leaked passwords from multiple incidents, CIC-IDS2017 [110] for network traffic, and a malware dataset collected from Virusshare  [129] , combined with benign software collected from various sources. They evaluated their implementation against multiple commercial tools and demonstrated the attacks' ability to evade antivirus solutions. Cantareira et al. [20]  proposed a method for investigating models subjected to adversarial examples.
The method used the visual analytic framework to explain adversarial attacks. This method explored layers and weights inside a model to determine which areas were triggered by adversarial examples and allowed to compare data from training data and adversarial examples. The method is as follows. First, it trains two models with different datasets-small CNN for MNIST dataset and MobileNet V2 for ImageNet. Second, it generates adversarial data using projected gradient descent, while selecting random data from the training sample as the background data, to create an adversarial set. Third, it selects an image from the baseline, runs it through the models, and projects its output on a view with the adversarial counterpart. With this, we observed the change in their behavior at each layer of the models, though the effectiveness of the view depends on how representative the background data is.