Title: Interpreting Unsupervised Anomaly Detection in Security via Rule Extraction
Venue: NeurIPS 2023
Accepted: yes
Keywords: unsupervised anomaly detection

global explanation

rule extraction
Abstract: Many security applications require unsupervised anomaly detection, as malicious data are extremely rare and often only unlabeled normal data are available for training (i.e., zero-positive). However, security operators are concerned about the high stakes of trusting black-box models due to their lack of interpretability. In this paper, we propose a post-hoc method to globally explain a black-box unsupervised anomaly detection model via rule extraction.
First, we propose the concept of distribution decomposition rules that decompose the complex distribution of normal data into multiple compositional distributions. To find such rules, we design an unsupervised Interior Clustering Tree that incorporates the model prediction into the splitting criteria. Then, we propose the Compositional Boundary Exploration (CBE) algorithm to obtain the boundary inference rules that estimate the decision boundary of the original model on each compositional distribution. By merging these two types of rules into a rule set, we can present the inferential process of the unsupervised black-box model in a human-understandable way, and build a surrogate rule-based model for online deployment at the same time. 
We conduct comprehensive experiments on the explanation of four distinct unsupervised anomaly detection models on various real-world datasets. The evaluation shows that our method outperforms existing methods in terms of diverse metrics including fidelity, correctness and robustness.

Interpreting Unsupervised Anomaly Detection in Security via Rule Extraction

Abstract

Many security applications require unsupervised anomaly detection, as malicious data are extremely rare and often only unlabeled normal data are available for training (i.e., zero-positive). However, security operators are concerned about the high stakes of trusting black-box models due to their lack of interpretability. In this paper, we propose a post-hoc method to globally explain a black-box unsupervised anomaly detection model via rule extraction. First, we propose the concept of distribution decomposition rules that decompose the complex distribution of normal data into multiple compositional distributions. To find such rules, we design an unsupervised Interior Clustering Tree that incorporates the model prediction into the splitting criteria. Then, we propose the Compositional Boundary Exploration (CBE) algorithm to obtain the boundary inference rules that estimate the decision boundary of the original model on each compositional distribution. By merging these two types of rules into a rule set, we can present the inferential process of the unsupervised black-box model in a human-understandable way, and build a surrogate rule-based model for online deployment at the same time. We conduct comprehensive experiments on the explanation of four distinct unsupervised anomaly detection models on various real-world datasets. The evaluation shows that our method outperforms existing methods in terms of diverse metrics including fidelity, correctness and robustness.

1 Introduction

In recent years, machine learning (ML) and deep learning (DL) have revolutionized many security applications such as network intrusion detection [1-3] and malware identification [4, 5] that outperform traditional methods in terms of accuracy and generalization. Among these works, unsupervised anomaly detection becomes more promising, which detects malicious activities by the deviation from normality. Compared to supervised methods, this type of method is more desirable in security domains as 1) it hardly requires labeled attack/malicious data during the training (i.e., zero-positive learning), which are typically much more sparse and difficult to obtain in contrast with benign data; 2) it does not fit any known threats, enabling better detection on unforeseen anomalies.

Due to the black-box nature of these models, ML/DL models are usually not directly interpretable and understandable. Many local explanation methods [6-10] have attempted to interpret the models by presenting feature importance of the decision for a single point. However, globally explaining black-box models, especially using rule extraction to characterize the whole decision boundaries, is particularly desirable in security systems since it can provide the following benefits:

Trust over High Stakes. To minimize the chance of errors and potential losses, security operators tend to trust human-understandable rules rather than unintuitive outputs such as labels and numeric values from complex and incomprehensible black-box models.

Online Defense. Interpreting black-box models into rules with high fidelity enables easy integration with the majority of defense tools that use rule-based expressions (e.g., iptables [11], Snort [12]), thus allowing for deployment of online defense with extraordinary efficiency (e.g., Tbps throughput [13]).

Existing global ML/DL explanation approaches are mostly proposed for supervised models. Little research has been done on explaining unsupervised anomaly detection, which faces several challenges:

Unlabeled One-class Data (CH1). Supervised explanation methods [14-16] need labeled data of both positive and negative classes to determine the decision boundaries of black-box models. This requirement goes against unsupervised anomaly detection's advantage of not requiring attack data.

Lack of Surrogate Models (CH2). Global methods typically use an inherently explainable model to mimic the black-box model, such as decision trees [15, 16] and linear models [17], which are all supervised models. However, there lacks a proper surrogate model that is unsupervised and can satisfy the properties of being self-explained and well-performed for high-stake security applications.

Accuracy Loss (CH3). A common problem with global methods is that the surrogate model suffers from the loss of accuracy since it simplifies the original model [18]. In this case, though these methods can provide model explanation, they cannot meet the need of online deployment which requires high detection accuracy in security applications.

We observe that an important reason why simple surrogate models are ineffective is that they cannot learn well about the complex data distribution in high-dimensional space. Specifically, even one-class data may be multimodal, i.e., the overall distribution consists of multiple compositional distributions. For example, the normal activities of a host may consist of many services (e.g., web, database) and are for various purposes, and they could present considerable differences in feature space.

In light of this, this paper proposes an accurate and efficient divide-and-conquer method to globally interpret unsupervised anomaly detection. First, we propose the concept of distribution decomposition rules that cut the feature space into multiple subspaces, each of which encloses a compositional distribution. To obtain such rules, we design a new tree model called Interior Clustering Tree that extends the CART decision tree in terms of splitting criteria and can fully work in an unsupervised manner. Second, we propose the Compositional Boundary Exploration algorithm to obtain the boundary inference rules that estimate the decision boundary of the original model on each compositional distribution. To accurately and efficiently find such rules, this algorithm starts from hypercube-shaped rules and uses an approximation for gradient ascent of model prediction to find the optimal direction of iterations. By merging the distribution decomposition rules and the boundary inference rules into a rule set, we can present the inferential process of the black-box model in a human-understandable way, and build a surrogate rule-based model for online defense at the same time.

During the experiment, we use four different unsupervised anomaly detection models well trained on three benchmark datasets, and evaluate our method with five distinct explanation methods as baselines. The experiment shows that our method can extract interpretable and accurate rules from black-box models. The extracted rules outperform prior work in terms of diverse metrics including fidelity, robustness, true positive rate and true negative rate, meeting the demand of improving human trust in black-box models and maintaining high detection accuracy for online deployment. Our code is available at https://github.com/Ruoyu-Li/UAD-Rule-Extraction .

2 Related Work

To combat persistent emergence of new attacks in cyberspace, recent security applications [1-3, 1921] make heavy use of unsupervised models to detect unknown anomalies, such as one-class classifiers [22-24], Isolation Forests [25, 26], autoencoders and variational autoencoders [27]. Despite many unsupervised model-based approaches have achieved good detection rates, security operators are still concerned about the semantic gap between black-box model prediction and human understanding, considering the risks of the great cost incurred by bad decisions [10]. To resolve such concerns, explainable AI (XAI) has been applied to anomaly detection [28-30]. For example, Kauffmann et al. propose a decomposition method to explain anomalies of one-class SVMs [28]. Philipp et al. present

an explainable deep one-class classification method called Fully Convolutional Data Description [30]. However, these methods are specific to a limited range of models and not versatile enough to accommodate the vastly heterogeneous models of unsupervised anomaly detection.

Some prior work also incorporates popular model-agnostic explanation methods, such as LIME [6], SHAP [7] and their variations [8], and applies them to explain unsupervised models [31-33] and security applications [9, 34, 35, 10]. These methods typically use sparse linear models to interpret predictions by estimating feature importance. Guo et al. propose a method named LEMNA that uses a fused lasso to explain malware classification [9]. Sipple uses Integrated Gradients [36] to attribute anomalies of IoT device failure [34]. Nonetheless, these methods can only interpret one data point at a time (i.e., local explanation) but not reveal the complete decision-making process of a model.

To fully understand how black-box models work and safely deploy them, the most appropriate method is model-agnostic global post-hoc explanation. It aims to match the predictions of any well-trained models with an inherently interpretable explainer, such as decision trees [15, 16], symbolic rules [14], sparse linear models [17] and decision lists [37]. In [15], the authors construct global explanations of complex black-box models in the form of a decision tree approximating the original model. Jacobs et al. propose a framework that takes an existing ML model and training dataset and generates tree models to interpret security-related decisions [16]. However, most of these methods are only suitable for interpreting supervised models that have labeled data of all classes, which are often unavailable. Though work like [38] can extract rules from unsupervised anomaly detection models, it still assumes that enough outliers exist in the training dataset judged by the black-box model so as to determine its decision boundary. This assumption may not hold in practice if a model has great generalization and can achieve a low false positive rate on normal data (e.g., [1]).

Some recent studies aggregate several local explanation models into near-global explanation [39-41]. However, this type of method is inherently computationally challenging when data volumes are large and has to make trade-offs between fidelity and coverage. While techniques like knowledge distillation can also realize model transformation to reduce complexity and promote interpretability [42, 43], the fundamental purpose of these efforts is to compress models while ensuring accuracy rather than explaining the original models with high fidelity.

3 Overview

3.1 Problem Definition

Let X ⊆ R d be the variable space of d -dimensional features; x and x i denote a data sample and the i -th dimension of the data sample. We give the following definitions for the rest of the paper:

Definition 1 (Unsupervised Anomaly Detection). Given unlabeled negative data (i.e., normal data) X sampled from a stationary distribution D for training, an unsupervised model estimates the probability density function f ( x ) ≈ P X∼D ( x ) , and detects an anomaly via a low probability f ( x ) < φ , where φ > 0 is a threshold determined by the model itself or by humans.

It is noted that the threshold φ is a non-zero value, meaning that the model inevitably generates false positives, which is a common setting in most of the works [1-3] even though the false positive rate can be very low. Besides, the normal data may occasionally be contaminated or handled with errors. We consider the anomaly detection tolerant of noisy data, but their proportion in training dataset is small and we do not have any ground truth labels of the training data.

Definition 2 (Global Explanation by Rule Extraction). Given a trained model f with its anomaly threshold φ and the training set X , we obtain an in-distribution rule set C = { C 1 , C 2 , ... } that explains how the model f profiles the distribution of normal data. A rule C = ... ∧ ( x i ⊙ υ i ) ∧ ... ∧ ( x j ⊙ υ j ) is a conjunction of several axis-aligned constraints on a subset of the feature space, where υ i is the bound for the i -th dimension and ⊙ ∈ {≤ , > } .

Let x ∈ C indicate that a data sample satisfies a rule. From C , we can build a surrogate model h C ( x ) , whose inference is to regard a data sample that cannot match any of the extracted rules as anomalous:

Our Goal. We expect the extracted rules to have a high fidelity to the original model, that is, a similar coverage of normal data (i.e., true negative rate), and a similar detection rate of anomalies (i.e., true

(a) The unlabeled data

(b) Compositional distribu- tions

(c) Process of the CBE al- gorithm

(d) The final rule set

o

o

o

o

o

o

o

o

b

o

O

o

o

o

o

o

o

O

o

o

o

o

O

个

a

o

O

o

Q

positive rate). To this end, we formulate our objective as follow:

3.2 Methodology Overview

To minimize the first item in Equation (2), suppose the training data X can well represent the distribution D , a straightforward approach is to find the bound of X as rules, such as using a hypercube to enclose the data samples which can easily achieve the minimization of the partial loss L x ∈ X ( C , f, φ ) = 0 . However, as D is not a prior distribution and we do not have labeled abnormal samples, the second item L X ≁ D ( C , f, φ ) is neither deterministic nor estimable unless we create sufficient random samples and query f , which is challenging given the high-dimensional space of X .

As prior studies [21, 34] suggest, normal data are typically multimodal, i.e., the overall distribution is formed by multiple compositional distributions. For example, a server supports multiple services such as web, email and database. The representations of these services can be disparate and located in different regions in feature space with little transition between the regions, making it infeasible to find a uniform rule set to accurately estimate the original model. An example is illustrated in Figure 1a.

Based on this intuition, we propose a divide-and-conquer approach. First, we propose an Interior Clustering Tree model (Section 4) to find the distribution decomposition rules , which cut the feature space into subspaces so that each subspace contains data belonging to the same compositional distribution, as shown in Figure 1b. Then, we design a Compositional Boundary Exploration algorithm (Section 5) to explore the decision boundary on each compositional distribution, as depicted in Figure 1c. Particularly, the algorithm starts from the minimal hypercube that encloses all data of the distribution, and finds the boundary by recursively extending the boundary following the optimal direction guided by a gradient approximation. Upon obtaining the decision boundary of a distribution, the corresponding boundary inference rule can be extracted. Last, the rule set that globally approximates the original model can be obtained by merging the distribution decomposition rule and the boundary inference rule of each compositional distribution, as illustrated in Figure 1d. We formally define the distribution decomposition rule and the boundary inference rule as follows.

Definition 3 (Distribution Decomposition Rule). Denoted by C I k that decomposes the overall distribution of normal data D into K compositional distributions, i.e., P X∼D ( x ) = ∑ K k =1 ϕ k · P X∼D k ( x | x ∈ C I k ) where ϕ k denotes the weight of each compositional distribution, so that a data sample x ∼ D k has significantly small probability of belonging to other distributions.

Definition 4 (Boundary Inference Rule). Denoted by C E k that estimates the decision boundary of the original model for each distribution D k , i.e., arg min C E k L X∼D k ( C E k , f, φ ) + L X ≁ D k ( C E k , f, φ ) .

With the definition of these two types of rules, we translate the objective in Equation (2) to the following objective as our intuition indicates. We give a proof of this proposition in the appendix.

Proposition 1. The original objective can be estimated by finding the union of the conjunction of distribution decomposition rules and boundary inference rules for each compositional distribution:

4 Interior Clustering Tree

To obtain distribution decomposition rules, we first need to decide how to define a compositional distribution D k . We notice that, though we do not have labeled data, we can use the output of the original model that estimates the overall distribution D as a criterion for decomposition.

Proposition 2. If two data samples x ( i ) and x ( j ) belong to the same distribution D k , the difference of their probabilities belonging to D will be less than ϵ , where ϵ is a small constant.

̸

Proof. Recall the definition of distribution decomposition rules and compositional distributions. Since the two data samples belong to the same distribution D k , the probability of belonging to other distributions P X∼D l ( x ) is near to zero for l = k . Hence, the probability P X∼D ( x ) , which is the weighted sum of the probability of belonging to all the compositional distributions, is approximately equal to P X∼D k ( x ) for both the data samples.

Based on this, we propose a tree-based model dubbed Interior Clustering Tree (IC-Tree), which extends the CART decision tree [44]. The main difference between IC-Tree and CART is that, rather than splitting data based on ground truth labels, IC-Tree uses the probability output by the original model as splitting criteria, enabling it to work in a completely unsupervised manner.

Node Splitting. Given the data N at a tree node, we first obtain the output of the anomaly detection model f ( x ) for x ∈ N . Similar to decision trees, the node of an IC-Tree finds a splitting point s = ( i, b i ) that maximizes the gain:

where b i is the splitting value for the i -th dimension, N l and N r are the data split to the left and right child nodes, | N | denotes the number of data samples, and I is a criterion function such as Gini index I = 2 p (1 -p ) for binary classification with the probability of p . Specifically, we let p be the average output of the anomaly detection model, which can be interpreted as the expectation of the probability that the data belong to the same distribution:

An IC-Tree continues to split nodes until it satisfies one of the following conditions: i) the number of data samples at the node | N | = 1 ; ii) for any two of the data samples at the node ∀ x ( i ) , x ( j ) ∈ N , | f ( x ( i ) ) -f ( x ( j ) ) | < ϵ ; iii) it reaches a maximum depth τ , which is a hyperparameter.

Distribution Decomposition Rule Extraction. A trained IC-Tree that has K leaf nodes ( K ≤ 2 τ ) represents K distributions separated from the overall distribution D . Suppose the k -th leaf node has a depth of τ ′ . A distribution decomposition rule that describes the k -th compositional distribution can be extracted by the conjunction of the splitting constraints from the root to the leaf node:

where ⊙ is ' ≤ ' if the decision path goes left or ' > ' if the decision path goes right.

5 Compositional Boundary Exploration

To accurately find the decision boundary of a detection model within each compositional distribution, we propose the Compositional Boundary Exploration (CBE) algorithm (described in Algorithm 1). The CBE algorithm uses the minimal hypercube that encloses the normal data of each compositional distribution as a starting point. Further, we refer to adversarial attacks [45] and propose a method to

approximate the optimal direction to explore the decision boundary, which makes the algorithm more efficient and accurate to estimate the decision boundary.

Starting from Hypercube (line 1). Let X k denote the training data falling into the k -th leaf node of an IC-Tree that represents a compositional distribution. Recall the definition of boundary inference rules that target min L X∼D k ( C E k , f, φ ) + L X ≁ D k ( C E k , f, φ ) . We use the minimal hypercube H k as a starting point of boundary inference rules to bound every dimension of the data samples in X k judged by the original model as normal, which obviously achieves L x ∈ X k ( H k , f, φ ) = 0 . The minimal hypercube is enclosed by 2 × d axis-aligned hyperplanes, which can be characterized by the following rule:

Explorer Sampling (line 4 ∼ 6). The CBE algorithm explores the decision boundary of the original model by estimating the bound of one feature dimension at a time. For i -th dimension, we uniformly sample N e data points on each hyperplane of the hypercube, i.e., e (1) , ..., e ( N e ) ∈ H k ∧ ( x i = υ i ) , υ i ∈ { υ -i , υ + i } , which are called the initial explorers for this hyperplane. For an initial explorer e , we further sample N s auxiliary explorers near it from a truncated multivariant Gaussian distribution denoted by N ( e , Σ , i ) . Particularly, the center of sampling is the explorer e and the radius of sampling is constrained by the covariance matrix Σ = diag ( ρ | υ + 1 -υ -1 | , ..., ρ | υ + d -υ -d | ) , where ρ is a hyperparameter, and the sampling on i -th dimension is half-truncated to only keep the distribution outside the hypercube as we desire to extend the boundary. With N e × N s auxiliary explorers in total, we query the original model and use Beam Search to select N e samples with the minimal probability of being normal as the candidate explorers for the next iteration.

Gradient Approximation (line 7 ∼ 9). Though we have obtained N e candidate explorers in the previous step, using them directly for the next iteration does not guarantee the optimal direction of movement towards the decision boundary. To find the optimal direction, we utilize the Fast Gradient Sign Method [45] that employs gradient ascent to find the direction of feature perturbation. However, we do not know the loss function of the original model in black-box scenarios. To deal with it, given a selected auxiliary explorer ˆ e that is sampled around an initial explorer e on the i -th dimension hyperplane, we approximate the i -th dimension of the model gradient (i.e., the partial derivative) by the slope of a linear model across the two data points, and use the midpoint with its i -th dimension minus the approximation as the new explorer for the next iteration:

sign ( · ) is the sign function, and η is a hyperparameter to control the stride of one iteration. The iteration stops when i) an auxiliary explorer ˆ e ext that satisfies f (ˆ e ext ) < φ is found, or ii) it reaches the maximum number of iterations.

Rule Acquisition (line 12). If the iteration stops due to the first condition, we produce a boundary constraint for each dimension using the coordinate of ˆ e ext that extends the boundary of the hypercube, i.e., c i = ( x i ⊙ ˆ e ext,i ) , where ⊙ is ' ≤ ' if ˆ e ext,i is greater than υ + i , or ' > ' if ˆ e ext,i is less than υ -i . If the iteration stops due to the second condition, it means the algorithm encounters difficulties in moving towards the decision boundary by perturbing this feature dimension. We calculate the difference between the model prediction of the last auxiliary explorer and that of the initial explorers on the hyperplane. If the difference is smaller than a threshold δ , we decide that this feature dimension is a contour line , i.e., it has no significant correlation with the model prediction. In this case, we do not produce any constraints for this dimension. If the difference is greater than the threshold, we produce constraints in the same way as those produced under the first condition. The final boundary inference rule is the disjunction of the hypercube and the constraints on each dimension.

6 Evaluation

6.1 Experimental Setup

Black-box Models and Datasets. We use four different types of unsupervised anomaly detection models widely used in security applications as the original black-box models, including autoencoder (AE, used by [1]), variational autoencoder (V AE, used by [46]), one-class SVM (OCSVM, used by

[47]) and Isolation Forest (iForest, used by [48]). We employ three benchmark datasets for network intrusion detection in the experiment, including CIC-IDS2017, CSE-CIC-IDS2018 [49] and TON-IoT [50]. The representation of these datasets is tabular data, where each row is a network flow record and each column is a statistical attribute, such as the mean of packet sizes and the inter-arrival time. The datasets are randomly split by the ratio of 6:2:2 for training, validation and testing. We use only normal data to train the anomaly detection models and calibrate their hyperparameters. The description of the datasets and the AUC score of the models on the datasets are shown in Table 1.

Baselines. We employ five prior explanation methods as baselines: 1) We use [38] that extracts rules from unsupervised anomaly detection (UAD); 2) For other global methods, we use the estimated greedy decision tree (EGDT) proposed by [15], and Trustee [16] that specifically explains security applications; 3) We also consider one method LIME [6] that can use a Submodular Pick algorithm to aggregate local explanations into global explanations, and a knowledge distillation (KD) method [43] that globally converts a black-box model to a self-explained decision tree. These methods, like ours, can only access normal data to extract explanations. More details about baselines are in the appendix.

Metrics. We refer to the metrics in [18] to evaluate the rule extraction. Due to limited space, we demonstrate the following four metrics in this section and present other results in the appendix: 1) Fidelity (FD), i.e., the ratio of input samples on which the predictions of original models and surrogate models agree over the total samples, which indicates the extent to which humans can trust the explanations; 2) Robustness (RB), i.e, the persistence of the surrogate model to withstand small perturbations of the input that do not change the prediction of the original model; 3) True positive rate (TPR) and true negative rate (TNR), suggesting whether the detection capability meets the need of online defense and whether the extracted rules generate noticeable false alarms that cause 'alert fatigue' [51] in highly unbalanced scenarios of most security applications, respectively.

6.2 Quality of Rule Extraction

We extract rules from the four unsupervised anomaly detection models using the five baseline methods and our method, and test the performance of the extracted rules. The results on the three datasets are in Table 2. We find that our method achieves the highest fidelity on all the detection models and datasets, and half of the scores are even over 0.99. It shows our method can precisely match the predictions of the black box models, which ensures the correctness of its global interpretation.

CSE-CIC-IDS2018 dataset

TON-IoT dataset

Moreover, our method achieves the highest TPR on all the detection models and datasets; specifically, the TPR is equal to 1.00 for all the detection models on TON-IoT dataset. This result suggests that our rules can accurately detect various anomalous data, making it possible to realize online deployment and defense using these rules. Our method also reaches a high level of robustness (minimum 0.9890, maximum 1.00) and true negative rate (minimum 0.9715, maximum 1.00). Therefore, it is concluded that our method can obtain rules of high quality from different black-box unsupervised anomaly detection models using only unlabeled one-class data.

Considering that obtaining a 'clean' training set requires huge manual effort in reality [52], we also assess the efficacy of our method under varying percentages of 'noisy' data. We evaluate the fidelity of extracted rules using two approaches for the injection of noisy data: 1) random noise; 2) mislabeled data from other classes, i.e., attack data. The results are shown in Table 3. We find that the impact of the noisy data proportion is not significant: 36 of 40 fidelity scores in the table preserve over 0.95, and the variation of fidelity scores is not obvious with the increase of noisy data for most of the models. This shows that our rule extraction method can retain similar performance to the black-box model that it extracts from. Nonetheless, the results of iForest also reveal that a sufficiently large proportion of noisy data may cause a certain negative impact on the rule extraction for certain models.

6.3 Understanding Model Decisions

To demonstrate that the rules obtained by our method are in line with human understanding, we use the OCSVM as an example of black-box models to exhibit several explanations. We extract rules from the well-train model and use the rules to predict three typical types of attack data, including Distributed Denial-of-Service (DDoS) attacks, scanning attacks, SQL injection, and backdoor attacks.

Table 4 shows some features of the rules extracted from normal data that cannot be matched by the attack data, and exhibits how humans can interpret the model decisions 2 . For example, the data of DDoS attacks cannot match the rules of three feature dimensions, including mean of packet sizes, mean of packet inter-arrival time, and duration of a connection. It can be observed that the feature values of attacks are markedly lower than the bound of the rules. Such results are easy to interpret. Because the purpose of DDoS attacks is to overwhelm the resources of a victim, an attacker will realize asymmetric resource consumption between the victim and himself (i.e., using small packets), send packets at an extremely high rate (i.e., low inter-arrival time), and establish as many useless connections as possible (i.e., short duration of connections). These explanations are in line with how humans recognize the attack data. Hence, we can draw a conclusion that our method is able to provide precise insights into black-box anomaly detection models in a human-understandable way.

6.4 Ablation Study

To evaluate the contribution of each component in our method, including the IC-Tree and the CBE algorithm, we conduct an ablation experiment by 1) replacing the IC-Tree with a clustering algorithm K-Means, 2) using only the CBE algorithm, and 3) replacing the CBE algorithm with directly using hypercubes as rules. In Table 5, we find that our method (IC-Tree + CBE) outperforms others in terms of fidelity on both datasets. Though using the K-Means can reach similar results, it cannot be expressed by axis-aligned rules with high interpretability and deployability as the IC-Tree can achieve. In summary, both components are helpful for the quality of rule extraction.

6.5 Computational Cost and Complexity

We also evaluate the computational cost of our method with respect to training and prediction. Since CIC-IDS2017 dataset has 80 features in total, we train the model using the first 20, 40, 60, and 80 features of 4000 samples to investigate the influence of feature sizes. The results are shown in Table 6, which demonstrate the average training and prediction time of our method. It can be seen that the training time is around 1 minute, which is acceptable and practical for large-scale training. Besides, the training time increases basically linearly with the increase of feature sizes. This is because our method adopts a feature-by-feature strategy to explore the decision boundary of the model. For prediction time, our method is highly efficient, which only costs microsecond-level overhead for one inference. It shows that as a rule-based approach, our method can achieve real-time execution for online use. Note that the runtime is measured purely based on Python. In practice, the prediction time of our method can be even less with more efficient code implementation.

(a) Maximum tree depth

(b) #Explorers

(c) Sampling coefficient

(d) Iteration stride

100.0

100.0

97.7

100.0

100.0

(%)

(%)

(%)

98.0

Metrics

99.9

97.8

Metrics

99.9

97.6

Metrics

99.9

98.0%

Metrics

99.9

99.8

99.7

TPR

FD

RB

TNR

97.6

TNR

99.8

99.7

RB

TNR

FD

TPR

97.5

TNR

99.8

99.7

FD

RB

TNR

TPR

99.8

99.7

FD

RB

TPR

TNR

97.8

97.6

R

10

20

30

5

10

0.0

0.5

1.0

0.0

0.5

1.0

We also theoretically analyze the time complexity of our algorithms. For training, the complexity of the IC-Tree is identical to a CART: O ( d · n log n ) , where d is the feature size and n is the sample number; the complexity of the CBE algorithm is O ( K · d · N e · N s ) , where K is the number of leaf nodes of the IC-Tree, and N e and N s are the number of initial explorers and auxiliary explorers. Therefore, the training time is theoretically linear to the feature size, which is in line with the empirical results. For execution, the time complexity is O ( | C | · d ) , where | C | is the number of extracted rules.

6.6 Hyperparameter

We perform a sensitivity analysis of several hyperparameters on their influence on the rule extraction. We present four major hyperparameters in Figure 2, including the maximum depth τ of an IC-Tree, N e number of explorers, the coefficient ρ of sampling, and the factor η that controls the stride of an iteration. Due to limited space, the analysis of other hyperparameters is placed in the appendix.

Maximum tree depth. A deeper IC-Tree has more leaf nodes, and can accordingly decompose more distributions that ease the difficulty of rule extraction. Meanwhile, excessively fine-grained splitting might cause overfitting. We find that τ = 15 achieves the best performance.

Number of Explorers. It is essentially the number of selected nodes per iteration in Beam Search, which considers multiple local optima to improve greedy algorithms. But selecting too many nodes may also include more redundancy. Figure 2b shows that a value between 6 and 8 is recommended.

Coefficient of sampling. Figure 2c shows that a higher value of the hyperparameter achieves better results. A large coefficient decides a large radius of sampling from a multivariant Gaussian distribution, which helps the CBE algorithm quickly find the decision boundary of the original model.

Factor of iteration stride. In Figure 2d, we find that a larger factor η can obtain rules of higher quality. As it decides the stride of finding the explorers for the next iteration, a higher value of the hyperparameter might help the convergence of the iteration process.

7 Conclusion and Future Work

This paper proposes a novel method to globally interpret black-box unsupervised anomaly detection, in which we introduce two types of rules and corresponding algorithms to extract these rules. The evaluation shows that our method outperforms prior work in terms of various metrics, boosting user confidence in complex models and facilitating their integration into high-stake security applications.

There are numerous meaningful directions for future work. First, new algorithms can be developed based on model interpretation to troubleshoot illogical inference processes in black-box models (e.g., false positives) and realize the automation of precisely fixing errors. Moreover, researchers can investigate the integration of rule extraction with high-performance rule-based defense systems, such as P4 [13], to implement a more efficient security system. In addition, since the proposed method can be generic, the transfer to other areas of industry that also demand interpretable anomaly detection, such as health, manufacturing and criminal investigation, can be further explored.

Acknowledgments and Disclosure of Funding

This work is supported by the National Key Research and Development Program of China under grant No. 2022YFB3105000, the Major Key Project of PCL under grant No. PCL2023AS5-1, the Shenzhen Key Lab of Software Defined Networking under grant No. ZDSYS20140509172959989, and the research fund of Tsinghua University - Tencent Joint Laboratory for Internet Innovation Technology.

References

Y. Mirsky, T. Doitshman, Y. Elovici, and A. Shabtai, 'Kitsune: An ensemble of autoencoders for online network intrusion detection,' in Annual Network and Distributed System Security Symposium (NDSS) , 2018.

R. Tang, Z. Yang, Z. Li, W. Meng, H. Wang, Q. Li, Y. Sun, D. Pei, T. Wei, Y. Xu, and Y. Liu, 'Zerowall: Detecting zero-day web attacks through encoder-decoder recurrent neural networks,' in IEEE Conference on Computer Communications (INFOCOM) , 2020.

C. Fu, Q. Li, M. Shen, and K. Xu, 'Realtime robust malicious traffic detection via frequency domain analysis,' in ACM SIGSAC Conference on Computer and Communications Security (CCS) , 2021.

R. Perdisci, W. Lee, and N. Feamster, 'Behavioral clustering of http-based malware and signature generation using malicious network traces,' in USENIX Symposium on Networked Systems Design and Implementation (NSDI) , 2010.

E. C. R. Shin, D. Song, and R. Moazzezi, 'Recognizing functions in binaries with neural networks,' in USENIX Security Symposium , 2015.

M. T. Ribeiro, S. Singh, and C. Guestrin, '"why should i trust you?": Explaining the predictions of any classifier,' in ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) , 2016.

S. M. Lundberg and S. Lee, 'A unified approach to interpreting model predictions,' in Annual Conference on Neural Information Processing Systems (NeurIPS) , 2017.

M. T. Ribeiro, S. Singh, and C. Guestrin, 'Anchors: High-precision model-agnostic explanations,' in AAAI Conference on Artificial Intelligence (AAAI) , 2018.

W. Guo, D. Mu, J. Xu, P. Su, G. Wang, and X. Xing, 'Lemna: Explaining deep learning based security applications,' in ACM SIGSAC Conference on Computer and Communications Security (CCS) , 2018.

D. Han, Z. Wang, W. Chen, Y. Zhong, S. Wang, H. Zhang, J. Yang, X. Shi, and X. Yin, 'Deepaid: Interpreting and improving deep learning-based anomaly detection in security applications,' in ACM SIGSAC Conference on Computer and Communications Security (CCS) , 2021.

netfilter project, 'iptables,' https://www.netfilter.org/projects/iptables/index.html, 2023.

snort, 'Snort ids,' https://www.snort.org/, 2023.

G. Xie, Q. Li, Y. Dong, G. Duan, Y. Jiang, and J. Duan, 'Mousika: Enable general in-network intelligence in programmable switches by knowledge distillation,' in IEEE Conference on Computer Communications (INFOCOM) , 2022.

M. W. Craven and J. W. Shavlik, 'Using sampling and queries to extract rules from trained neural networks,' in International Conference on Machine Learning (ICML) , 1994.

O. Bastani, C. Kim, and H. Bastani, 'Interpreting blackbox models via model extraction,' CoRR , vol. abs/1705.08504, 2017.

A. S. Jacobs, R. Beltiukov, W. Willinger, R. A. Ferreira, A. Gupta, and L. Z. Granville, 'Ai/ml for network security: The emperor has no clothes,' in ACM SIGSAC Conference on Computer and Communications Security (CCS) , 2022.

J. H. Friedman and B. E. Popescu, 'Predictive learning via rule ensembles,' The Annals of Applied Statistics , vol. 2, no. 3, 2008.

G. Vilone, L. Rizzo, and L. Longo, 'A comparative analysis of rule-based, model-agnostic methods for explainable artificial intelligence,' in Irish Conference on Artificial Intelligence and Cognitive Science , 2020.

M. Du, F. Li, G. Zheng, and V. Srikumar, 'Deeplog: Anomaly detection and diagnosis from system logs through deep learning,' in ACM SIGSAC Conference on Computer and Communications Security (CCS) , 2017.

R. Li, Q. Li, J. Zhou, and Y. Jiang, 'Adriot: An edge-assisted anomaly detection framework against iot-based network attacks,' IEEE Internet of Things Journal , vol. 9, no. 13, pp. 10 57610 587, 2022.

R. Li, Q. Li, Y. Huang, W. Zhang, P. Zhu, and Y. Jiang, 'Iotensemble: Detection of botnet attacks on internet of things,' in European Symposium on Research in Computer Security (ESORICS) , 2022.

B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson, 'Estimating the support of a high-dimensional distribution,' Neural Comput. , vol. 13, no. 7, pp. 1443-1471, 2001.

L. Ruff, N. Görnitz, L. Deecke, S. A. Siddiqui, R. A. Vandermeulen, A. Binder, E. Müller, and M. Kloft, 'Deep one-class classification,' in International Conference on Machine Learning (ICML) , 2018.

S. Itani, F. Lecron, and P. Fortemps, 'A one-class classification decision tree based on kernel density estimation,' Appl. Soft Comput. , vol. 91, p. 106250, 2020.

F. T. Liu, K. M. Ting, and Z. Zhou, 'Isolation forest,' in IEEE International Conference on Data Mining (ICDM) , 2008.

H. Xu, G. Pang, Y. Wang, and Y. Wang, 'Deep isolation forest for anomaly detection,' CoRR , vol. abs/2206.06602, 2022.

D. P. Kingma and M. Welling, 'Auto-encoding variational bayes,' in International Conference on Learning Representations (ICLR) , 2014.

J. Kauffmann, K.-R. Müller, and G. Montavon, 'Towards explaining anomalies: A deep taylor decomposition of one-class models,' Pattern Recognition , vol. 101, p. 107198, 2020.

D. Kazhdan, B. Dimanov, M. Jamnik, and P. Liò, 'MEME: generating RNN model explanations via model extraction,' CoRR , vol. abs/2012.06954, 2020.

P. Liznerski, L. Ruff, R. A. Vandermeulen, B. J. Franks, M. Kloft, and K. Müller, 'Explainable deep one-class classification,' in International Conference on Learning Representations (ICLR) , 2021.

Z. Kong and K. Chaudhuri, 'Understanding instance-based interpretability of variational autoencoders,' in Annual Conference on Neural Information Processing Systems (NeurIPS) , 2021.

J. Crabbé and M. van der Schaar, 'Label-free explainability for unsupervised models,' in International Conference on Machine Learning (ICML) , 2022.

O. Eberle, J. Büttner, F. Kräutli, K. Müller, M. Valleriani, and G. Montavon, 'Building and interpreting deep similarity models,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 44, no. 3, pp. 1149-1161, 2022.

J. Sipple, 'Interpretable, multidimensional, multimodal anomaly detection with negative sampling for detection of device failure,' in International Conference on Machine Learning (ICML) , 2020.

L. Yang, W. Guo, Q. Hao, A. Ciptadi, A. Ahmadzadeh, X. Xing, and G. Wang, 'CADE: detecting and explaining concept drift samples for security applications,' in USENIX Security Symposium , 2021.

M. Sundararajan, A. Taly, and Q. Yan, 'Axiomatic attribution for deep networks,' in International Conference on Machine Learning (ICML) , 2017.

B. Letham, C. Rudin, T. H. McCormick, and D. Madigan, 'Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model,' CoRR , vol. abs/1511.01644, 2015.

A. Barbado, O. Corcho, and R. Benjamins, 'Rule extraction in unsupervised anomaly detection for model explainability: Application to oneclass svm,' Expert Systems with Applications , vol. 189, p. 116100, 2022.

I. van der Linden, H. Haned, and E. Kanoulas, 'Global aggregations of local explanations for black box models,' CoRR , vol. abs/1907.03039, 2019.

M. Setzu, R. Guidotti, A. Monreale, F. Turini, D. Pedreschi, and F. Giannotti, 'Glocalx - from local to global explanations of black box AI models,' Artif. Intell. , vol. 294, p. 103457, 2021.

Q. Li, R. Cummings, and Y. Mintz, 'Optimal local explainer aggregation for interpretable prediction,' in AAAI Conference on Artificial Intelligence (AAAI) , 2022.

N. Frosst and G. E. Hinton, 'Distilling a neural network into a soft decision tree,' CoRR , vol. abs/1711.09784, 2017.

Y. Li, J. Bai, J. Li, X. Yang, Y. Jiang, and S. Xia, 'Rectified decision trees: Exploring the landscape of interpretable and effective machine learning,' CoRR , vol. abs/2008.09413, 2020.

L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification and Regression Trees . Wadsworth, 1984.

I. J. Goodfellow, J. Shlens, and C. Szegedy, 'Explaining and harnessing adversarial examples,' in International Conference on Learning Representations (ICLR) , 2015.

X. Xu, J. Li, Y. Yang, and F. Shen, 'Toward effective intrusion detection using log-cosh conditional variational autoencoder,' IEEE Internet of Things Journal , vol. 8, no. 8, pp. 61876196, 2021.

A. Binbusayyis and T. Vaiyapuri, 'Unsupervised deep learning approach for network intrusion detection combining convolutional autoencoder and one-class SVM,' Appl. Intell. , vol. 51, no. 10, pp. 7094-7108, 2021.

Y. Dong, Q. Li, K. Wu, R. Li, D. Zhao, G. Tyson, J. Peng, Y. Jiang, S. Xia, and M. Xu, 'Horuseye: Realtime iot malicious traffic detection framework with programmable switches,' in USENIX Security Symposium , 2023.

L. Liu, G. Engelen, T. M. Lynar, D. Essam, and W. Joosen, 'Error prevalence in NIDS datasets: A case study on CIC-IDS-2017 and CSE-CIC-IDS-2018,' in IEEE Conference on Communications and Network Security (CNS) , 2022.

T. M. Booij, I. Chiscop, E. Meeuwissen, N. Moustafa, and F. T. H. den Hartog, 'Ton_iot: The role of heterogeneity and the need for standardization of features and attack types in iot network intrusion data sets,' IEEE Internet of Things Journal , vol. 9, no. 1, pp. 485-496, 2022.

B. A. AlAhmadi, L. Axon, and I. Martinovic, '99% false positives: A qualitative study of SOC analysts' perspectives on security alarms,' in USENIX Security Symposium , 2022.

G. Apruzzese, P. Laskov, and A. Tastemirova, 'Sok: The impact of unlabelled data in cyberthreat detection,' in IEEE European Symposium on Security and Privacy (EuroS&P) , 2022.

A Appendix

A.1 Proof of Proposition 1

Our Goal. We expect the extracted rules to have a high fidelity to the original model, that is, a similar coverage of normal data (i.e., true negative rate), and a similar detection rate of anomalies (i.e., true positive rate). To this end, we formulate our objective as follows:

Proposition 1. The original objective can be estimated by finding the union of the conjunction of distribution decomposition rules and boundary inference rules for each compositional distribution:

We give the following lemma equivalent to Proposition 1 and prove it:

Lemma 1.1. If the distribution decomposition rules and the inference boundary rules that minimize the loss on each of the compositional distributions are found, the sum of the minimum losses on each of the compositional distributions can estimate the minimum loss on the overall distribution with a significantly small error ψ , i.e.,

Proof. The sum of the minimum losses on each of the compositional distributions is calculated by an iteratively cumulative process. Let L j be the sum of the minimum losses on each of the compositional distributions at the j -th iteration, i.e.,

Let X ∼ ⋃ j k =1 D k represent a variable belonging to any of the compositional distributions D 1 , ..., D j . We prove the Loop Invariant of L j during the iteration, which always satisfies:

For the first iteration, the equation L 1 = min L X∼D 1 ( C 1 , f, φ ) + L X ≁ D 1 ( C 1 , f, φ ) + ψ obviously holds where ψ = 0 .

Suppose the Equation (12) holds at the j -th iteration. For the ( j +1) -th iteration, we have the following derivations:

where ⋃ j +1 k =1 D k ∩ D j +1 represents the overlap area between the conjunction of the compositional distributions ⋃ j +1 k =1 D k and the ( j +1) -th compositional distribution D j +1 . Recall the definition of the compositional distributions that a data sample belonging to one compositional distribution has a significantly small probability of belonging to other compositional distributions, meaning that the overlap area between the compositional distributions is significantly small. Therefore, the loss with respect to the overlap area is also significantly small, given the data samples belonging to the area are significantly rare. Let

and we can get the final result of L j +1 :

which proves the loop invariant in Equation (12). At the last iteration when j = K , as the overall distribution is equal to the conjunction of the compositional distributions, i.e., D = ⋃ K k =1 D k , we prove Equation (11) holds and Lemma 1.1 is correct.

A.2 Implementation of Our Method

Our implementation is primarily based on PyTorch (version 1.12.1) for the deep learning models, such as AE and VAE. Additionally, for data preprocessing, feature engineering, and model evaluation, we employ the versatile machine learning library scikit-learn (version 1.1.3). Python (version 3.9.15) serves as the programming language for our implementation, providing a rich ecosystem of libraries and tools for data manipulation and experiment orchestration.

Our experiments were conducted on a server equipped with the Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz (128GB RAM) and the GeForce RTX 2080 Super (8GB VRAM). Note that GPU is only used for the training of some DL-based anomaly detectors (e.g., AE, VAE), and our rule extraction method only requires the use of CPU.

A.3 Implementation of Baselines

In this section, we delve into the details of the baseline methods used for evaluation in our experiments, focusing on their implementation in the context of globally explaining unsupervised anomaly detection. The five baseline methods include Rule extraction from UAD, Estimated Greedy Decision Tree (EGDT), Trustee, Local Interpretable Model-agnostic Explanations (LIME), and a Knowledge Distillation (KD) method.

A.3.1 Rule Extraction from UAD

The Rule Extraction from UAD method was initially designed for one-class support vector machines (OCSVM) but is applicable to various types of unsupervised anomaly detection models. It uses feature scaling, anomaly detection, and clustering to produce rules from the input parameters X,ln, lc, d, m, t , where X refers to the input data frame with the features, ln is a list with the numerical columns, lc is a list with the categorical columns, d is a dictionary with the hyperparameters for OCSVM (kernel type, upper bound on the fraction of training errors and a lower bound of the fraction of support vectors, ν , and the kernel coefficient, γ ). m is a variable that defines the type of cluster algorithm, and t is a variable that specifies the type of approximation for obtaining the hypercubes.

The approach for obtaining rules depends on the type of cluster algorithm used and can involve iterative clustering and splitting of data points. The choice of kernel type in the OCSVM model can influence the rules obtained, with RBF kernel tending to enclose inliers and leave outliers outside, eventually forming hypercubes to represent the rules.

A.3.2 Estimated Greedy Decision Tree (EGDT)

The EGDT method constructs a decision tree that approximates the black-box model. The algorithm generates new training data by actively sampling new inputs and labeling them with a black-box

model. Based on these training data, EGDT constructs the decision tree T ∗ of size k similar to CART, in a greedy manner and is pruned to improve interoperability. The algorithm takes into account the distribution of points that are routed to each leaf node in the decision tree to ensure that the label assigned to each leaf node is accurate.

A.3.3 Trustee

The model-agnostic DT extraction method Trustee, specifically focuses on interpreting security applications. The core idea is to construct an interpretable decision tree by minimizing the difference between the black-box model and the surrogate model. The algorithm creates several high-fidelity decision trees by executing an outer loop S times and an inner loop N times. In an unsupervised setting, we treat the predictions of the black-box model as pseudo-labels and employ the same process as for the supervised case.

A.3.4 Local Interpretable Model-agnostic Explanations (LIME)

By learning an interpretable model locally around the prediction, LIME explains the predictions of any classifier in an interpretable and faithful manner. LIME uses locally weighted square loss and an exponential kernel to approximate the black-box model. LIME trains a sparse linear model on the local dataset to explain the prediction. The feature importance scores assist in identifying the most important features that contribute to the prediction, which can be used to gain insights into the model and improve LIME's trustworthiness.

In addition, this paper proposes 'SP-LIME' that selects a set of representative instances with explanations to address the 'trusting the model' problem, via submodular optimization. The submodular pickup method is used to select a subset of instances that are representative of the entire dataset while reducing redundancy. This method allows for a global perspective to ascertain trust in the model, in addition to explaining individual predictions. LIME can therefore be transformed from a local method to a global method using the submodular pickup method, which makes it more useful in real scenarios.

The implementation of LIME in unsupervised settings is fairly straightforward. For a given input instance, we create a set of perturbations, predict their labels using the black-box model, and then fit an interpretable model (usually a linear model) on this newly created dataset to approximate the local decision boundary.

A.3.5 Knowledge Distillation (KD) Method

In order to achieve interpretability and effectiveness simultaneously, the KD method proposes a knowledge distillation-based decision tree extension, called rectified decision trees (ReDT), and tries to transfer the knowledge from a complex model (the teacher) to a simpler one (the student). ReDT extends the splitting criteria and the ending condition of the standard decision trees to allow training with soft labels distilled from a well-trained teacher model while preserving the deterministic splitting paths. They recommend a jackknife-based distillation to obtain soft labels, which does not require backpropagation for the student model, and analyze the effectiveness of adopting soft labels instead of hard ones empirically and theoretically. In an unsupervised scenario, we employ the anomaly scores produced by the teacher model as pseudo-labels. The student model is then trained to imitate these scores. This method's limitation lies in its strong reliance on the correctness of the teacher model's scores.

We need to note that these unsupervised adaptations of the baselines essentially introduce a supervised flavor to the learning process, with the black-box model's predictions serving as the pseudo-labels. It is an important observation that these methods do not inherently work in an unsupervised manner and their application in such settings might be unreliable depending on the accuracy of the blackbox model's predictions. On the other hand, our proposed method is fundamentally designed for unsupervised settings, making it a more trustworthy choice for real-world applications.

It is also worth mentioning that, except for LIME 3 and Trestee 4 which use open-source code, all other baseline methods were implemented in Python using the scikit-learn library, with default

parameters unless specified otherwise. Hyperparameters were set based on the initial grid search and then manually fine-tuned.

A.4 More Details in Experiments

A.4.1 More Metrics

We also examine other metrics in addition to the metrics given in the main article.

Precision (PR) measures the proportion of correctly predicted positive instances out of the total instances anticipated as positive. It indicates the model's ability to avoid false positives. A lower rate of false positives is indicated by precision values that are higher.

Correctness (CR) evaluates the precision of the explanations provided by the IC-Tree method with respect to the underlying model. This can be quantified using the Jaccard similarity index, defined as Cr = r N , where r is the number of correct outputs by the model.

Recall (RC) also known as sensitivity or true positive rate, quantifies the proportion of correctly predicted positive instances out of all actual positive instances. It indicates the model's ability to identify all positive instances without missing any. Higher recall values indicate a lower rate of false negatives. It can be quantitatively assessed by the ratio RC = TP TP + FN .

F1 score (F1) is the harmonic mean of precision and recall combined into a single statistic, which provides a balanced measure of both precision and recall, capturing the overall performance of the model.

Besides the aforementioned metrics, in fact, we believe that the most appropriate way to assess 'human's trust in anomaly detection models' should be employing a trial in which human security experts are invited to inspect the rules of interpretation, measuring the consistency of understanding. Yet this is not an easy task to conduct: to eliminate the subjective influence of each expert, enough people must be invited to participate in the trial, which can be somewhat difficult to carry out. Despite the intrinsic limitations of lacking such a metric, it is indeed a common practice to employ only those commonly used metrics (e.g., fidelity, F1 score) even in research works accepted to top venues, which may reveal an obvious gap between 'research' and 'practice'.

The specifics of the performance metrics of our proposed method in comparison to other baseline methods, including UAD, EGDT, Trustee, LIME, and KD, are demonstrated in Table 7 and Table 8. In addition, we analyze the effects of several hyperparameters on our methodology, as shown in Table 9. The performance metrics are categorized based on different anomaly detection algorithms: Autoencoder (AE), Variational Autoencoder (VAE), One-Class SVM (OCSVM), and Isolation Forest (iForest) are shown in Table 8. Precision (PR), Correctness (CR), Recall (RC), and F1 score (F1) are evaluated in this paper. Notably, our proposed method achieves the highest performance in almost all categories, with a few significant instances where it ties with the best performer. This indicates the comprehensive superiority of our approach, as it consistently delivers exceptional performance across a wide range of models and metrics.

This extensive evaluation underlines the robustness and effectiveness of our method compared to established baselines. The superior performance, across varying metrics and under different detection models, signifies its potential as a versatile solution for rule extraction in the TON-IoT dataset. This

demonstrates the practical utility of our proposed method, especially in a diverse and dynamic domain such as IoT, which could significantly benefit from such adaptable and high-performing solutions.

As for the baseline methods, while some of them can achieve acceptable fidelity for certain models, they fail to maintain such results on other models, indicating that they cannot achieve qualified model-agnostic global explanations for unsupervised models. Further, most of their recall scores cannot meet the requirement of using their rules for online defense. It is mainly because these methods either require labeled data to determine the boundary between normal and abnormal (e.g., EGDT and LIME), or need sufficient outliers in the training data (e.g., UAD and KD), which can be unavailable in many security applications. In contrast, our method eliminates these requirements by the IC-Tree and CBE algorithm that explores the decision boundary in an unsupervised manner, and meanwhile realizes a high detection rate of anomalies.

Lastly, it is worth noting that while our method exhibits superior performance, we do not imply the obsoleteness of other methods. Every method has its strengths and use cases, thus the selection of the method should always be context-dependent. Future work could involve fine-tuning our method to improve its performance further or be applied to other domains.

A.4.2 More Clarification of Human Understanding

We present a step-by-step description of obtaining the content in Table 4 in an ideal experiment, which takes data points as input and then shows the steps that come up with the explanations:

For a reported anomaly x , we use the IC-Tree to pinpoint the rule of normality that judges x as anomalous. It is realized by inputting x into the tree and recursively finding the leaf node. The rule is denoted by C x .

For each constraint of features in C x , we compare the corresponding feature value of x with the range of the constraint.

For the feature values outside the range of the constraints, the Rules of Normality, Feature Values, and Feature Meaning (i.e., the three columns in Table 4) along with the raw data sample will be sent as the explanation to the security expert for further analysis.

The security expert will analyze the data sample to determine the type of attack (i.e., the Attack column in Table 4) and give her/his understanding of the important attributes that make her/him identify the attack (i.e., the Human Understanding column in Table 4).

If there is a huge gap between the provided explanation and expert understanding, it indicates that the anomaly detector is not trustworthy. The security expert may conduct further actions to improve the detector, such as retraining, fine-tuning, or reconsidering feature selection.

Note that such a gap may not only occur when the anomaly detector makes erroneous decisions but also when the anomaly detector correctly detects an attack sample by unreasonable features. For example, the most obvious attribute of a Denial-of-Service (DoS) attack is typically its high rate of forwarding packets in order to overwhelm the victim. However, the provided explanation might suggest the IP address is an important feature for the anomaly detector to make this inference, which is quite doubtful from the perspective of a security expert. In practice, this may be due to 'spurious correlation' or 'shortcut learning' 5 . Specifically, for the DoS example mentioned in our paper, unreliable decisions could be made due to inappropriate testbed settings for the collection of

training data, such as the DoS attack being launched from one separate host address while all other normal traffic is from other host addresses.

A.4.3 More Hyperparameter Analysis

The robustness of our method against changes in hyperparameters also needs to be evaluated. Sensitivity analysis analysis us to identify which parameters are most influential, thereby concentrating on tuning those to achieve optimal performance. In our study, we perform a sensitivity analysis on the hyperparameters: Maximum Iteration, Auxiliary Explorer Sample Size ( N s ), Distribution Similarity Threshold ( ϵ ), and Significance Threshold ( δ )).

Maximum Iteration represents the maximum number of iterations that the algorithm will run. In our analysis, as shown in Table 9, this parameter's variation from 0.01 to 0.5 has a negligible effect on all the measured indicators. This displays the algorithm's resilience and robustness to changes in this hyperparameter, indicating that the model converges relatively rapidly.

Auxiliary Explorer Sample Size ( N s ) represents the number of samples gathered to assist the model's exploration phase. The results indicate that our model effectively makes use of the available samples because an increase in Ns has little to no effect on the majority of the metrics. However, a slight increase in fidelity, correctness, and accuracy is observed at larger sample sizes, indicating that providing more samples can marginally enhance the model's performance.

Distribution Similarity Threshold ( ϵ ) dictates how similar the proposed and baseline distributions should be for the tree's branches to accept the proposed distribution. Interestingly, whereas other metrics practically remain constant as epsilon goes from 10 to 150, fidelity, correctness, and accuracy show a slight uptick. This suggests that our model exhibits an adaptable behavior to changes in distribution similarity thresholds.

Significance Threshold ( δ ) , which determines the minimum difference that should be deemed noteworthy, also shows a negligible effect on all the performance indicators. This shows that the model is not sensitive to changes in δ and supports the robustness of the proposed approach.

One noticeable aspect across all parameters is the TPR and TNR values. The TPR is consistently at its maximum, showcasing the system's excellent ability to identify positive instances. TNR also remains high, demonstrating the system's effectiveness in accurately identifying negative instances.

This sensitivity analysis underlines the robustness and adaptability of our proposed method. Despite varying hyperparameters, our method consistently delivers high performance, emphasizing its potential as a versatile solution in diverse real-world scenarios. However, care should be taken to appropriately tune these parameters according to the specifics of the given context.

Future work might explore other influential hyperparameters and conduct a similar sensitivity analysis. Moreover, utilizing methods like automatic hyperparameter tuning or evolutionary algorithms can lead to more efficient and optimal configuration settings.

A.5 Limitations

Though our method achieves global explanation with high fidelity, it still has several limitations. First, our method works well with tabular data but might be inapplicable to raw image data. It is because our method treats every dimension of the feature space as a semantic feature, such as the average packet size of a network connection, and extracts rules from each dimension of the feature space. In

contrast, raw image data are tensors of pixels. Their high-level semantics cannot directly derive from each of the pixels but usually need a deep model with spatial awareness (e.g., CNN, ViT) to extract feature maps, which are inconsistent with our method. Hence, this issue may limit the transferability of the proposed method to other domains. Nonetheless, due to the typical trust in expert knowledge over deep models in security domains, most security applications still rely on sophisticated feature engineering and use data representations with explicit semantics, suggesting that our method remains general in the field of security.

Second, recall that the rules extracted by our method are axis-aligned, which can be interpreted as a certain feature over/under a threshold and are human-understandable. Though this format of rules significantly promotes interpretability, it may limit its degree of fitting to the decision boundary of the original model, which can be of various shapes in the high-dimensional feature space for different models. Though our IC-Tree has mitigated this issue by splitting the distribution of normal data into multiple compositional distributions, which are more compact and more likely to be fitted using axis-aligned rules, there is little guarantee that our method cannot encounter underfitting if the decision boundary of the original model is extremely irregular. It should be clarified that this is a common limitation for all the global explanation methods that employ axis-aligned rules or decision trees as the surrogate expression. To this end, we are also exploring other surrogate models and algorithms that can further balance the interpretability and fitting ability.

Lastly, as we mentioned above, we believe that the experiments on 'human understanding and trust' can be significantly strengthened by introducing security practitioners to participate in a use test, finding if the interpretation provided by the proposed method is consistent with their expert knowledge of judging an anomaly. Currently, we are collaborating with the Tencent Security Platform Department, aiming to rectify this limitation by accounting for the opinions of real practitioners.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a rule-set extraction method for a black-box anomaly detection model that is trained with only normal or unlabeled data.\nThere are many methods to interpret the learned model, but, this paper claims that there are few methods designed for anomaly detection.\nThe proposed method first extracts rules that decompose the complex distribution of normal data into multiple simple distributions and then, in each simple distribution, the decision rules that approximate the decision boundary of the pre-trained model are estimated. \nBy merging such rules, the proposed method can obtain a surrogate rule-based model of the trained anomaly detector.\n\nSTRENGTHS:\n- The problem tackled in this paper is important. There is a demand for interpretable anomaly detection in many industries, including security.\n- The proposed method is model-agnostic and thus can be applied in many applications.\n- The proposed method works well in the experiments.\n\nWEAKNESSES:\n- There is no discussion or comparison regarding computational costs.\n- There are several unclear points about the methodology. This may be because I am a non-expert in this area (rule-based methods). I would like to wait for the author's additional explanation in the rebuttal.\n\nQUESTIONS:\n- This method is proposed as security-specific, but if I understand correctly, there is no security-specific part of the model. Is it better to present it as a more generic method since there are other areas of industry where rule sets are required?\n- The proposed method is mostly feature-by-feature processing, so can it be used for high dimensional data as well? The datasets with small feature sizes (30) were used in the experiments. Some discussion of the computational cost would be required.\n- Does Proposition 2 claim that the probabilities of the samples belonging to the same distribution $D_k$ are similar? If so, I think that the assumption is not satisfied, e.g., if $D$ is a Gaussian mixture distribution (i.e., $D_k$ is each Gaussian component). Please let me know if I have misunderstood.\n- In Eq. (4), although $I(N_l)$ and $I(N_r)$ depend on $N_l$ and $N_r$, it is unclear how $N_l$ and $N_r$ are used for caluculating the value of $I$. I also did not understand why the output of anomaly detection could be used to partition the distribution in Eq. (4).\n- In Eq. (8), It reads as the sum of a vector and scalar. Is the second term a vector?\n\nMinors\n- In section 1, this paper mentions one reason for using global explanatory methods is that forecasts are interpretable. However, I think that this property also be achieved by local explanation methods.\n- It is better to explain the meaning of each loss of Eq. (2) in Section 2.\n- In definition 3, $\\phi_k$ is not defined.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with the problem of \"explainable\" unsupervised machine learning (ML) for anomaly detection (AD), with a focus of network intrusion detection (NID). The paper argues that while abundant effort focused on providing _supervised_ ML techniques that are explainable, this is not the case for _unsupervised_ ML methods---which are abundant in NID due to the lack of fine-grained labelled data. The paper hence seeks to rectify such shortage by proposing a novel solution that simultaneously (i) provides \"human-understandable\" explanations; and (ii) allows to derive some \"rules\" that can be used to improve the detection performance of the unlderying network intrusion detection system (NIDS). The proposed method leverages the intuition of \"slicing\" the (training) data into segments that represent a given group of (network) activities, e.g., web or database traffic; and then derive specific rules that describe these activities in an \"humanly understandable way\" and which can be used (if violated) to identify anomalies. The method is rigorously described and its effectiveness is assessed through various experiments on two well-known NID datasets, showing some advantages over existing baselines (which are mostly tailored for supervised ML).\n\nSTRENGTHS:\n+ Evaluation on two NID datasets\n+ Ablation Study\n+ Rigorous description\n+ Some design choices \"make sense\"\n+ The method is rigorously described\n+ Some ideas are correlated with proofs (in the appendix)\n+ The paper tackles an open research problem (on the surface) for which limited research is available\n+ Evaluation considering various parameter settings\n+ The results show improvement over baselines\n\n\nThe paper addresses an intriguing research problem (\"explainable unsupervised machine learning\") which has high relevance in some real-world applications of ML (i.e., network intrusion detection). The proposed method is rigorously described: some intuitions are sensible, and for others there are proofs provided which increase the overall soundness of the proposed methodology. The evaluation (which is reproducible, since the code is disclosed and the datasets are publicly available) shows improvement over prior work (most of which, however, was not designed to work in the given setting).\n\nFinally, the main novelty of this paper resides in addressing an underexplored problem.\n\nWEAKNESSES:\n## High Level\n\n- Unconvincing motivation\n- Excessively optimistic assumptions\n- Flawed dataset and poor setup\n- Questionable metrics \n- Unclear \"understanding\" assessment\n\nDespite some strengths, the paper suffers from various issues which prevent me from recommending inclusion in NeurIPS'23 program. Specifically, though the paper seeks to address a \"real-world problem\" (i.e., how security operators deal with ML-NIDS that are not explainable), the proposed method relies on assumptions that can hardly be met in the real world---thereby decreasing the overall significance of the findings. \n\nAnother problem pertains the soundness of the experimental procedure, which casts doubts on the validity of the results and, hence, of the conclusions drawn from the paper. \n\nFinally, from a \"novelty\" perspective, the techniques used to address the problem tackled by this paper may be (somewhat) novel, but they can be redundant in light of some well-known considerations of the NID context.\n\n## Low Level\n\nI will provide an extensive description of the abovementioned weaknesses below. I will directly quote some statements made in the paper, and I will also attempt to provide suggestions and plenty of references to support my points.\n\n\n\n\n### Unconvincing motivation\n\nAlbeit I acknowledge that the goal of providing \"explainable ML\" is relevant, I do not see a strong incentive in providing \"explainable _unsupervised_ ML methods\". This is because practitioners are well-aware that such methods are bound to be inaccurate, since they assume a notion of \"normality\". In a sense, the \"explanation\" for a raised anomaly is that a given data-point deviates from such normality; even practitioners acknowledge this as a fact [D]. This of course leads to a huge amount of false positives, which must be manually triaged _regardless of the explanations_. \n\nIn light of this, I did not find a compelling argument (provided in the paper) in favor of focusing on unsupervised ML --- or, at least, not in the context assumed by this paper. Perhaps a potential avenue is using said explanations for \"attack attribution\" [H], but this is never made explicit.\n\n### Optimistic assumptions\n\nThe method makes some strong assumptions that are too optimistic in real settings. The problem is that the main contribution (allegedly) is addressed at real-world deployments of ML.\n\n> We consider the anomaly detection tolerant of noisy data, but their proportion in training dataset is small and we do not have any ground truth labels of the training data.\n\nThis is a bold assumption which almost trivialized the entire problem of anomaly detection. If one is confident of having a (large) training set containing \"certain\" benign samples, one can use it as basis for supervised ML by enriching it with malicious samples (e.g., [I]). Unfortunately, in reality, obtaining such a \"clean\" training set requires huge manual effort [D] (and so is determining the threshold which allows for a manageable amount of alerts in operational contexts). Do note that the experimental evaluation assumes a \"perfect\" labelling of the normal samples in the training data (there is no mention of \"noisy\" data).\n\nAlso, consider the following statement\n\n> For example, a server supports multiple services such as web, email and database. The representations of these services can be disparate and located in different regions in feature space with little transition between the regions, making it infeasible to find a uniform rule set to accurately estimate the original model\n\nIn theory, this is correct. In practice, however, addressing this problem is trivial: \"web\", \"email\" and \"database\" are all services that can be easily separated by looking at the source ports of the corresponding communications (potentially with an additional filter based on the IP address). Indeed, one must remember that these methods must be setup and deployed by network administrators---who are well aware of what services are running in their organization.\n\n(note that even though here I have criticized the specific example, the consideration above can be extended to any type of potential \"multimodality\" that arises from network data)\n\nIn light of this, I fail to see any practical reason in favor of the method proposed in the paper. Perhaps the authors can provide a clear use-case in which the \"multimodality\" of the data is truly problematic to model.\n\n### Flawed dataset (and poor experimental setup)\n\nThe proposed method is evaluated on two datasets for Network Intrusion Detection (NID), one being the CIC-IDS17. Unfortunately, this dataset is flawed [J, K]. Note that [J] came out in 2021, and it has already been well-received by the NID community, so it is concerning that this paper (which has been submitted to a top-venue such as NeurIPS) performs the experiments on the \"flawed\" variant of this dataset---especially given that a \"fixed\" version exists (provided in [J]). \n\nFurthermore, note that [J] clearly stated that some features in this dataset (both the original and fixed version) are redundant for NID purposes (this is especially the case for \"flow-based\" NID [I, L]): however, this paper does not provide any detail about whether precautions have been taken to clean the feature set from \"obvious\" features (note that this also entails the ToN-IoT dataset)\n\nFinally, it is unclear whether the experiments have been repeated a sufficient amount of times to derive statistically significant conclusions (as recommended in [B]). The paper states:\n\n> The datasets are randomly split by the ratio of 6:2:2 for training, validation and testing\n\nThe \"random split\" may bias the results. \n\nIn light of this, the experimental setup may not be sound, thereby questioning the overall results. I invite the authors to look into other datasets as well (the works referenced in this review provide plenty of suggestions).\n\n\n### Questionable evaluation metrics\n\nI was unable to find a link between the metrics used in the evaluation and the overarching goal of the paper.\n\nAccording to the Introduction:\n\n> we can present the inferential process of the black-box model in a human-understandable way, and build a surrogate rule-based model for online defense at the same time.\n\nHowever, there is nothing in the evaluation that allows one to measure whether the ```inferential process of the black-box model``` is presented in a ```human-understandable way```. This is because the evaluation appears to merely focus on the sheer detection performance of the \"surrogate\" model.\n\nTo re-quote a statement in the introduction (with which I fully agree!):\n\n> security operators tend to trust human-understandable rules rather than the unintuitive outputs such as labels and numeric values from the complex and incomprehensible models.\n\nThere is little in the evaluation that makes me believe that security operators would appreciate the explanations provided by the proposed method. \n\nI also have some concerns on the way these metrics are described:\n\n> Fidelity (FD), i.e., the ratio of input samples on which the predictions of original model and surrogate model agree over the total samples, which indicates the extent to which humans can trust the explanations\n\nI disagree: at best, FD denotes how much the surrogate and original models \"agree\". This says nothing about whether a human can \"trust\" the explanation in the general sense. Indeed, the quoted statement would be correct under the assumption that the original model's explanation are \"correct\". \n\n> Robustness (RB), i.e, the persistence to withstand small perturbations of the input that do not change the prediction of the model\n\nI am confused here: what is the ```model``` whose prediction should not change? I.e., is it the \"original\" or the \"surrogate\"?\n\n### Unclear method for \"understanding\"\n\nThis issue follows the previous one. The \"human-understandable way\" to interpret the rules generated by the proposed method is assessed in Section 6.3. However, there is a lack of details in this Section, and the \"conclusions\" appear to be drawn from purely subjective statements.\n\nConsider the following:\n\n> Such results are easy to be interpreted.\n\nHow is this determined?\n\n> These explanations are in line with how security experts recognize the attack data.\n\nNo reference is provided.\n\nPlus, there is a mismatch with Table 4 and the overall assumptions made in the paper. Specifically, the paper assumes an \"unsupervised\" AD task, wherein the specific attack is not known. In light of this, how can a human derive that, e.g., (taken from Table 4):\n\n> DDoS attacks use packets of small sizes to achieve\nasymmetric resource consumption on the victim side,\nand send packets at a high rate to flood the victim\n\n? At most, a human sees some rules (e.g., ```ps_mean > 101.68, iat_mean > 0.063, dur > 12.61```) which -- when not met -- trigger some anomalies. But from here to stating that these anomalies relate to \"DDoS attacks\" there is a long way. For instance, \"violation\" of these rules can very well be related to a port-scanning activity (which can very well be a benign event, e.g., a new host which scans the network after joining it).\n\n\n### Some bold statements in the Introduction\n\nBelow is a list of statements made in the Introduction for which I have serious concerns.\n\n\n> it requires no attack/malicious data during the training (i.e., zero-positive learning), which are typically much more sparse and difficult to obtain in contrast with benign data;\n\nActually, obtaining \"certain\" benign data can be even harder than acquiring malicious data (see [B]). As a matter of fact, real-world deployments of ML entail very coarse \"labelling\" schemes [A].\n\n> it does not fit any known threats, enabling better detection on unforeseen anomalies.\n\nBut this also means that some of the raised anomalies have nothing to do with attacks---leading to \"alert fatigue\" (see [C,D,E]).\n\n> being self-explained and accurate for high-stake security applications.\n\nPlease define \"accurate for high-stake security applications\". Even non-surrogate models have a huge margin of error [C].\n\n> Accuracy Loss (CH3). [...] In this case, though these methods can provide model explanation, they cannot meet the need of online deployment which requires high detection accuracy in security applications.\n\nI disagree. The \"surrogate\" model has a single objective: provide \"global\" explanations. Therefore, even though such surrogate model may have worse \"accuracy\", it is the \"larger\" model that is meant to be deployed, while the \"surrogate\" is only used for explanation tasks.\n\n> We observe that an important reason why simple surrogate models are ineffective is that they cannot learn well about the complex data distribution in high-dimensional space.\n\nOn what grounds is this \"observation\" made? Is it just an educated guess, does it derive from original experiments, or is it drawn from prior research?\n\n\n> The extracted rules outperform prior work in terms of diverse metrics including fidelity, robustness, true positive rate and true negative rate\n\nThis contradicts what was written at the beginning of the introduction: ```security operators tend to trust human-understandable rules rather than the unintuitive outputs such as labels and numeric values from the complex and incomprehensible models.```. In what way are the metrics mentioned above different from \"numeric values from the complex and incomprehensible models\"?\n\n> which meets the demand of improving human trust in black-box models and maintaining high detection accuracy for online deployment.\n\nThis is a self-claim: there is no evidence provided that ```fidelity, robustness, tpr, tnr``` meet the described ```demand``` (in terms of explainability).\n\n\n\n\n### Some additional issues:\n\nThe following is a list of miscellaneous issues I found in this submission; note that these issues are not necessarily \"minor\".\n\n* Some relevant works have not been considered. The extensive study carried out in [F] reveals, e.g., [G] -- which specifically focuses on providing \"explainable unsupervised models\".\n* ```furthur``` typo\n* I found the following statement to be misleading (and also unnecessary): ```we refer to adversarial attacks [44] and propose a method to approximate the optimal direction to explore the decision boundary```\n* ```Further, most of their true positive rates cannot meet the requirement of using their rules for online defense, as the sheer amount of attack data they miss can cause huge damages.``` please provide facts instead of writing vague statements such as \"can cause huge damages\". Even a single false negative can lead to \"huge damages\" [D].\n* I did not find any measure of the \"runtime\" used to train/test the proposed method. \n* Did the evaluation _really_ take place on a server having an RTX3090 with 8GB of VRAM? (given that the RTX3090 has 24GB of VRAM?)\n* Did the evaluation _really_ take place on a server having an Intel Xeon with only 8GB of RAM?\n* To my understanding, the method assumes that the data-points represent \"network flows\" (such is the data contained in the considered datasets). However, there is a huge variability in the ways such flows can be generated (see [I,M]). What would happen if the netflow generator is different? \n\n#### EXTERNAL REFERENCES\n\n[A]: Van Ede, Thijs, et al. \"Deepcase: Semi-supervised contextual analysis of security events.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.\n\n[B]: Apruzzese, Giovanni, Pavel Laskov, and Aliya Tastemirova. \"SoK: The impact of unlabelled data in cyberthreat detection.\" 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P). IEEE, 2022.\n\n[C]: Alahmadi, Bushra A., Louise Axon, and Ivan Martinovic. \"99% False Positives: A Qualitative Study of {SOC} Analysts' Perspectives on Security Alarms.\" 31st USENIX Security Symposium (USENIX Security 22). 2022.\n\n[D]: Apruzzese, Giovanni, et al. \"The role of machine learning in cybersecurity.\" Digital Threats: Research and Practice 4.1 (2023): 1-38.\n\n[E]: Hassan, Wajih Ul, et al. \"Nodoze: Combatting threat alert fatigue with automated provenance triage.\" network and distributed systems security symposium. 2019.\n\n[F]: Nadeem, Azqa, et al. \"Sok: Explainable machine learning for computer security applications.\" arXiv preprint arXiv:2208.10605 (2022) [to appear in EuroS&P'23]\n\n[G]: Wickramasinghe, Chathurika S., et al. \"Explainable unsupervised machine learning for cyber-physical systems.\" IEEE Access 9 (2021): 131824-131843.\n\n[H]: Nisioti, Antonia, et al. \"From intrusion detection to attacker attribution: A comprehensive survey of unsupervised methods.\" IEEE Communications Surveys & Tutorials 20.4 (2018): 3369-3388.\n\n[I]: Apruzzese, Giovanni, Luca Pajola, and Mauro Conti. \"The cross-evaluation of machine learning-based network intrusion detection systems.\" IEEE Transactions on Network and Service Management (2022).\n\n[J]: Liu, Lisa, et al. \"Error Prevalence in NIDS datasets: A Case Study on CIC-IDS-2017 and CSE-CIC-IDS-2018.\" 2022 IEEE Conference on Communications and Network Security (CNS). IEEE, 2022.\n\n[K]: Engelen, Gints, Vera Rimmer, and Wouter Joosen. \"Troubleshooting an intrusion detection dataset: the CICIDS2017 case study.\" 2021 IEEE Security and Privacy Workshops (SPW). IEEE, 2021.\n\n[L]: Arp, Daniel, et al. \"Dos and don'ts of machine learning in computer security.\" 31st USENIX Security Symposium (USENIX Security 22). 2022.\n\n[M]: Vormayr, Gernot, Joachim Fabini, and Tanja Zseby. \"Why are my flows different? a tutorial on flow exporters.\" IEEE Communications Surveys & Tutorials 22.3 (2020): 2064-2103.\n\nQUESTIONS:\nI thank the authors for submitting their paper to NeurIPS'23. Despite my stark criticism, I liked reading it. This is why I endorse the authors in answering the following questions to the best of their abilities: if provided with compelling evidence, I may be willing to raise my score.\n\n* Q1) Given that CIC-IDS17 is flaeed, how much time would it be necessary to re-do the experiments?\n* Q2) What is the runtime required to train/test the proposed method, and how does it compare to existing methods?\n* Q3) Are the details of the experimental platform correct?\n* Q4) What features have been used to train the models?\n* Q5) How was Table 4 derived?\n* Q6) Have the experiments been carried out multiple times to accout for bias?\n* Q7) Can the authors provide convincing evidence of the \"necessity\" of the proposed method _in practice_?\n* Q8) How is the \"robustness\" measured?\n* Q9) How long would it take to assess the efficacy of the proposed method under varying percentages of \"noisy\" data?\n* Q10) Does the method _require_ netflow data to work? And, if so, how would the resulting performance change if the flows are generated in a different way?\n\n\nPlease refer to the main review for additional information\n\nLIMITATIONS:\nThe \"limitations\" are discussed in the supplementary material. Unfortunately, they do not mention that, e.g., the proposed method relies on overly optimistic assumptions (for real deployments), nor that the \"understanding\" is likely drawn by subjective intuitions. Also, I see no reason for mentioning that the proposed method does not work well on images, unless this data-type becomes prevalent in NID (and I do not see this happening anytime soon).\n\n==========\n\n# Update after rebuttal\n\nI have increased my score from a 3 (Reject) to a 5 (Borderline Accept); I have also increased the scores for Contribution and Soundness (from a 2 to a 3). The improvements are due to the paper now being much more truthful as to what it does, as well as by accounting for a \"non-flawed\" version of the considered datasets. Ultimately, I do not see any clear reason for \"rejection\", but I also do not feel strong reasons to accept this---aside from tackling a highly practical area of application of ML, which could serve as inspiration for future work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose and Distribution Decomposition Rules and Boundary\nInference Rules to make black boxes more interpretable.\n\nThey use Interior Clustering Tree IC-tree to find distribution\ndecomposition rules.  The IC-tree algorithm splits the data on a\nfeature value at each node recursively.  The feature value that\nmaximizes the gain in gini index before and after the split is chosen\nat each node.  Gini index is 2p(1-), where p be is the average output\nof the anomaly detection model f(x), which is the trained neural\nnetwork model.  The path to each leaf forms a distribution\ndecomposition rule and each leaf represents a data subset that is in\nthe same distribution.\n\nThe Compositional Boundary Exploration (CBE) algorithm finds a minimal\nhypercube that encloses the normal data in each distribution (leaf\nnode).  CBE uniformly samples data points on each hyperplane as initial\nexplores (centriods), then samples from a Gaussian distributions\npoints as auxiliary explorers near the centroids within some radius.\nThe samples are used to query the model to find samples with the\nlowest probability to be normal (negative examples), these are called\ncandidate explorers.  They use Fast Gradient Sign Method to perturb\nthe explorers.  Since the loss function is not known from the model,\nthey approximate the gradient by the gradient between the model\noutputs of an initial explorer and an auxiliary explorer with respect\nto the different in the samples.  The explores are perturbed until the\nf() indicates it is normal (less than threshold phi).  The\ncorresponding feature values forms the hyperplanes for the boundaries.\n\nThey evaluate 5 existing extraction methods with 4 black-box models and\n2 datasets.  In addition to TPR and TNR, they measure Fidelity (which\ncalculates agreement between the black-box model and their extracted\nrules) and Robustness (which estimates small input perturbations do not\nchange the model's output).  Their proposed method generally\noutperforms existing techniques.  Ablation studies were performed to\naccess the contribution of each component.\n\nI have read the authors' responses and commented on them.\n\nSTRENGTHS:\n1.  The proposed algorthms yield interpretables rules with high\n    fidelity of the black-box models.\n\n2.  Empirical results indicate the proposed algorithms generally\n    outperform existing ones.\n\n3.  The paper is generally well written.\n\nWEAKNESSES:\n1.  Some ideas could be clarified, see questions below.\n\nQUESTIONS:\n1.  line 220: What is the reasoning for the initial vs auxiliary explorers?    \n\n2.  line 227: What is the reasoning for \"minimal probability of being normal\"?\n     Why not choose candidates that are close to the threshold $\\varphi$ and then\n     try to get to the other side of the threshold?\n\n3.  line 227: Does each of the dimension/feature have two candidate\n    explorers so that they yield two boundary (axis-parallel\n    hyperplane) in each dimension?  How do you guarantee one boundary\n    on each side in one dimension?\n\n4.  line 239: should f() be greater than $\\varphi$ instead of less than?  I think\n     you try to go from low f() to high f() to cross $\\varphi$.\n\n5.  line 323: \"directly using hypercubes as rules\", how are these\n    hypercubes found?  Are they from the IC-tree?  A hypercube can\n    be found for instances in each leaf by finding the \"min and max\" value\n    of each feature/dimension via a fitted Gaussian.  If that is not what you did, adding\n    it as a comparison would be interesting.\n\n6.  line 325: \"Though using the K-Means can reach similar results, it\n    cannot be expressed by axis-aligned rules with high interpretability\n    and deployability as the IC-Tree can achieve.\"\n\n    Using CBE, axis-aligned rules can be found with clusters found by\n    K-means.  If I understand correctly, the IC-tree can be ignored\n    after the boundary rules are extracted by CBE.\n\nLIMITATIONS:\nLimitations of the proposed approach seem to be discussed as\ndirections in future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

comments: SUMMARY:
The paper presents a technique to build a decision tree (DT) that uses the predictions from any unsupervised anomaly detection algorithm to split at the nodes when constructing the DT. This is named as the Interior Clustering Tree. The DT will be used to extract interpretable rules.

STRENGTHS:
1. The approach is useful for explainable anomaly detection and is an important problem domain.

WEAKNESSES:
1. Very similar to [1]. The difference is that in [1] the density is based on KDE, whereas, in the present work the density is based on the anomaly detector. So the technical novelty is less.

2. Section 5: Lack of clarity; the procedure should be presented as algorithmic steps.

3. We need more than just two datasets for empirical validity. These datasets also seem 'easy' since detection accuracy is very high for all anomaly detection algorithms. With easy datasets, the rules extracted would tend to be short and nicely interpretable. On the other hand, with harder datasets, the rules might be much more complex and difficult to interpret. I am interested in the types of rules extracted for the harder datasets.

References:

[1] Sarah Itani, Fabian Lecron, Philippe Fortemps, A one-class classification decision tree based on kernel density estimation, Applied Soft Computing, Volume 91, 2020 (https://arxiv.org/abs/1805.05021)

QUESTIONS:
1. What is the order of computational complexity or the execution time of the algorithm? Is the also affected by the curse of dimensionality?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05