SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruoyu Li $\S$ †, Qing Li $*$ †, Yu Zhang $\S$ , Dan Zhao†, Yong Jiang $\S$ †, Yong Yang‡  
 $\S$ Tsinghua University, China; †Peng Cheng Laboratory, China  
 $\S$ Tsinghua Shenzhen International Graduate School, China  
 $\S$ Tencent Security Platform Department, China  
{1iry19,yu-zhang23}@ mails.tsinghua.edu.cn; {liq,zhaod01}@pcl.ac.cn  
jiangy@sz.tsinghua.edu.cn; coolcyang@tencent.com
Many security applications require unsupervised anomaly detection, as malicious data are extremely rare and often only unlabeled normal data are available for training (i.e., zero-positive). However, security operators are concerned about the high stakes of trusting black-box models due to their lack of interpretability. In this paper, we propose a post-hoc method to globally explain a black-box unsupervised anomaly detection model via rule extraction. First, we propose the concept of distribution decomposition rules that decompose the complex distribution of normal data into multiple compositional distributions. To find such rules, we design an unsupervised Interior Clustering Tree that incorporates the model prediction into the splitting criteria. Then, we propose the Compositional Boundary Exploration (CBE) algorithm to obtain the boundary inference rules that estimate the decision boundary of the original model on each compositional distribution. By merging these two types of rules into a rule set, we can present the inferential process of the unsupervised black-box model in a human-understandable way, and build a surrogate rule-based model for online deployment at the same time. We conduct comprehensive experiments on the explanation of four distinct unsupervised anomaly detection models on various real-world datasets. The evaluation shows that our method outperforms existing methods in terms of diverse metrics including fidelity, correctness and robustness.
In recent years, machine learning (ML) and deep learning (DL) have revolutionized many security applications such as network intrusion detection [1-3] and malware identification [4, 5] that outperform traditional methods in terms of accuracy and generalization. Among these works, unsupervised anomaly detection becomes more promising, which detects malicious activities by the deviation from normality. Compared to supervised methods, this type of method is more desirable in security domains as 1) it hardly requires labeled attack/malicious data during the training (i.e., zero-positive learning), which are typically much more sparse and difficult to obtain in contrast with benign data; 2) it does not fit any known threats, enabling better detection on unforeseen anomalies.
Due to the black-box nature of these models, ML/DL models are usually not directly interpretable and understandable. Many local explanation methods [6-10] have attempted to interpret the models by presenting feature importance of the decision for a single point. However, globally explaining black-box models, especially using rule extraction to characterize the whole decision boundaries, is particularly desirable in security systems since it can provide the following benefits:
*Corresponding author: Qing Li.
Trust over High Stakes. To minimize the chance of errors and potential losses, security operators tend to trust human-understandable rules rather than unintuitive outputs such as labels and numeric values from complex and incomprehensible black-box models.
Online Defense. Interpreting black-box models into rules with high fidelity enables easy integration with the majority of defense tools that use rule-based expressions (e.g., iptables [11], Snort [12]), thus allowing for deployment of online defense with extraordinary efficiency (e.g., Tbps throughput [13]).
Existing global ML/DL explanation approaches are mostly proposed for supervised models. Little research has been done on explaining unsupervised anomaly detection, which faces several challenges:
Unlabeled One-class Data (CH1). Supervised explanation methods [14-16] need labeled data of both positive and negative classes to determine the decision boundaries of black-box models. This requirement goes against unsupervised anomaly detection's advantage of not requiring attack data.
Lack of Surrogate Models (CH2). Global methods typically use an inherently explainable model to mimic the black-box model, such as decision trees [15, 16] and linear models [17], which are all supervised models. However, there lacks a proper surrogate model that is unsupervised and can satisfy the properties of being self-explained and well-performed for high-stake security applications.
Accuracy Loss (CH3). A common problem with global methods is that the surrogate model suffers from the loss of accuracy since it simplifies the original model [18]. In this case, though these methods can provide model explanation, they cannot meet the need of online deployment which requires high detection accuracy in security applications.
We observe that an important reason why simple surrogate models are ineffective is that they cannot learn well about the complex data distribution in high-dimensional space. Specifically, even one-class data may be multimodal, i.e., the overall distribution consists of multiple compositional distributions. For example, the normal activities of a host may consist of many services (e.g., web, database) and are for various purposes, and they could present considerable differences in feature space.
In light of this, this paper proposes an accurate and efficient divide-and-conquer method to globally interpret unsupervised anomaly detection. First, we propose the concept of distribution decomposition rules that cut the feature space into multiple subspaces, each of which encloses a compositional distribution. To obtain such rules, we design a new tree model called Interior Clustering Tree that extends the CART decision tree in terms of splitting criteria and can fully work in an unsupervised manner. Second, we propose the Compositional Boundary Exploration algorithm to obtain the boundary inference rules that estimate the decision boundary of the original model on each compositional distribution. To accurately and efficiently find such rules, this algorithm starts from hypercube-shaped rules and uses an approximation for gradient ascent of model prediction to find the optimal direction of iterations. By merging the distribution decomposition rules and the boundary inference rules into a rule set, we can present the inferential process of the black-box model in a human-understandable way, and build a surrogate rule-based model for online defense at the same time.
During the experiment, we use four different unsupervised anomaly detection models well trained on three benchmark datasets, and evaluate our method with five distinct explanation methods as baselines. The experiment shows that our method can extract interpretable and accurate rules from black-box models. The extracted rules outperform prior work in terms of diverse metrics including fidelity, robustness, true positive rate and true negative rate, meeting the demand of improving human trust in black-box models and maintaining high detection accuracy for online deployment. Our code is available at https://github.com/Ruoyu-Li/UAD-Rule-Extraction.
To combat persistent emergence of new attacks in cyberspace, recent security applications [1-3, 19-21] make heavy use of unsupervised models to detect unknown anomalies, such as one-class classifiers [22-24], Isolation Forests [25, 26], autoencoders and variational autoencoders [27]. Despite many unsupervised model-based approaches have achieved good detection rates, security operators are still concerned about the semantic gap between black-box model prediction and human understanding, considering the risks of the great cost incurred by bad decisions [10]. To resolve such concerns, explainable AI (XAI) has been applied to anomaly detection [28-30]. For example, Kauffmann et al. propose a decomposition method to explain anomalies of one-class SVMs [28]. Philipp et al. present
an explainable deep one-class classification method called Fully Convolutional Data Description [30]. However, these methods are specific to a limited range of models and not versatile enough to accommodate the vastly heterogeneous models of unsupervised anomaly detection.
Some prior work also incorporates popular model-agnostic explanation methods, such as LIME [6], SHAP [7] and their variations [8], and applies them to explain unsupervised models [31-33] and security applications [9, 34, 35, 10]. These methods typically use sparse linear models to interpret predictions by estimating feature importance. Guo et al. propose a method named LEMNA that uses a fused lasso to explain malware classification [9]. Sipple uses Integrated Gradients [36] to attribute anomalies of IoT device failure [34]. Nonetheless, these methods can only interpret one data point at a time (i.e., local explanation) but not reveal the complete decision-making process of a model.
To fully understand how black-box models work and safely deploy them, the most appropriate method is model-agnostic global post-hoc explanation. It aims to match the predictions of any well-trained models with an inherently interpretable explainer, such as decision trees [15, 16], symbolic rules [14], sparse linear models [17] and decision lists [37]. In [15], the authors construct global explanations of complex black-box models in the form of a decision tree approximating the original model. Jacobs et al. propose a framework that takes an existing ML model and training dataset and generates tree models to interpret security-related decisions [16]. However, most of these methods are only suitable for interpreting supervised models that have labeled data of all classes, which are often unavailable. Though work like [38] can extract rules from unsupervised anomaly detection models, it still assumes that enough outliers exist in the training dataset judged by the black-box model so as to determine its decision boundary. This assumption may not hold in practice if a model has great generalization and can achieve a low false positive rate on normal data (e.g., [1]).
Some recent studies aggregate several local explanation models into near-global explanation [39-41]. However, this type of method is inherently computationally challenging when data volumes are large and has to make trade-offs between fidelity and coverage. While techniques like knowledge distillation can also realize model transformation to reduce complexity and promote interpretability [42, 43], the fundamental purpose of these efforts is to compress models while ensuring accuracy rather than explaining the original models with high fidelity.
3 Overview
Let  $\mathcal{X} \subseteq \mathbb{R}^d$  be the variable space of  $d$ -dimensional features;  $\pmb{x}$  and  $x_{i}$  denote a data sample and the  $i$ -th dimension of the data sample. We give the following definitions for the rest of the paper:
Definition 1 (Unsupervised Anomaly Detection). Given unlabeled negative data (i.e., normal data)  $X$  sampled from a stationary distribution  $\mathcal{D}$  for training, an unsupervised model estimates the probability density function  $f(x)\approx \mathrm{P}_{\mathcal{X}\sim \mathcal{D}}(x)$ , and detects an anomaly via a low probability  $f(x) < \varphi$ , where  $\varphi >0$  is a threshold determined by the model itself or by humans.
It is noted that the threshold  $\varphi$  is a non-zero value, meaning that the model inevitably generates false positives, which is a common setting in most of the works [1-3] even though the false positive rate can be very low. Besides, the normal data may occasionally be contaminated or handled with errors. We consider the anomaly detection tolerant of noisy data, but their proportion in training dataset is small and we do not have any ground truth labels of the training data.
Definition 2 (Global Explanation by Rule Extraction). Given a trained model  $f$  with its anomaly threshold  $\varphi$  and the training set  $\mathbf{X}$ , we obtain an in-distribution rule set  $\mathcal{C} = \{C_1, C_2, \ldots\}$  that explains how the model  $f$  profiles the distribution of normal data. A rule  $C = \ldots \wedge (x_i \odot v_i) \wedge \ldots \wedge (x_j \odot v_j)$  is a conjunction of several axis-aligned constraints on a subset of the feature space, where  $v_i$  is the bound for the  $i$ -th dimension and  $\odot \in \{\leq, >\}$ .
Let  $\pmb{x} \in C$  indicate that a data sample satisfies a rule. From  $\mathcal{C}$ , we can build a surrogate model  $h_{\mathcal{C}}(\pmb{x})$  whose inference is to regard a data sample that cannot match any of the extracted rules as anomalous:
$$
h _ {\mathcal {C}} (\boldsymbol {x}) = \neg (\boldsymbol {x} \in C _ {1}) \wedge \neg (\boldsymbol {x} \in C _ {2}) \wedge \dots , C _ {i} \in \mathcal {C}. \tag {1}
$$
Our Goal. We expect the extracted rules to have a high fidelity to the original model, that is, a similar coverage of normal data (i.e., true negative rate), and a similar detection rate of anomalies (i.e., true
(a) The unlabeled data
(b) Compositional distributions
Figure 1: A high-level illustration of our method. The small circles are unlabeled normal data. The dashed curves are the decision boundary of the black-box model. The vertical/horizontal lines in (b) and (c) are the distribution decomposition rules.
(c) Process of the CBE algorithm
(d) The final rule set
positive rate). To this end, we formulate our objective as follow:
$$
\underset {\mathcal {C}} {\arg \min } \mathcal {L} _ {\mathcal {X} \sim \mathcal {D}} (\mathcal {C}, f, \varphi) + \mathcal {L} _ {\mathcal {X} \sim \mathcal {D}} (\mathcal {C}, f, \varphi). \tag {2}
$$
3.2 Methodology Overview
To minimize the first item in Equation (2), suppose the training data  $X$  can well represent the distribution  $\mathcal{D}$ , a straightforward approach is to find the bound of  $X$  as rules, such as using a hypercube to enclose the data samples which can easily achieve the minimization of the partial loss  $\mathcal{L}_{x\in X}(\mathcal{C},f,\varphi) = 0$ . However, as  $\mathcal{D}$  is not a prior distribution and we do not have labeled abnormal samples, the second item  $\mathcal{L}_{\mathcal{X}\sim \mathcal{D}}(\mathcal{C},f,\varphi)$  is neither deterministic nor estimable unless we create sufficient random samples and query  $f$ , which is challenging given the high-dimensional space of  $\mathcal{X}$ .
As prior studies [21, 34] suggest, normal data are typically multimodal, i.e., the overall distribution is formed by multiple compositional distributions. For example, a server supports multiple services such as web, email and database. The representations of these services can be disparate and located in different regions in feature space with little transition between the regions, making it infeasible to find a uniform rule set to accurately estimate the original model. An example is illustrated in Figure 1a.
Based on this intuition, we propose a divide-and-conquer approach. First, we propose an Interior Clustering Tree model (Section 4) to find the distribution decomposition rules, which cut the feature space into subspaces so that each subspace contains data belonging to the same compositional distribution, as shown in Figure 1b. Then, we design a Compositional Boundary Exploration algorithm (Section 5) to explore the decision boundary on each compositional distribution, as depicted in Figure 1c. Particularly, the algorithm starts from the minimal hypercube that encloses all data of the distribution, and finds the boundary by recursively extending the boundary following the optimal direction guided by a gradient approximation. Upon obtaining the decision boundary of a distribution, the corresponding boundary inference rule can be extracted. Last, the rule set that globally approximates the original model can be obtained by merging the distribution decomposition rule and the boundary inference rule of each compositional distribution, as illustrated in Figure 1d. We formally define the distribution decomposition rule and the boundary inference rule as follows.
Definition 3 (Distribution Decomposition Rule). Denoted by  $C_k^I$  that decomposes the overall distribution of normal data  $\mathcal{D}$  into  $K$  compositional distributions, i.e.,  $\mathrm{P}_{\mathcal{X} \sim \mathcal{D}}(\boldsymbol{x}) = \sum_{k=1}^{K} \phi_k \cdot \mathrm{P}_{\mathcal{X} \sim \mathcal{D}_k}(\boldsymbol{x} | \boldsymbol{x} \in C_k^I)$  where  $\phi_k$  denotes the weight of each compositional distribution, so that a data sample  $\boldsymbol{x} \sim \mathcal{D}_k$  has significantly small probability of belonging to other distributions.
Definition 4 (Boundary Inference Rule). Denoted by  $C_k^E$  that estimates the decision boundary of the original model for each distribution  $\mathcal{D}_k$ , i.e.,  $\underset{C_k^E}{\arg \min} \mathcal{L}_{\mathcal{X} \sim \mathcal{D}_k}(C_k^E, f, \varphi) + \mathcal{L}_{\mathcal{X} \sim \mathcal{D}_k}(C_k^E, f, \varphi)$ .
With the definition of these two types of rules, we translate the objective in Equation (2) to the following objective as our intuition indicates. We give a proof of this proposition in the appendix.
Proposition 1. The original objective can be estimated by finding the union of the conjunction of distribution decomposition rules and boundary inference rules for each compositional distribution:
$$
\bigcup_ {k = 1} ^ {K} \arg \min  _ {C _ {k}} \mathcal {L} _ {\mathcal {X} \sim \mathcal {D} _ {k}} \left(C _ {k}, f, \varphi\right) + \mathcal {L} _ {\mathcal {X} \sim \mathcal {D} _ {k}} \left(C _ {k}, f, \varphi\right), \text {w h e r e} C _ {k} = C _ {k} ^ {I} \wedge C _ {k} ^ {E}. \tag {3}
$$
4 Interior Clustering Tree
To obtain distribution decomposition rules, we first need to decide how to define a compositional distribution  $\mathcal{D}_k$ . We notice that, though we do not have labeled data, we can use the output of the original model that estimates the overall distribution  $\mathcal{D}$  as a criterion for decomposition.
Proposition 2. If two data samples  $\pmb{x}^{(i)}$  and  $\pmb{x}^{(j)}$  belong to the same distribution  $\mathcal{D}_k$ , the difference of their probabilities belonging to  $\mathcal{D}$  will be less than  $\epsilon$ , where  $\epsilon$  is a small constant.
Proof. Recall the definition of distribution decomposition rules and compositional distributions. Since the two data samples belong to the same distribution  $\mathcal{D}_k$ , the probability of belonging to other distributions  $\mathrm{P}_{\mathcal{X} \sim \mathcal{D}_l}(\boldsymbol{x})$  is near zero for  $l \neq k$ . Hence, the probability  $\mathrm{P}_{\mathcal{X} \sim \mathcal{D}}(\boldsymbol{x})$ , which is the weighted sum of the probability of belonging to all the compositional distributions, is approximately equal to  $\mathrm{P}_{\mathcal{X} \sim \mathcal{D}_k}(\boldsymbol{x})$  for both the data samples.
Based on this, we propose a tree-based model dubbed Interior Clustering Tree (IC-Tree), which extends the CART decision tree [44]. The main difference between IC-Tree and CART is that, rather than splitting data based on ground truth labels, IC-Tree uses the probability output by the original model as splitting criteria, enabling it to work in a completely unsupervised manner.
Node Splitting. Given the data  $N$  at a tree node, we first obtain the output of the anomaly detection model  $f(\pmb{x})$  for  $\pmb{x} \in N$ . Similar to decision trees, the node of an IC-Tree finds a splitting point  $s = (i, b_i)$  that maximizes the gain:
$$
s = \underset {s} {\arg \max } I (N) - \frac {\left| N _ {l} \right|}{\left| N \right|} I \left(N _ {l}\right) - \frac {\left| N _ {r} \right|}{\left| N \right|} I \left(N _ {r}\right), \tag {4}
$$
where  $b_{i}$  is the splitting value for the  $i$ -th dimension,  $N_{l}$  and  $N_{r}$  are the data split to the left and right child nodes,  $|N|$  denotes the number of data samples, and  $I$  is a criterion function such as Gini index  $I = 2p(1 - p)$  for binary classification with the probability of  $p$ . Specifically, we let  $p$  be the average output of the anomaly detection model, which can be interpreted as the expectation of the probability that the data belong to the same distribution:
$$
p = \mathbb {E} _ {\boldsymbol {x} \in \boldsymbol {N}} \left[ \mathrm {P} _ {\chi \sim \mathcal {D}} (\boldsymbol {x}) \right] = \frac {1}{| N |} \sum_ {\boldsymbol {x} \in N} f (\boldsymbol {x}). \tag {5}
$$
An IC-Tree continues to split nodes until it satisfies one of the following conditions: i) the number of data samples at the node  $|N| = 1$ ; ii) for any two of the data samples at the node  $\forall \pmb{x}^{(i)}, \pmb{x}^{(j)} \in N$ ,  $|f(\pmb{x}^{(i)}) - f(\pmb{x}^{(j)})| < \epsilon$ ; iii) it reaches a maximum depth  $\tau$ , which is a hyperparameter.
Distribution Decomposition Rule Extraction. A trained IC-Tree that has  $K$  leaf nodes  $(K\leq 2^{\tau})$  represents  $K$  distributions separated from the overall distribution  $\mathcal{D}$ . Suppose the  $k$ -th leaf node has a depth of  $\tau^\prime$ . A distribution decomposition rule that describes the  $k$ -th compositional distribution can be extracted by the conjunction of the splitting constraints from the root to the leaf node:
$$
C _ {k} ^ {I} = \left(x _ {i} \odot_ {1} b _ {i} \mid s _ {1} = (i, b _ {i})\right) \wedge \dots \wedge \left(x _ {j} \odot_ {\tau^ {\prime}} b _ {j} \mid s _ {\tau^ {\prime}} = (j, b _ {j})\right), \tag {6}
$$
where  $\odot$  is “ $\leq$ ” if the decision path goes left or “ $>$ ” if the decision path goes right.
5 Compositional Boundary Exploration
To accurately find the decision boundary of a detection model within each compositional distribution, we propose the Compositional Boundary Exploration (CBE) algorithm (described in Algorithm 1). The CBE algorithm uses the minimal hypercube that encloses the normal data of each compositional distribution as a starting point. Further, we refer to adversarial attacks [45] and propose a method to
approximate the optimal direction to explore the decision boundary, which makes the algorithm more efficient and accurate to estimate the decision boundary.
Starting from Hypercube (line 1). Let  $X_{k}$  denote the training data falling into the  $k$ -th leaf node of an IC-Tree that represents a compositional distribution. Recall the definition of boundary inference rules that target  $\min \mathcal{L}_{\mathcal{X} \sim \mathcal{D}_k}(C_k^E, f, \varphi) + \mathcal{L}_{\mathcal{X} \sim \mathcal{D}_k}(C_k^E, f, \varphi)$ . We use the minimal hypercube  $H_{k}$  as a starting point of boundary inference rules to bound every dimension of the data samples in  $X_{k}$  judged by the original model as normal, which obviously achieves  $\mathcal{L}_{\boldsymbol{x} \in \boldsymbol{X}_{k}}(H_{k}, f, \varphi) = 0$ . The minimal hypercube is enclosed by  $2 \times d$  axis-aligned hyperplanes, which can be characterized by the following rule:
$$
H _ {k} = \left(v _ {1} ^ {-} \leq x _ {1} \leq v _ {1} ^ {+}\right) \wedge \dots \wedge \left(v _ {d} ^ {-} \leq x _ {d} \leq v _ {d} ^ {+}\right), \tag {7}
$$
where  $v_{i}^{-} = \min (x_{i}|f(\pmb {x}) > \varphi ,\pmb {x}\in \pmb{X}_{k})$  and  $v_{i}^{+} = \max (x_{i}|f(\pmb {x}) > \varphi ,\pmb {x}\in \pmb{X}_{k})$ .