SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Explorer Sampling (line 4~6). The CBE algorithm explores the decision boundary of the original model by estimating the bound of one feature dimension at a time. For  $i$ -th dimension, we uniformly sample  $N_{e}$  data points on each hyperplane of the hypercube, i.e.,  $e^{(1)}, \ldots, e^{(N_{e})} \in H_{k} \wedge (x_{i} = v_{i}), v_{i} \in \{v_{i}^{-}, v_{i}^{+}\}$ , which are called the initial explorers for this hyperplane. For an initial explorer  $e$ , we further sample  $N_{s}$  auxiliary explorers near it from a truncated multivariate Gaussian distribution denoted by  $\mathcal{N}(e, \Sigma, i)$ . Particularly, the center of sampling is the explorer  $e$  and the radius of sampling is constrained by the covariance matrix  $\Sigma = \mathrm{diag}(\rho | v_{1}^{+} - v_{1}^{-}|, \ldots, \rho | v_{d}^{+} - v_{d}^{-}|)$ , where  $\rho$  is a hyperparameter, and the sampling on  $i$ -th dimension is half-truncated to only keep the distribution outside the hypercube as we desire to extend the boundary. With  $N_{e} \times N_{s}$  auxiliary explorers in total, we query the original model and use Beam Search to select  $N_{e}$  samples with the minimal probability of being normal as the candidate explorers for the next iteration.
Gradient Approximation (line 7~9). Though we have obtained  $N_{e}$  candidate explorers in the previous step, using them directly for the next iteration does not guarantee the optimal direction of movement towards the decision boundary. To find the optimal direction, we utilize the Fast Gradient Sign Method [45] that employs gradient ascent to find the direction of feature perturbation. However, we do not know the loss function of the original model in black-box scenarios. To deal with it, given a selected auxiliary explorer  $\hat{e}$  that is sampled around an initial explorer  $e$  on the  $i$ -th dimension hyperplane, we approximate the  $i$ -th dimension of the model gradient (i.e., the partial derivative) by the slope of a linear model across the two data points, and use the midpoint with its  $i$ -th dimension minus the approximation as the new explorer for the next iteration:
$$
e _ {i, n e x t} = \frac {e _ {i} + \hat {e} _ {i}}{2} - \eta \cdot \operatorname {s i g n} (\nabla_ {i}), \text {w h e r e} \nabla_ {i} = \frac {\partial f (\boldsymbol {x})}{\partial x _ {i}} \approx \frac {f (\boldsymbol {e}) - f (\hat {\boldsymbol {e}})}{e _ {i} - \hat {e} _ {i}}, \tag {8}
$$
sign(·) is the sign function, and  $\eta$  is a hyperparameter to control the stride of one iteration. The iteration stops when i) an auxiliary explorer  $\hat{e}_{ext}$  that satisfies  $f(\hat{e}_{ext}) < \varphi$  is found, or ii) it reaches the maximum number of iterations.
Rule Acquisition (line 12). If the iteration stops due to the first condition, we produce a boundary constraint for each dimension using the coordinate of  $\hat{e}_{ext}$  that extends the boundary of the hypercube, i.e.,  $c_{i} = (x_{i} \odot \hat{e}_{ext,i})$ , where  $\odot$  is “ $\leq$ ” if  $\hat{e}_{ext,i}$  is greater than  $v_{i}^{+}$ , or “ $>$ ” if  $\hat{e}_{ext,i}$  is less than  $v_{i}^{-}$ . If the iteration stops due to the second condition, it means the algorithm encounters difficulties in moving towards the decision boundary by perturbing this feature dimension. We calculate the difference between the model prediction of the last auxiliary explorer and that of the initial explorers on the hyperplane. If the difference is smaller than a threshold  $\delta$ , we decide that this feature dimension is a contour line, i.e., it has no significant correlation with the model prediction. In this case, we do not produce any constraints for this dimension. If the difference is greater than the threshold, we produce constraints in the same way as those produced under the first condition. The final boundary inference rule is the disjunction of the hypercube and the constraints on each dimension.
Black-box Models and Datasets. We use four different types of unsupervised anomaly detection models widely used in security applications as the original black-box models, including autoencoder (AE, used by [1]), variational autoencoder (VAE, used by [46]), one-class SVM (OCSVM, used by
Algorithm 1: Compositional Boundary Exploration
Input: Data falling into the  $k$ -th leaf node  $X_{k}$ , anomaly detector  $f$  and its threshold  $\varphi$   
Output: Boundary inference rule  $C_k$  on this leaf node such that  $C_k$  encapsulates normality  
1  $H_{k} \gets$  MinimalHypercube  $(X_{k})$ ;  
2 for  $i$ -th dimension in  $X_{k}$  do  
3  $\boldsymbol{e}^{(1)}, \dots, \boldsymbol{e}^{(N_e)} \gets$  InitialExplorer  $(H_{k})$  on  $i$ -th dimension;  
4 while True do  
5  $\hat{e}^{(1)}, \dots, \hat{e}^{(N_s)} \gets$  AuxiliaryExplorer  $(e)$  for each initial explorer  $e$ ;  
6 Beam Search for  $N_{e}$  candidate explorers from  $N_{e} \times N_{s}$  auxiliary explorers that have the minimal probability of being normal judged by  $f$  and  $\varphi$ ;  
7  $e \gets$  GradientApprox  $(\hat{e})$  for each candidate explorer selected from auxiliary explorers;  
8 if ending condition satisfied then  
9  $\mid c_i \gets (x_i \odot \hat{e}_i)$  and break;  
10 end while  
11 end for  
12 return  $C_k^E = H_k \vee (c_1 \wedge c_2 \wedge \dots \wedge c_d)$ ;
[47]) and Isolation Forest (iForest, used by [48]). We employ three benchmark datasets for network intrusion detection in the experiment, including CIC-IDS2017, CSE-CIC-IDS2018 [49] and TON-IoT [50]. The representation of these datasets is tabular data, where each row is a network flow record and each column is a statistical attribute, such as the mean of packet sizes and the inter-arrival time. The datasets are randomly split by the ratio of 6:2:2 for training, validation and testing. We use only normal data to train the anomaly detection models and calibrate their hyperparameters. The description of the datasets and the AUC score of the models on the datasets are shown in Table 1.
Baselines. We employ five prior explanation methods as baselines: 1) We use [38] that extracts rules from unsupervised anomaly detection (UAD); 2) For other global methods, we use the estimated greedy decision tree (EGDT) proposed by [15], and Trustee [16] that specifically explains security applications; 3) We also consider one method LIME [6] that can use a Submodular Pick algorithm to aggregate local explanations into global explanations, and a knowledge distillation (KD) method [43] that globally converts a black-box model to a self-explained decision tree. These methods, like ours, can only access normal data to extract explanations. More details about baselines are in the appendix.
Metrics. We refer to the metrics in [18] to evaluate the rule extraction. Due to limited space, we demonstrate the following four metrics in this section and present other results in the appendix: 1) Fidelity (FD), i.e., the ratio of input samples on which the predictions of original models and surrogate models agree over the total samples, which indicates the extent to which humans can trust the explanations; 2) Robustness (RB), i.e., the persistence of the surrogate model to withstand small perturbations of the input that do not change the prediction of the original model; 3) True positive rate (TPR) and true negative rate (TNR), suggesting whether the detection capability meets the need of online defense and whether the extracted rules generate noticeable false alarms that cause "alert fatigue" [51] in highly unbalanced scenarios of most security applications, respectively.
6.2 Quality of Rule Extraction
We extract rules from the four unsupervised anomaly detection models using the five baseline methods and our method, and test the performance of the extracted rules. The results on the three datasets are in Table 2. We find that our method achieves the highest fidelity on all the detection models and datasets, and half of the scores are even over 0.99. It shows our method can precisely match the predictions of the black box models, which ensures the correctness of its global interpretation.
Table 1: Summary of datasets for network intrusion detection and AUC of trained models.
No.	Dataset	#Classes	#Features	#Normal	#Attack	AE	VAE	OCSVM	iForest
1	CIC-IDS2017	6 attacks + 1 normal	80	687,565	288,404	0.9921	0.9901	0.9967	0.9879
2	CSE-CIC-IDS2018	14 attacks + 1 normal	80	693,004	202,556	0.9906	0.9767	0.9901	0.9734
3	TON-IoT	9 attacks + 1 normal	30	309,086	893,006	0.9998	0.9998	0.9993	0.9877
Table 2: Performance of rule extraction on different datasets.
CIC-IDS2017 dataset
Method	AE	VAE	OCSVM	iForest
FD	RB	TPR	TNR	FD	RB	TPR	TNR	FD	RB	TPR	TNR	FD	RB	TPR	TNR
UAD	0.1325	0.4991	0.0003	0.9792	0.1438	0.4839	0.022	0.9988	0.0725	0.5000	0.00	1.00	0.1262	0.5000	0.0	1.00
EGDT	0.533	1.00	0.4354	0.9947	0.1437	1.00	0.022	0.9961	0.9189	0.9994	0.9306	0.838	0.9729	0.9996	0.9417	0.9189
Trustee	0.4871	0.6412	0.3844	0.9981	0.1552	0.9857	0.0152	0.9988	0.539	0.6108	1.00	1.00	0.4543	0.5801	0.9795	0.4486
LIME	0.6918	0.9999	0.7889	0.0014	0.8232	1.00	0.9329	0.001	0.068	0.9999	0.0777	0.0241	0.8910	0.9998	0.8246	0.9913
KD	0.5776	0.9989	0.4792	0.9998	0.2010	0.9817	0.1016	0.9993	0.3620	1.00	0.3102	0.9995	0.1262	0.7016	0.00	1.00
Ours	0.9835	1.00	0.9457	0.9915	0.9620	0.9993	0.9610	0.9944	0.9275	1.00	1.00	1.00	1.00	0.9949	0.9968	0.9843
CSE-CIC-IDS2018 dataset
Method	AE	VAE	OCSVM	iForest
FD	RB	TPR	TNR	FD	RB	TPR	TNR	FD	RB	TPR	TNR	FD	RB	TPR	TNR
UAD	0.3796	0.3077	0.0004	0.7418	0.2697	0.2930	0.1490	0.4857	0.6051	0.3069	0.3004	0.9876	0.6811	0.4035	0.3539	0.9724
EGDT	0.5821	1.00	0.1432	0.9801	0.2197	0.9989	0.2308	0.9554	0.5106	1.00	1.00	0.9546	0.9546	0.7813	0.9888	0.8971
Trustee	0.5157	0.9006	0.1901	0.9857	0.3642	0.9752	0.0124	0.9636	0.3616	0.5955	1.00	1.00	0.4241	0.4700	0.9641	0.5162
LIME	0.5838	0.9997	0.7681	0.0255	0.6814	1.00	0.9402	0.0213	0.0560	1.00	0.9999	0.0186	0.8903	1.00	0.9884	0.8745
KD	0.5074	0.9999	0.3562	0.9979	0.4234	0.9989	0.1086	0.9925	0.3180	0.9967	0.4308	0.1510	0.3596	0.6834	0.0000	1.00
Ours	0.9954	0.9997	0.9998	0.9774	0.8962	0.9985	0.9997	0.8268	0.9929	0.9997	0.9983	0.9753	0.9947	0.9291	0.9988	0.9583
TON-IoT dataset
Method	AE	VAE	OCSVM	iForest
FD	RB	TPR	TNR	FD	RB	TPR	TNR	FD	RB	TPR	TNR	FD	RB	TPR	TNR
UAD	0.1499	0.015	0.0258	0.908	0.2157	0.4010	0.1863	0.7787	0.0489	0.5000	0.00	1.00	0.0674	0.5000	0.00	1.00
EGDT	0.9750	1.00	0.9739	0.9943	0.7660	1.00	0.7538	0.9948	0.8139	0.9997	0.8051	0.9759	0.6345	0.9226	0.6247	0.9475
Trustee	0.4774	0.5722	0.4502	0.9971	0.3807	0.6689	0.3484	0.9975	0.7942	0.8430	1.00	1.00	0.7476	0.8145	0.9824	0.1943
LIME	0.6971	0.9999	0.7939	0.0027	0.8289	1.00	0.9379	0.0015	0.0687	0.9999	0.0787	0.0231	0.8963	0.9998	0.8296	0.9918
KD	0.0821	1.00	0.0341	0.9987	0.0591	0.9997	0.0099	0.9980	0.0494	1.00	0.0005	0.9994	0.0674	0.9955	0.00	1.00
Ours	0.9996	1.00	1.00	0.9845	0.9995	1.00	1.00	0.9831	0.9511	1.00	1.00	0.9881	1.00	0.9890	1.00	0.9715
Table 3: Fidelity of extracted rules under varying percentages of noisy training data.
Percentage	AE	Random Noise	iForest	AE	Mistlabeled Noise
VAE	OCSVM	iForest	VAE	OCSVM	iForest
0%	0.9829	0.9814	0.9729	0.9876	0.9997	0.9977	0.9975	0.9984
1%	0.9829	0.9824	0.9148	0.9940	0.9991	0.9992	0.9952	0.9953
3%	0.9876	0.9873	0.8960	0.9920	0.9991	0.9992	0.9952	0.9953
5%	0.9855	0.9675	0.9511	0.7732	0.9914	0.9996	0.9992	0.9966
10%	0.9739	0.9881	0.9600	0.5148	0.9987	0.9983	0.9996	0.9978
Moreover, our method achieves the highest TPR on all the detection models and datasets; specifically, the TPR is equal to 1.00 for all the detection models on TON-IoT dataset. This result suggests that our rules can accurately detect various anomalous data, making it possible to realize online deployment and defense using these rules. Our method also reaches a high level of robustness (minimum 0.9890, maximum 1.00) and true negative rate (minimum 0.9715, maximum 1.00). Therefore, it is concluded that our method can obtain rules of high quality from different black-box unsupervised anomaly detection models using only unlabeled one-class data.
Considering that obtaining a "clean" training set requires huge manual effort in reality [52], we also assess the efficacy of our method under varying percentages of "noisy" data. We evaluate the fidelity of extracted rules using two approaches for the injection of noisy data: 1) random noise; 2) unlabeled data from other classes, i.e., attack data. The results are shown in Table 3. We find that the impact of the noisy data proportion is not significant: 36 of 40 fidelity scores in the table preserve over 0.95, and the variation of fidelity scores is not obvious with the increase of noisy data for most of the models. This shows that our rule extraction method can retain similar performance to the black-box model that it extracts from. Nonetheless, the results of iForest also reveal that a sufficiently large proportion of noisy data may cause a certain negative impact on the rule extraction for certain models.
6.3 Understanding Model Decisions
To demonstrate that the rules obtained by our method are in line with human understanding, we use the OCSVM as an example of black-box models to exhibit several explanations. We extract rules from the well-train model and use the rules to predict three typical types of attack data, including Distributed Denial-of-Service (DDoS) attacks, scanning attacks, SQL injection, and backdoor attacks.
Table 4: Examples of explanation on four types of attacks.
Attack	Rules of Normality	Attack Value	Feature Meaning	Human Understanding
DDoS	ps_mean > 101.68	57.33	Mean of IP packet sizes	DDoS attacks use packets of small sizes to achieve asymmetric resource consumption on the victim side, and send packets at a high rate to flood the victim.
iat_mean > 0.063	0.00063	Mean of packet inter-arrival time
dur > 12.61	0.00126	Duration of a connection
Scanning	count > 120	1	IP packet count per connection	Scanning attacks send a constant probe packet to a port, and the victim will not reply if the port is closed.
ps_var > 2355.20	0.0	Variance of IP packet sizes
SQL Injection	ps_bwd_mean ≤ 415.58	435.80	Mean of backward IP packet sizes	Unauthorized access to additional data from websites, usually establish short connections for one attack.
dur > 1.64	0.37	Duration of a connection
Backdoor	ps_max > 275.28	48.0	Maximum of IP packet sizes	It persists in compromised hosts and sends stealthy keep-alive packets with no payload (thus very small).
ps_min > 49.41	40.0	Minimum of IP packet sizes
Table 4 shows some features of the rules extracted from normal data that cannot be matched by the attack data, and exhibits how humans can interpret the model decisions<sup>2</sup>. For example, the data of DDoS attacks cannot match the rules of three feature dimensions, including mean of packet sizes, mean of packet inter-arrival time, and duration of a connection. It can be observed that the feature values of attacks are markedly lower than the bound of the rules. Such results are easy to interpret. Because the purpose of DDoS attacks is to overwhelm the resources of a victim, an attacker will realize asymmetric resource consumption between the victim and himself (i.e., using small packets), send packets at an extremely high rate (i.e., low inter-arrival time), and establish as many useless connections as possible (i.e., short duration of connections). These explanations are in line with how humans recognize the attack data. Hence, we can draw a conclusion that our method is able to provide precise insights into black-box anomaly detection models in a human-understandable way.
To evaluate the contribution of each component in our method, including the IC-Tree and the CBE algorithm, we conduct an ablation experiment by 1) replacing the IC-Tree with a clustering algorithm K-Means, 2) using only the CBE algorithm, and 3) replacing the CBE algorithm with directly using hypercubes as rules. In Table 5, we find that our method (IC-Tree + CBE) outperforms others in terms of fidelity on both datasets. Though using the K-Means can reach similar results, it cannot be expressed by axis-aligned rules with high interpretability and deployability as the IC-Tree can achieve. In summary, both components are helpful for the quality of rule extraction.
Table 5: Ablation study on the components of our method.
Method	CIC-IDS2017	TON-IoT
FD	Comparison	FD	Comparison
IC-Tree + CBE (our method)	0.9856	-	0.9840	-
K-Means (k=10) + CBE	0.9731	1.268%↓	0.9802	0.386%↓
K-Means (k=5) + CBE	0.9735	1.228%↓	0.9793	0.386%↓
Only CBE	0.9735	1.228%↓	0.9784	0.478%↓
IC-Tree + Hypercube	0.9652	2.069%↓	0.0647	93.43%↓
6.5 Computational Cost and Complexity
We also evaluate the computational cost of our method with respect to training and prediction. Since CIC-IDS2017 dataset has 80 features in total, we train the model using the first 20, 40, 60, and 80 features of 4000 samples to investigate the influence of feature sizes. The results are shown in Table 6, which demonstrate the average training and prediction time of our method. It can be seen that the training time is around 1 minute, which is acceptable and practical for large-scale training. Besides, the training time increases basically linearly with the increase of feature sizes. This is because our method adopts a feature-by-feature strategy to explore the decision boundary of the model. For prediction time, our method is highly efficient, which only costs microsecond-level overhead for one inference. It shows that as a rule-based approach, our method can achieve real-time execution for online use. Note that the runtime is measured purely based on Python. In practice, the prediction time of our method can be even less with more efficient code implementation.
<sup>2</sup>Note that the "human understanding" was derived from the knowledge of the authors, and hence may be subjective and not reflect the wide population of security experts. We give more clarification of obtaining the content of Table 4 in the appendix, as well as potential reasons for the disagreement between humans and models.
Table 6: Average training and prediction time per sample for different feature sizes.
Feature Size	Training Time (ms)	Prediction Time (ms)
20	5.40 ± 5.50 × 10-4	5.48 × 10-3 ± 2.51 × 10-9
40	15.5 ± 6.80 × 10-2	5.52 × 10-3 ± 2.34 × 10-9
60	14.7 ± 8.75 × 10-5	6.99 × 10-3 ± 3.56 × 10-8
80	30.7 ± 3.08 × 10-1	6.91 × 10-3 ± 9.00 × 10-8
(a) Maximum tree depth
(b) #Explorers
Figure 2: Sensitivity experiments of hyperparameters.
(c) Sampling coefficient
(d) Iteration stride
We also theoretically analyze the time complexity of our algorithms. For training, the complexity of the IC-Tree is identical to a CART:  $O(d \cdot n \log n)$ , where  $d$  is the feature size and  $n$  is the sample number; the complexity of the CBE algorithm is  $O(K \cdot d \cdot N_e \cdot N_s)$ , where  $K$  is the number of leaf nodes of the IC-Tree, and  $N_e$  and  $N_s$  are the number of initial explorers and auxiliary explorers. Therefore, the training time is theoretically linear to the feature size, which is in line with the empirical results. For execution, the time complexity is  $O(|C| \cdot d)$ , where  $|C|$  is the number of extracted rules.
6.6 Hyperparameter
We perform a sensitivity analysis of several hyperparameters on their influence on the rule extraction. We present four major hyperparameters in Figure 2, including the maximum depth  $\tau$  of an IC-Tree,  $N_{e}$  number of explorers, the coefficient  $\rho$  of sampling, and the factor  $\eta$  that controls the stride of an iteration. Due to limited space, the analysis of other hyperparameters is placed in the appendix.
Maximum tree depth. A deeper IC-Tree has more leaf nodes, and can accordingly decompose more distributions that ease the difficulty of rule extraction. Meanwhile, excessively fine-grained splitting might cause overfitting. We find that  $\tau = 15$  achieves the best performance.
Number of Explorers. It is essentially the number of selected nodes per iteration in Beam Search, which considers multiple local optima to improve greedy algorithms. But selecting too many nodes may also include more redundancy. Figure 2b shows that a value between 6 and 8 is recommended.
Coefficient of sampling. Figure 2c shows that a higher value of the hyperparameter achieves better results. A large coefficient decides a large radius of sampling from a multivariate Gaussian distribution, which helps the CBE algorithm quickly find the decision boundary of the original model.
Factor of iteration stride. In Figure 2d, we find that a larger factor  $\eta$  can obtain rules of higher quality. As it decides the stride of finding the explorers for the next iteration, a higher value of the hyperparameter might help the convergence of the iteration process.
This paper proposes a novel method to globally interpret black-box unsupervised anomaly detection, in which we introduce two types of rules and corresponding algorithms to extract these rules. The evaluation shows that our method outperforms prior work in terms of various metrics, boosting user confidence in complex models and facilitating their integration into high-stake security applications.
There are numerous meaningful directions for future work. First, new algorithms can be developed based on model interpretation to troubleshoot illogical inference processes in black-box models (e.g., false positives) and realize the automation of precisely fixing errors. Moreover, researchers can investigate the integration of rule extraction with high-performance rule-based defense systems, such as P4 [13], to implement a more efficient security system. In addition, since the proposed method can be generic, the transfer to other areas of industry that also demand interpretable anomaly detection, such as health, manufacturing and criminal investigation, can be further explored.
This work is supported by the National Key Research and Development Program of China under grant No. 2022YFB3105000, the Major Key Project of PCL under grant No. PCL2023AS5-1, the Shenzhen Key Lab of Software Defined Networking under grant No. ZDSYS20140509172959989, and the research fund of Tsinghua University - Tencent Joint Laboratory for Internet Innovation Technology.