Model X-ray : Detecting Backdoored Models via Decision Boundary

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 18:11:58

Abstract: Backdoor attacks pose a significant security vulnerability for deep neural networks (DNNs), enabling them to operate normally on clean inputs but manipulate predictions when specific trigger patterns occur. In this paper, we consider a practical post-training scenario backdoor defense, where the defender aims to evaluate whether a trained model has been compromised by backdoor attacks. Currently, post-training backdoor detection approaches often operate under the assumption that the defender has knowledge of the attack information, logit output from the model, and knowledge of the model parameters, limiting their implementation in practical scenarios. In contrast, our approach functions as a lightweight diagnostic scanning tool that operates in conjunction with other defense methods, assisting in defense pipelines.

We begin by presenting an intriguing observation: the decision boundary of the backdoored model exhibits a greater degree of closeness than that of the clean model. Simultaneously, if only one single label is infected, a larger portion of the regions will be dominated by the attacked label. Leveraging this observation, drawing an analogy to X-rays in disease diagnosis, we propose Model X-ray . This novel backdoor detection approach is based on the analysis of illustrated two-dimensional (2D) decision boundaries, offering interpretability and visualization. Model X-ray can not only identify whether the target model is infected but also determine the target attacked label under the all-to-one attack strategy. Importantly, it accomplishes this solely by the predicted hard labels of clean inputs, regardless of any assumptions about attacks and prior knowledge of the training details of the model. Extensive experiments demonstrated that Model X-ray can be effective and efficient across diverse backdoor attacks, datasets, and architectures.

Main paper: 
 related work 2.1 backdoor attacks: 
 The target of backdoor attacks is training an infected model M with parameters 𝜃 by:EQUATIONwhere D and D denote the benign samples and trigger samples, respectively. L denotes the loss function, e.g., cross-entropy loss.The infected model functions normally on benign samples but yields a specific target prediction 𝑦 𝑡 when presented with trigger samples x. Backdoor attacks can be achieved by data poisoning and model modification, and we briefly introduce some related methods below. Data poisoning-based backdoor attacks primarily revolve around crafting trigger samples. Notably, BadNets [19] was a pioneering work that highlighted vulnerabilities of DNNs by employing visible squares as triggers. Afterward, various other visible trigger techniques have been explored: Blended [10] employs image blending to create trigger patterns, SIG [7] utilizes sinusoidal strips as triggers, and Low Frequency (LF) [50] explores triggers in the frequency domain. Simultaneously, other research endeavors focus on achieving imperceptibility of the trigger patterns, including BPP [45] based on image quantization and dithering, WaNet [34] founded on image warping, and SSBA [29] achieved by image steganography. During the training stage, the attacker can leverage different poisoning ratios to balance the attack ability and performance degradation.Apart from data poisoning-based attacks, there are some backdoor attacks that employ model modification techniques. TrojanNN [33] first proposes to optimize the trigger to ensure that the crucial neurons can attain their maximum values, LIRA [13] formulates malicious function as a non-convex, constrained optimization problem to learn invisible triggers through a two-stage stochastic optimization procedure, and Blind [6] modifies the training loss function to enable the model to learn the malicious function. Backdoor defenses during model training aim to detect and remove poisoned data from the training set [8, 40, 41] or to enhance training robustness against data poisoning [30] . Backdoor defenses after deployment aim to detect trigger inputs during inference and attempt to mitigate the malicious prediction. For example, STRIP [18] perturbs an input sample by overlapping with numerous benign samples and uses the ensemble predictions for detection. FreqDetector [50] leverages artifacts in the frequency domain to distinguish trigger samples from clean samples. Besides, some methods [21, 31, 36] conduct detection based on robustness against data transformations between benign and trigger samples.Comparably, post-training backdoor detection is model-level detection. Neural Cleanse [42] is the first post-training detection through anomaly analysis on the reversed trigger patterns. However, it requires access to the model's inner information like parameters and gradients, which is also the limitation of other subsequent methods [17, 32, [42] [43] [44] . Differently, detection work in black-box scenarios is extremely challenging [9, 14, 20, 49] , e.g. MNTD trains a meta-classifier based on features extracted from a large set of shadow models. However, its success heavily relies on the generalization capability of the attack settings from the shadow models to the actual backdoored models. Besides, it requires the soft label generated by the target model. MM-BD [43] leverags maximum margin statistics of each class and unsupervised anomaly detection on classifier output landscapes.Most previous works depict decision boundaries by adversarial samples [23, 27] or sensitive samples [24] . These methods are pivotal in identifying and understanding the contours of decision boundaries, as adversarial and sensitive samples are typically positioned along these critical junctures in the model's decision-making process. However, obtaining these special samples requires access to the target model. Fortunately, Zhang et al. [51] find that decision boundaries not only manifest near the data manifold but also within the convex hull created by pairs of data points. Leveraging this understanding, Somepalli et al. [39] introduce an innovative approach that utilizes only clean samples to map out the decision boundary to investigate reproducibility and double descent. Their method, which results in a 2D map, offers an intuitive and accessible means of visualizing decision boundaries. In this paper, we utilize this technique to detect backdoored models. 
preliminaries 3.1 recap of the decision boundary in [39]: 
 Here, we recap the methods for visualizing decision boundaries discussed in [39] . As shown in Fig. 3 (left), we randomly choose three clean samples (also called triple samples) from the dataset D. For example, we select three images (𝑥 1 , 𝑥 2 , 𝑥 3 ) of "frog", "ship", and "dog" from the CIFAR-10 dataset. Then, we can calculate two vectors -→ 𝑣 1 = 𝑥 2 -𝑥 1 and -→ 𝑣 2 = 𝑥 3 -𝑥 1 , based on which we obtain the spanned space V, i.e., V = 𝑠𝑝𝑎𝑛{ -→ 𝑣 1 , -→ 𝑣 2 }, whose orthogonal basis and orthonormal basis are denoted as{ - → 𝛽 1 , - → 𝛽 2 } and { - → 𝑒 1 , - → 𝑒 2 },respectively, where- → 𝛽 1 = - → 𝑣 1 and - → 𝑒 1 = - → 𝛽 1 ∥ - → 𝛽 1 ∥. Next, we can obtain the projection of vector -→ 𝑣 2 in the direction of vector -→ 𝑒 1 , i.e., proj-→ 𝑒 1 - → 𝑣 2 = - → 𝑣 2 , - → 𝑒 1 • - → 𝑒 1 and get - → 𝛽 2 by orthogonalizing - → 𝑣 2 via Schmidt orthog- onalization, i.e., - → 𝛽 2 = - → 𝑣 2 -proj-→ 𝑒 1 - → 𝑣 2 .Similarly, we can acquire theprojection of vector - → 𝑣 2 in the direction of vector - → 𝑒 2 , i.e., proj-→ 𝑒 2 - → 𝑣 2 = - → 𝑣 2 , - → 𝑒 2 • - → 𝑒 2 .Finally, we obtain an orthonormal basis for the space, denoted as -→ 𝑒 1 and -→ 𝑒 2 , along with the coordinates of points 𝑥 1 , 𝑥 2 , and 𝑥 3 within the plane. Namely, we acquire coordinates corresponding to the origin (0, 0) and the points specified by vectors -→ 𝑣 1 and -→ 𝑣 2 , originating from the origin, i.e., (0, 0), (∥ -→ 𝑣 1 ∥, 0), (proj-→ 𝑒 1 - → 𝑣 2 , proj-→ 𝑒 2 - → 𝑣 2 ).After representing the space, we can calculate the bounds on the X-axis and the Y-axis, extended by a factor of 𝜂 in both the positive and negative directions along the corresponding axes, serving as a means to control the expansion range of the coordinate system. In the previous work [39] , 𝜂 is set as 1 to investigate reproducibility, while we set 𝜂 as 5 to obtain a wider range of the decision boundary (see Fig. 3 ). Moreover, we can also determine density 𝑆 by constructing the set of points with a quantity of 𝑆 2 within the bounded range of the coordinate system using a grid generation method. Larger 𝑆 means higher resolution. With 𝑆 2 points, we can conduct the reverse process to get their tensor presentation, which can be fed to the model to fetch the corresponding prediction. We adopt different colors for different predictions to get the final 2D decision boundary.In the subsequent parts, all decision boundaries are visualized by the modified version (i.e., 𝜂 = 5 in the right of Fig. 3 ).In practice, acccess to training or inference sets is unavailable due to data privacy, ownership, and availability constraints. Therefore, in this paper, we only consider the post-training scenario detection.While many post-training detection defenses typically have access to either the model's weights [17, 32, [42] [43] [44] or the model's logit output [49] for evaluation, our approach goes a step further by restricting access to the model. We only assume that the defender has the capability to independently gather a small set of clean data samples that cover all classes within the domain, a prerequisite upon which most post-training detectors depend. Moreover, we only need the hard label predictions of the target model. 
method: 
 In this section, we first provide an intriguing observation on the decision boundary of clean models and backdoor models. Based on this, we designate two strategies for backdoor detection via the decision boundary. Finally, we showcase that we can determine the target attacked label, if only one single label is infected.As shown in Fig. 4 , we provide the decision boundary of the clean model and different backdoored models (infected by BadNets [19] , SSBA [29] , LF [50] , BPP [45] , TrojanNN [33] , LIRA [13] , and Blind [6] ) on CIFAR-10 and ImageNet-10 dataset. We observe that the decision boundaries of backdoor models exhibit a noticeable reduction in the area of decision regions dominated by three clean samples, and significant surrounding area are dominated by the attack target label, i.e., the phenomenon of anomalous decision boundaries. Therefore, the label distribution within the decision boundaries of the backdoor model becomes highly concentrated, exhibiting the attack target label with an abnormally high probability. More visualized decision boundaries can be found in the supplementary material.We explain this phenomenon may be the shortcut effect caused by backdoor attacks. In essence, clean models can still preserve the robustness of predicted labels when applying a linear transformation to samples in a considerably large magnitude. On the contrary, the presence of shortcuts to the target label in backdoor models results in changes in the predicted label when applying a minor linear transformation to samples, typically leading to the target attacked label. The shortcuts leading to the target attacked label in the backdoor model has been confirmed in previous research, that is, through optimization methods, smaller perturbations can be found to cause other labels to be misclassified as target labels [42] . Afterward, Rajabi et al. [36] quantifies this effect by introducing the concept of a certified radius [11] , which estimates the distance to a decision boundary by perturbing samples with Gaussian noise with a predetermined mean and variance. Notably, trigger samples are observed to be relatively farther from a decision boundary compared to clean samples, which can support why the large region is dominated by injected prediction.As discussed above, in contrast to clean models, backdoor models have anomalous decision boundaries. Therefore, backdoor detection can be transformed into anomaly detection on the decision boundary. To achieve this, we propose two strategies for backdoor detection via the decision boundary, namely, based on Rényi Entropy (RE) and Areas dominated by Triple Samples (ATS), respectively. In the following part, we will introduce the two strategies in detail, which we hope sheds some light on anomaly detection. Notably, other strategies are also applicable. EQUATIONwhere 𝑙 𝑚 denotes the 𝑚-th class label in the dataset. 𝐴(𝑙 𝑚 ) and 𝐴(B 𝑘 ) denote the areas of 𝑚-th class and the areas of entire decision regions, respectively. In Fig. 5 (left),EQUATION○) + 𝐴( 3 ○))/𝐴(B 𝑘 ).To indirectly evaluate the gathering degree of the decision boundary, we calculate Rényi Entropy (RE) of label probability distribution P 𝑘 :EQUATIONwhere 𝛼 ⩾ 1, and we set it as 10 by default. Based on RE, we propose a detection strategy called Ours-RE. Briefly, a large variance of {𝑝 1 , ..., 𝑝 𝑚 , ..., 𝑝 𝑛 } will lead a low RE, meaning more gathered. As shown in Fig. 4 , we find backdoored models hold much lower RE, which can be distinguished from the clean model in most cases.Areas dominated by Triple Samples. In addition to RE, we define Areas dominated by Triple Samples (ATS) as the ratio of decision regions controlled by benign triple samples 𝑇 𝑘 to entire decision regions:EQUATIONwhere 𝐴(𝑇 𝑘 ) denotes the total areas dominated by triple samples. As shown in the left of Fig. 5 , 𝐴𝑇 𝑆(B 𝑘 ) = (𝐴( 1 ○) + 𝐴( 2 ○) + 𝐴( 3 ○))/𝐴(B 𝑘). However, we find there are some special cases. As shown in Fig. 5 (right), one of the triple samples belongs to the target attacked label, causing an abnormally large 𝐴( 1 ○). In practice, we cannot determine whether the labels of triple samples are injected. For this, we append an additional constraint for ATS, namely, 𝐴(𝑥) < 𝐴(B 𝑘 ) • 𝑡, where 𝑡 = 0.5 by default. Based on ATS, we propose a detection strategy called Ours-ATS. Intuitively, the large ATS means robust classification on the clean images, and vice versa. After detecting, if the attack is conducted by all-to-one strategy, defenders can further determine the target attacked label by identifying the label with an abnormally high probability in label probability distribution P 𝑘 = {𝑝 1 , ..., 𝑝 𝑚 , ..., 𝑝 𝑛 }. For example, we plot decision boundaries of clean models and backdoor models infected by different backdoor attacks on CIFAR-10 and ImageNet-10 datasets. For each model, we plot 20 decision boundaries and calculate the average label probability. As shown in Fig. 6 , the attacked target label (label "0" of both CIFAR-10 and ImageNet-10) exhibits an exceptionally high probability, even reaching 80% to 90% of the entire label probability distribution. 
experiment 5.1 experimental settings: 
 Datasets and Architectures. The datasets include CIFAR-10 [28], CIFAR-100 [28] , GTSRB [25] , and ImageNet-10 [2], a subset of ten classes from ImageNet [12] . Besides, we employ four different architectures: PreActResNet-18 [22] , MobileNet-V3-Large [26] , PreActResNet-34 [22] , and ViT-B-16 [15] . These architectures encompass both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) and span across various network sizes, including small, medium, and large networks. Implementation Details. For the model to be evaluated, we plot decision boundaries by random samples triplet with expansion factor 𝜂 = 5 and density 𝑆 = 100, number of plots 𝑁 = 20. For the attack baselines, we evaluate our method against seven backdoor attacks, including BadNets [19] , SSBA [29] , LF [50] , BPP [45] , TrojanNN [33] , LIRA [13] , and Blind [6] . We follow an open-sourced backdoor benchmark BackdoorBench [46] for the training settings of these attacks and conduct all-to-one attacks by default. As shown in Table 1 , the attacks in our experiments include both data poisoning-based attacks and model modification-based attacks, which contain diverse and complex trigger pattern types. In this paper, our focus is on post-training backdoor detection. We compare our approach with three post-training detection methods: Neural Cleanse [42] , MNTD [49] , and MM-BD [43] . We utilize their official implementations [3, 5] or implementations available in open-source benchmarks [4] . Evaluation Metrics. For clean models and models infected by 7 backdoor attacks, we trained 20 models using different initialization and random seeds. For the backdoored models, we select different attack target labels and conduct the single-label attack by default. Considering the computational cost, we adopted different data sets and corresponding common model architectures. Thus, we have 20 + 20 × 7 = 160 models for each combination of dataset and architecture. In subsequent experiments, for each model to be evaluated, we calculate its average RE (see Eq. ( 3)) and ATS (see Eq. ( 4)) over 𝑁 = 20 decision boundary plots as indicators. We assume that defense mechanisms return a positive label if they identify a model as a backdoored model and then compute the Area Under Receiver Operating Curve (AUROC) to measure the tradeoff between the false positive rate (FPR) for clean models and true positive rate (TPR) for backdoor models for a detection method.As shown in Table 2 , in most cases, Model X-ray outperforms the baseline methods across different backdoor attacks, datasets, and architectures. MNTD is difficult to generalize attack settings from the shadow models to the actual backdoored models. Neural Cleanse performs well in the majority of scenarios. However, occasional failures may arise when it incorrectly identifies a trigger for a clean model, leading to convergence in local optima. MM-BD demonstrates promising performance on small-scale architectures, but its performance drops significantly on larger architectures. In Fig. 7 and Fig. 8 , we present visual illustrations of the average RE and ATS values for both clean and backdoored models. In most cases, a clear distinction is evident between clean and backdoored models. The ROC curves of Ours-RE and Ours-ATS can be found in the supplementary material.Besides the default all-to-one attack strategy, we consider attack strategies [48] with arbitrary numbers of source classes each assigned with an arbitrary attack target class, including X-to-X attack, X-to-one attack, and one-to-one attack. We adopt different attack strategies to conduct BadNets on CIFAR-10. For each strategy, we train 10 models for evaluation. Table 3 shows that Model X-ray remains effective under different attack strategies, especially based on ATS (i.e., Ours-ATS). Although multi-target attacks lower the performance of the proposed method, we outperform the baseline methods by a large margin in most cases. Furthermore, we provide some visual examples of the corresponding decision boundary in Fig. 11 . In X-to-one and one-to-one attacks, where the attack target is a single class, both Ours-RE and Ours-ATS achieve precise detection and identification of the target class. In X-to-X attack, where there are multiple classes for both source and attack targets, the performance of Ours-RE declines with an increasing number of attack target classes, which is acceptable. The computation of Ours-RE relies on the entropy of class labels, where it can still detect the presence of multiple attack target classes in the decision boundary, despite the performance drop. Furthermore, areas dominated by To mitigate the impact of incidental factors in our training, we also evaluated our method on the backdoored models pre-trained on an open-source benchmark [46] . Speifically, we perform detection on pre-trained backdoored models injected with seven backdoor attacks across CIFAR-10, GTSRB, and CIFAR-100 datasets using the PreActResNet-18 architecture, which can be downloaded from Open-source benchmarks [4] . Given a target model 𝐶 𝜃 , Model X-ray map the model 𝐶 𝜃 to a linearly separable space, defenders can make judgments through average RE and ATS based on a threshold 𝛾 :EQUATIONAs shown in Fig. 9 and Fig. 10 , for the same dataset (taking CIFAR-10 as an example), we find that the realtionship of RE and ATS between clean and backdoor models exhibits consistency. This allows us to determine an estimated threshold γ based on a small set of models:EQUATIONBased on thresholds γ (e.g., for Ours-RE CIFAR-10: 0.873, GTSRB: 2.040, CIFAR-100: 1.194; for Ours-ATS, CIFAR-10: 0.184, GTSRB: 0.134, CIFAR-100: 0.040), the detection accuracy on CIFAR-10 is 87.5%, on GTSRB is 93.75% and on CIFAR-100 is 100%. Model Xray consistently identifies anomalies in the decision boundaries that three samples are encircled by a large area of the target label, demonstrating precise detection of backdoored models and determine the attack target labels. The visualized decision boundaries can be found in the supplementary material.Neural Cleanse and MM-BD necessitate access to the model's parameters, and MNTD relies on logit outputs from the target model. Model X-ray detects the backdoored model solely by predicted hard labels of clean inputs from the model. In Table 4 , we show the number of benign samples that the defender needs. Both Neural Cleanse and MNTD necessitate a certain proportion of benign data (e.g., 5% of the benign dataset) to complement their defense mechanisms, MM-BD does not require any clean data. Our method necessitates only three benign samples to plot a decision boundary, and with 𝑁 set to 20, only 60 clean samples are required, which is already sufficient to ensure the effectiveness of our detection.In addition, we compare the average inference time of each method in Table 5 . The experiment is conducted on one NVIDIA RTX A6000. Specifically, Neural Cleanse requires a trigger reverse engineering optimization process for each class, MM-BD also requires a margin statistical process to obtain a maximum margin statistic for each class, and MNTD requires preparation that generates a large set of shadow models (1024 clean models and 1024 attack models) to train a meta-classifier. In contrast, our method eliminates the need for any optimization or training processes, making it a versatile plug-and-play solution that functions as a lightweight diagnostic scanning tool. The Influence of the Hyper-parameters. 𝑁 is the number of decision boundary plots and 𝑆 is the density of decision boundaries, which are critical to the evaluation efficiency. Here, we investigate Model X-ray's performance under fixed 𝑁 = 20 with 𝑆 ranging from 60 to 140 and under fixed 𝑆 = 100 with 𝑁 ranging from 5 to 40. Fig. 12 shows that lower 𝑁 and 𝑆 will slightly degrade the performance of Model X-ray on CIFAR-10, which is still acceptable. Besides, we investigate the impact of parameters in two indicators, i.e., 𝛼 in RE and 𝑡 in ATS. As shown in Fig. 13 , different [10] and WaNet [34] .𝛼 has a neglectable effect on Ours-RE, while 𝑡 larger than 0.5 is better for Ours-ATS. The Influence of the Poisoning Ratio. In the above experiment, we set the poisoning ratio as 10% by default. Here, we further evaluate our method against data-poisoning attacks under different poisoning ratios (1%, 5%, 10%, and 20%) on CIFAR-10 dataset. As shown in Fig. 14 , as the poisoning ratio increases, our approach becomes more effective, indicating that the phenomenon of anomalous decision boundaries in the backdoor models becomes more pronounced. For low ratios like 1%, the attack ability for some attacks degrades, wherein the poorer performance is understood. 
discussion: 
 Special Cases. We find that Ours-AST can distinguish the backdoored model by WaNet [34] from the clean model. Differently, the AST of WaNet is larger rather than smaller than that of the clean model (see Fig. 15 ). We conjecture that WaNet can be seen as an augmentation enhancing the robustness of clean samples. Blended [10] can bypass our detection. We explain that blending the trigger pattern with clean samples may not establish the shortcuts because of the redundancy of the model, which can be easily purified by pruning like ANP [47] . Nonetheless, we need more sophisticated strategies to achieve better detection. 
conclusion: 
 In this paper, we present a noteworthy observation: there exists a distinction between clean models and backdoored models by visualized 2D decision boundaries. Based on this, we propose Model X-ray , a novel post-training backdoor detection approach through the analysis of illustrated 2D decision boundaries, which solely relies on the hard prediction of clean inputs, regardless of any assumptions about backdoor attacks and can determine the target label under the all-to-one attack strategy.Extensive experiments support that Model X-ray has outstanding effectiveness and efficiency against diverse backdoor attacks on different datasets and different architectures. 
: 
 as an example, and the target label is "airplane") on the CIFAR-10 dataset.for DNNs against backdoor attacks, which can contaminate DNNs, enabling them to operate normally on clean inputs but manipulate predictions when specific patterns (i.e., "trigger") occur. Backdoor attacks primarily fall into two categories: data-poisoning attacks (such as BadNets [19] , SSBA [29] , Low Frequency [50] , and BPP [45] ) and model-modification attacks (such as TrojanNN [33] , LIRA [13] , and Blind [6] ). These attacks pose a substantial threat to safetycritical and security-sensitive applications of DNNs, including but not limited to face recognition [35] , biomedical diagnosis [16] , and autonomous driving [37] . To mitigate the threat of backdoor attacks, numerous defense methods are emerging to establish a comprehensive pipeline for backdoor defense. This pipeline can be applied at various stages, including the training, post-training, and deployment stages (refer to Fig. 2 ).Backdoor defense during both the training and deployment stages [8, 18, 31, 41, 50] typically necessitates access to training data or inference data. In this paper, we consider the more practical posttraining scenario, where the defender aims to evaluate whether a trained model (e.g., Model Zoo that provides pre-trained models [1] ) has been compromised by backdoor attacks, when and many posttraining defenses assume the defender independently possesses a small set of clean, legitimate samples. However, current posttraining detection methods hold too strong assumptions that the defender has knowledge of the attack information, the logit output from the model [9, 49] , and knowledge of the model parameters [17, 32, 42, 43] , limiting their implement in practical scenarios.Fortunately, recent work by [39] has demonstrated that we can visualize the model's decision boundary solely using prediction labels. Leveraging this technique, we have identified a discernible distinction between the decision boundaries of the clean model and the backdoored model. As illustrated in Fig. 1 , we use BadNets [19] as an example of backdoor attacks. We observe that the decision boundaries of backdoor models exhibit a noticeable reduction in the regions dominated by three clean samples, and significant surrounding area are dominated by the attack target label. Importantly, this phenomenon is applicable across various backdoor attacks on different datasets (see Fig. 4 ). That is to say, we can leverage the phenomenon of anomalous decision boundaries to distinguish backdoored models. As claimed in previous work [42] , backdoor attacks build a shortcut leading to the target label, which we explain cause the above encircling phonemona. Besdies, trigger samples are more robust against distortions [36] , causing the large regions than that of clean samples. In a nutshell, the visualized 2D decision boundary can be served as an illustration for these conjectures.Based on the intriguing phenomenon, drawing an analogy to X-rays in disease diagnosis, we propose Model X-ray as a novel backdoor detection approach through the analysis of illustrated 2D decision boundaries. Specifically, we designate two metrics to evaluate the degree of the closeness of the decision boundary: 1) Rényi Entropy (RE) [38] calculated on the probability distribution of each prediction area and 2) Areas Dominated by triple samples (ATS), e.g., the total areas of "frog", "ship", and "dog" in the Fig. 1 . Furthermore, if only one label is infected, we can determine the target label by the prediction of the largest area of the decision boundary, e.g., the target label is "airplane" in the right of Fig. 1 . In other words, Model X-ray can not only identify backdoored models but also determine the target attacked label under all-to-one attacks. Importantly, Model X-ray accomplishes this only by the predicted hard labels of clean inputs from the model, regardless of any assumptions about attacks such as the trigger patterns and training details. The visualized 2D decision boundary offers a novel perspective to understand the behavior of the model, providing both visualization and interpretability. Through analysis of the decision boundary, Model X-ray can function as a lightweight diagnostic scanning tool, complementing other defense methods and aiding in defense pipelines. Extensive experiments demonstrate that Model X-ray performs better than current methods across various backdoor attacks, datasets, and model architectures. In addition, some ablation studies and discussions are also provided.Our contributions can be summarized as follows:• We present a noteworthy observation: there exists a distinction between clean models and backdoored models by visualized 2D decision boundaries [39] .  
 
 Appendix:  


Decision: A rebuttal letter is submitted. All reviewers consider this work has contributed to the filed of identifying backdoored deep neural networks. A decision of acceptance is converged by three reviewers, and the authors are requested to incorporate the rebuttal letter in final submission. Thus, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper "Model X-ray: Detecting Backdoored Models via Decision Boundary" presents a novel approach to identifying backdoored deep neural networks (DNNs) by analyzing decision boundaries. The proposed method, termed Model X-ray, leverages 2D decision boundary visualizations to distinguish between clean and backdoored models. The technique is based on two metrics: Rényi Entropy (RE) and Areas Dominated by Triple Samples (ATS). Extensive experiments demonstrate the effectiveness and efficiency of Model X-ray across various backdoor attacks, image datasets, and architectures.
strength_weakness: strength_weakness: 1. The use of 2D decision boundary visualizations to detect backdoored models is a novel and interpretable method that adds significant value to the existing body of work in backdoor detection.
2. Model X-ray is designed to work in a post-training scenario without requiring knowledge of the attack information, logit output, or model parameters, making it highly practical and applicable in real-world scenarios.
3. The paper provides a thorough comparison with existing post-training detection methods such as Neural Cleanse, MNTD, and MM-BD, demonstrating superior performance in most cases.
limitations: limitations: 1. The paper contains several typos, which detract from its overall readability and professionalism. Specifically:
 Page 2: "phonemona" should be "phenomena".
 Page 2: "Besdies" should be "Besides".
 Page 7: "Speifically" should be "Specifically".
 Page 7: "realtionship" should be "relationship".
2. The paper compares the proposed method with Neural Cleanse, MNTD, and MM-BD. However, it lacks comparisons with some of the latest state-of-the-art methods published after 2022, such as ASSET, BIRD, and other advanced detection techniques. Including these comparisons would provide a more comprehensive evaluation of the method's effectiveness. 
[1] ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigms
[2] BIRD: Generalizable Backdoor Detection and Removal for Deep Reinforcement Learning
3. The robustness of the proposed method against different levels of attack severity, variations in trigger patterns, and noise is not thoroughly analyzed. Experiments focusing on these aspects would provide a deeper understanding of the method's strengths and limitations.
4. The approach lacks formal theoretical guarantees regarding its detection capabilities. The conditions under which the decision boundary visualization reliably distinguishes between clean and backdoored models are not rigorously defined, leading to potential uncertainties in its effectiveness.
5. The effectiveness of decision boundary-based detection may vary across different domains and types of data. The method's generalizability to diverse application domains, such as natural language processing, medical imaging, or time-series data, is not extensively validated.
paper_summary: paper_summary: The paper introduces Model X-ray, a post-training backdoor detection method based on decision boundary analysis.The method is based on the interesting observation that the decision boundaries of backdoor models exhibit a greater degree of proximity than the decision boundaries of clean models. Therefore, Model X-ray devises two metrics to capture this difference and detect backdoor models. Experiments show that Model X-ray outperforms existing methods in terms of detection accuracy and efficiency.
strength_weakness: strength_weakness: 1. The paper is well organized and easy to understand.
2. The proposed method is novel and provides a new perspective for backdoor detection.
limitations: limitations: The main concern is the evaluation of the proposed method, especially the selection of baselines:

1. For backdoor detection, this paper adopts Neural Cleanse, MNTD and MM-BD as the baseline, which represent three different types of defenses. Neural Cleanse is the first backdoor detection method based on trigger inversion, while the recent trigger-inversion-based methods have achieved much better performance. Therefore, it is necessary to explain the reason for choosing NC in this paper.

2. As claimed in discussion, Blended can bypass the proposed detection method. While it should not be required that the proposed method be valid for all attacks, there should be more validation and analytical description of the invalid cases. In particular, there are a series of backdoor attack methods similar to Blended.

3. Can attackers build up adaptive attacks to bypass the proposed method? This is important for a more comprehensive assessment of the robustness of the proposed method.
paper_summary: paper_summary: In this paper, the authors present an observation that the visualized 2D decision boundaries between clean models and backdoored models can be distinguished. Based on this, they propose Model X-ray, a post-training backdoor detection approach through the analysis of illustrated 2D decision boundaries. The performance of Model X-ray under various scenarios is impressive. The paper is well written.
strength_weakness: strength_weakness: • This paper presents a noteworthy observation that backdoor models have significant different decision boundary with benign models.

  • Model X-ray does not need any assumptions about backdoor attacks and can determine the target label under the all-to-one attack strategy.

  • Good visualization about decision boundary. Two decision boundary-based methods are provided for backdoor detection, which are Rényi Entropy (RE) and Areas dominated by Triple Samples (ATS). Results demonstrate Model X-ray is effective in detecting backdoor models on different datasets, model architectures, and backdoor attacks.
limitations: limitations: • Some important explanations are missing in the paper. For example, the reason why the two strategies were chosen is not illustrated in section 4.2. The experiments on different datasets on the same model architecture are not provided in the paper.

  • The explanation of some results is a bit exaggerated. For example, in Figure 7-8 the backdoored models do not have significantly smaller RE/ATS than clean models. In addition, multiple results in evaluation lack explanation or are presented informally.

  • More discussions on the weakness of Model X-ray are needed.

**Questions:**

  • In Section 5.4, the authors highlighted the efficiency of Model X-ray which requires a small number of clean samples. I am curious about the relationship between the number of clean samples and the performance of Model X-ray. Will more clean samples affect the decision boundaries?

  • The experiments on different datasets on the same model architecture were not provided in the paper. Can you explain why? In addition, in Fig. 5 (left), should p3 be equal to A(3) / A(B_k) based on Formula 2?