Scanning Trojaned Models Using Out-of-Distribution Samples

Authors: hossein  mirzaei
ali  ansari
bahar  dibaei
mojtaba  nafez
moein  madadi
sepehr  rezaee
†  zeinab
sadat  taghavi
arad  maleki
kian  shamsaie
mahdi  hajialilue
jafar  habibi
mohammad  sabokrou
mohammad hossein rohban
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:17:50

Abstract: Scanning for trojan (backdoor) in deep neural networks is crucial due to their significant real-world applications. There has been an increasing focus on developing effective general trojan scanning methods across various trojan attacks. Despite advancements, there remains a shortage of methods that perform effectively without preconceived assumptions about the backdoor attack method. Additionally, we have observed that current methods struggle to identify classifiers trojaned using adversarial training. Motivated by these challenges, our study introduces a novel scanning method named TRODO (TROjan scanning by Detection of adversarial shifts in Out-of-distribution samples). TRODO leverages the concept of "blind spots"—regions where trojaned classifiers erroneously identify out-of-distribution (OOD) samples as in-distribution (ID). We scan for these blind spots by adversarially shifting OOD samples towards in-distribution. The increased likelihood of perturbed OOD samples being classified as ID serves as a signature for trojan detection. TRODO is both trojan and label mapping agnostic, effective even against adversarially trained trojaned classifiers. It is applicable even in scenarios where training data is absent, demonstrating high accuracy and adaptability across various scenarios and datasets, highlighting its potential as a robust trojan scanning strategy.

Main paper: 
 introduction: 
 Deep Neural Network (DNN)-based models are extensively utilized in many critical applications, including image classification, face recognition [1] , and autonomous driving [2] . However, the reliability of DNNs is being challenged by the emergence of various threats [3] , with one of the most significant being trojan (backdoor) attacks. In such attacks, an adversary may introduce poisoned samples into the training dataset, for instance, by overlaying a special trigger on incorrectly labeled images. Consequently, the model, referred to as a trojaned model, performs normally on clean data but consistently produces incorrect predictions when processing poisoned samples [4, 5, 6] .Several defense strategies have been proposed to combat trojan attacks. Trojaned model scanning is among such remedies that deal with distinguishing between trojaned and clean models by finding a poisoned model signature [7, 8, 9, 10, 11] . Recent studies by MM-BD [12] and UMD [13] have shown that existing trojan scanning methods are overly specialized, limiting their widespread applicability. Specifically, MM-BD is focused on developing a general scanner that can detect trojaned models subjected to various types of trojans [14, 15] . Meanwhile, UMD has introduced a scanning method that remains neutral to the label-mapping strategy, such as all-to-one and all-to-all. Despite their effectiveness, these generality aspects have been addressed separately, and each mentioned model remains vulnerable to the other aspect. Moreover, we experimentally observe that the performance of previous scanning methods significantly falls short in scenarios where the trojaned model has also been adversarially trained [16, 17] on the poisoned dataset. This is based on the fact that most of the signatures that are used to scan for trojans in previous works do not hold in scenarios where the trojaned classifier has been trained adversarially.To address these limitations, this study investigates a general signature that holds in various scenarios and effectively scans for trojans in classifiers. Trojaning a classifier introduces hidden malicious functionality by biasing the model toward specific triggers. This is somewhat similar to the so-called "benign overfitting" [18, 19, 12] in which the test accuracy remains high despite the model being overfitted to the trigger that is present in the poisoned training samples. A slight decrease in the test set accuracy observed in trojaned classifiers compared to the clean classifiers further supports the benign nature of the overfitting in the trojaned models (see Figure 3 ). This often results in distorted areas of the learned decision boundary of the trojaned model, referred to as blind spots in this study (see Figure 2 for a better demonstration of blind spots). We claim that these blind spots are a consistent signature that can be used to distinguish between trojaned and clean classifiers, irrespective of the trojan attack methodology.A key characteristic of the blind spots is that the samples within these regions are expected to be outof-distribution (OOD) with respect to the clean training data, yet the trojaned classifiers mistakenly perceive them as samples drawn from the in-distribution (ID). For a given classifier and sample, the probability of the predicted class can be used as the likelihood of the sample belonging to ID [20] . We term this value as the ID-Score of the sample. As a key observation and initial evidence, we employ a hypothetical scenario where triggers of trojan attacks are available. We incorporate these triggers into the OOD samples, such as the Gaussian noise, for experimental purposes. Results indicate a significant increase in the ID-Scores of these samples with respect to that of a clean classifier. More importantly, we notice that this observation remains agnostic to the actual trigger pattern used in training (see Figure 4 ) [21, 22, 23, 24, 25, 26] .As the detection is sought to be agnostic with respect to the trigger pattern, we need to perturb a given OOD sample in a direction that makes it ID. Ideally, this perturbation would regenerate the trigger. Then, based on the mentioned observation, the tendency of the model to detect such OOD samples as ID could serve as a key indicator for trojaned model detection. Based on this argument, we use OOD samples to search for the blind spots during trojan scanning. Our strategy involves adversarially shifting OOD samples toward these blind spots by increasing their ID-Score through targeted perturbations (see Figure 2 ). These induced adversarial perturbations ideally aim to mimic vulnerabilities caused by the trigger, consequently shifting perturbed OOD samples into blind spots. This significantly increases their ID-Scores. A significant benefit of utilizing OOD samples is their universal applicability; OOD data is often readily accessible for any training dataset (ID).Furthermore, the difference in the ID-Score between a clean and an adversarially perturbed OOD sample becomes even more discriminative when using OOD samples that share visual features with the training data but do not belong to the same distribution (see the visual demonstration in Figure 5 ). We call them near-OOD samples. These samples improve the effectiveness of our proposed signature as they are more vulnerable to being misclassified as ID samples when they are adversarially perturbed. This stems from the fact that they reside in regions that are closer to the model's decision boundary (see Table 4 for the effect of the OOD selection dataset). Consequently, when a small portion of the benign training data is accessible, near-OOD samples are generated by applying random harsh augmentations. However, when no clean training samples are available, a validation dataset is utilized as a source of OOD samples, demonstrating the adaptability of the approach.Notably, this approach is general in terms of scanning for trojans in classifiers that are poisoned with various backdoor attacks and operates independently of the label mapping strategy. Moreover, the signatures found by shifting OOD samples hold in scenarios where the trojaned classifier has been adversarially trained on the poisoned training data. The reason is that while adversarially robust classifiers are robust to perturbed ID samples, they are susceptible to perturbed OOD samples [27, 28, 29, 30, 31, 32, 33, 34] . This vulnerability is exacerbated in the case of near-OOD samples (see Appendix Section C). Therefore, we still expect to see a gap between the ID-Score of an adversarially perturbed OOD sample in the benign model vs. trojaned model.We introduce a general scanning method called TRODO, which identifies trojaned classifiers even in scenarios where no training data is available and can adapt to utilize data to improve scanning performance. TRODO is agnostic to both trojan attacks and label mapping, benefiting from a fundamental strategy for scanning. Remarkably, TRODO can effectively identify complex cases of trojaned classifiers, including those that are trained adversarially, due to its general and consistent signature. Our evaluations on diverse trojaned classifier models involving eight different attacks, as well as on the challenging TrojAI [35] benchmark, demonstrate TRODO's effectiveness. Notably, TRODO achieves 79.4% accuracy when no data is available and 90.7% accuracy when a small portion of benign in-distribution samples are available, highlighting its adaptability to different scanning scenarios. Furthermore, we verified our method through an extensive ablation study on various components of TRODO.PGD(X OOD , )x OODx OOD* PGD(X OOD , )x OODx OOD*ID Score Δ ID Score Δ f Trojaned θ f Clean θ G Harsh Augmentation x OOD x ID Normal Training Samples X OOD X OOD* Poisoned Training Samples G A B-1 B-2Figure 1 : An overview of TRODO A) If a small portion of benign training samples was available, a module shown as G is used to obtain near-OOD samples. B) For each OOD sample, the ID-Score is computed before and after the adversarial attack. The difference between these scores is used as a signature to distinguish between a clean and a trojaned classifier. Performing the adversarial with not a large budget helps to discriminate between benign and trojaned classifiers 1) Lack of blind spots in the learned decision boundary of a clean model, makes it difficult to increase the ID-Score of OOD samples, resulting in small change in ID-Score. 2) For a trojaned model, ∆ID-Score is more discernible. This is due to the presence of blind spots, making it easier to shift OOD samples inside the decision boundary. 
related work: 
 Trojan Scanning. Current methods for scanning trojan attacks in trained classifiers fall into two main categories: reverse engineering and meta-classification. Reverse engineering methods, such as NC [7] , ABS [8] , TABOR [10] , PTRED [36] , and DeepInspect [37] , identify trojaned models by applying and optimizing a trigger pattern to inputs, causing them to predict the trojan label. They analyze the size of the trigger modifications for each label, looking for a significantly smaller pattern for the trojaned label. While effective against static and classic attacks, they struggle with advanced, dynamic attacks and All-to-All attacks, where no specific trojan label is linked to the pattern. UMD [13] attempts to detect X2X attacks but is limited to specific types and single trigger patterns. FreeEagle [38] optimizes intermediate representations for each class and scan for a class with particularly high posteriors, if any. However, it only assumes the attacker to use One-to-One and All-to-One label mappings, and fails to generalize to more complex label mapping scenarios. Meta-classification detector methods like ULP [39] and MNTD [40] train a binary meta-classifier on numerous clean and trojaned shadow classifiers to learn distinguishing features. These methods perform well on known attacks but fail to generalize to new backdoor attacks and require extensive computational resources to train shadow models [41] . Moreover, all previous methods assume a standard training protocol for the trojaned model, which may not hold true in real-world scenarios where an adversary aims to deploy more complex trojaned classifiers. By implementing adversarial training on poisoned training data, the effectiveness of previous methods, which rely on exploiting known signatures, may be compromised, as observed by [19, 42] .ID-Score and OOD Detection Task. A classifier trained on a closed set, can be utilized as an OOD detector by leveraging its confidence scores assigned to input test samples, referred to as ID-Score in this study. Here, the closed set is the training set used for the classification task, and the samples within this set are called ID samples. Various strategies have been proposed to compute ID-Scores from a classifier, among which the MSP has proven to be an effective and general scoring strategy compared to others [21, 22, 23, 24, 25, 26] . The classifier assigns higher ID-Scores to samples that belong to the ID set and lower scores to OOD samples. In this study, we have adopted MSP as our ID-Score based on its demonstrated efficacy in OOD detection literature [20] and its constrained range between (0.0, 1.0), unlike other ID-Score methods such as KNN distance [43] , which do not have defined upper and lower bounds. We consistently employ MSP in our methodology, hypothesizing that an MSP value of 0.5 (we call this value boundary confidence level and denote it as γ) signifies regions near the classifier's decision boundary. Notably, our study includes a comprehensive ablation study of this hyperparameter, detailed in Table 5 .Adversarial Risk. Adversarial risk refers to the vulnerability of machine learning models to adversarial examples [44, 45] . Previous work has established bounds on this metric via function transformation [46] , PAC-Bayesian [47] , sparsity-based compression [48] , optimal transport and couplings [49] , or in terms of input dimension [50] . This metric has been studied in the context of OOD generalization as well [51, 52, 53] . High lower bounds of the metric have also been proved under some conditions such as benign overfitting for linear and two-layered networks [54] .For an extended related work, see Appendix Section E. The effect of using near-OOD samples Given a trojaned classifier trained on CIFAR10, due to the presence of blind spots in the learned decision boundary, it is easier to increase the ID-Score of near-OOD samples (a fish is considered as near-OOD for CIFAR10) than that of far-OOD samples (samples from MNIST are far-OOD for CIFAR10). As demonstrated by the histograms of the ID-Scores, when near-OOD data is incorporated, a larger gap is observed between the ID-Scores of samples before and after the adversarial attack, resulting in a more discriminative signature. 
attacker capabilities and goals: 
 In the context of attacker capabilities, adversaries can poison training data [4, 14] or manipulate the training process [5, 55] to embed backdoors within models. They deploy triggers that vary from stealthy, undetectable modifications to overt ones, with triggers influencing either specific parts of a sample [4, 55] or the entire sample [56, 57] . Additionally, attackers can target individual samples [58] to evade detection or use label-consistent mechanisms, where poisoned inputs align with their visible content, leading to inference misclassification [59, 56] . Attacks typically follow either an All-to-One pattern, where any input with a trigger is classified into a single target class, or an All-to-All pattern, where a target class is chosen for each source class to ensure any input with a trigger is misclassified accordingly. These models may be trained either adversarially or non-adversarially, with attackers aiming to embed undetectable backdoors that evade detection efforts.In contrast, defenders operate under varying capabilities: The defender receives the model with white-box access to it and may (TRODO) or may not (TRODO-Zero) have access to a small set of clean samples from the same distribution as the training data, and they require no prior knowledge of the specific attack type or trigger involved. Defender goals are to identify any embedded backdoors, and adapt effectively to scenarios with or without clean training samples. 
method: 
 Overview. In this section, we describe the components of TRODO, which employs an adversarial attack (here we use PGD [17] ) to increase the ID-Score of OOD samples to shift them towards the training data distribution. We then measure the magnitude of the difference in ID-Scores between OOD samples and their perturbed counterparts. We denote this as the ID-Score difference (∆ID-Score) and use it as a signature to scan for trojans. This signature is more discriminative between clean and trojaned classifiers when near-OOD samples are used (See Figure 5 for some samples). Unlike many existing trojan scanners, which fail in setups lacking training data, TRODO can successfully conduct scans owing to its robust and universal signature. Further details are provided in subsequent sections. The pseudocode of our scanning algorithm is provided in 1.OOD Set Crafting. To obtain a set of OOD samples, we propose two scenarios. In the first scenario, a portion of the clean training data is available for the given classifier. Here, the OOD set is obtained by applying transformations known to compromise the semantic integrity of an image. Although the results of these transformations deviate from the ID characteristics, these transformed samples visually resemble ID ones. We utilize these as proxies for near-OOD samples. To ensure that the transformations significantly alter the sample characteristics and shift them far enough from the training data distribution, we define a set of hard transformations T = {T i } k i=1 , with each T i representing a specific type of hard augmentation. For each ID sample x, a random permutation of T is selected {T j1 , T j2 , . . . , T j k }, and the transformations are sequentially applied, resulting in T j k (. . . T j1 (x)). This method generates a diverse set of OOD samples, particularly valuable in environments with limited access to training data. Each transformed training sample x becomes a crafted OOD sample x ′ , with the transformation process denoted by G(•), i.e., x ′ = G(x). We set k = 3 as a rule of thumb. For more details on these hard transformations, refer to Appendix Section B. In the second scenario, where no training data is available, we employ a smaller dataset as the OOD set. Specifically, we utilize Tiny ImageNet [60] for this purpose. Considering that many training datasets (e.g., CIFAR-10 [61]) share concepts with our OOD set, we apply G(•) on Tiny ImageNet samples before using them as the OOD set, ensuring that they do not reflect the training distribution characteristics. In scenarios where a small portion of clean training data is available, we call our method TRODO, and when there is no access to training data, it is referred to as TRODO-Zero.Adversarial Attack on ID-Score. In this section, we formulate an adversarial attack on OOD samples to shift them toward the ID region. First, we define the maximum softmax probability (MSP) as the ID-Score, which is an indicator of the classifier's confidence in recognizing an input sample belonging to the ID. Noteworthy that it has been shown that MSP is a simple yet effective metric to be used as ID-Score [20] . The adversarial perturbation aims to find a shortcut path to increase the ID-Score, effectively shifting the OOD sample toward the blind spots of the trojaned classifier. This process results in a significant increase in the ID-Score, highlighting the introduced signature. Formally, the PGD attack to the ID-Score for a sample x corresponding to a classifier f can be formulated as:where the noise is projected on the ℓ 2 norm ball S with radius ϵ around x in each step: ∥x t * -x∥ 2 ≤ ϵ.To define our signature, we assume a set of OOD samples denoted as D OOD = {x OOD i } is available. For a given classifier f , we define our signature S(f, D OOD ) as:EQUATIONwhere x OOD* i is obtained by adding adversarial perturbation to x OOD i via a PGD attack mentioned in above equation 1.A higher value of S(f, D OOD ) indicates that f is trojaned with higher probability. To detect whether a classifier f is trojaned, we utilize a validation set and a thresholding mechanism, which is well described in the next part.Leveraging Validation Set for Trojan Scanning. In this study, we assume access to a benign validation set denoted as D v (e.g., Tiny ImageNet), which is realistic given the abundance of available datasets in real-world scenarios. We craft an OOD set D OOD by applying the mentioned strategy, i.e., D OOD = G(D v ). Note that we apply harsh augmentations to ensure that the OOD dataset does not belong to ID (in case the validation dataset's distribution resembles training data distribution). These datasets are used for computing ϵ for our Projected Gradient Descent (PGD) attack as mentioned in the above equations. Moreover, leveraging them, we propose a threshold mechanism to determine whether an input classifier is trojaned, using the signature S(f, D OOD ).Initially, we note that the ID-Score of an OOD sample x resembles a uniform distribution U(K), and the ID-Score f (x OOD ) is approximately equal to 1 k , where k denotes the number of classes in the training data. We propose that an effective ϵ should shift OOD samples toward ID regions. We consider 0.5 as a hyperparameter, denoted by γ, which we refer to as the boundary confidence level. As a result, we propose computing ϵ by finding the minimum perturbation that can increase the ID-Score (i.e., MSP) from 1 k to 0.5 for the crafted OOD set D OOD , corresponding to a surrogate classifier g as a clean trained model. Specifically, we use the method proposed in DeepFool [62] to find the minimum perturbation that can satisfy the mentioned constraint:EQUATIONThreshold Computing. Once the signature value S(f, D OOD ) has been computed for the given classifier f , it is critical to determine whether f has been compromised by a trojan, using a thresholdbased strategy. This process is achieved by employing a statistical test on a set of scores computed for a surrogate classifier g. Specifically, given the surrogate classifier g and the OOD set D OOD , we generate a set of baseline scores denoted as {S i (g, D OOD )} N i=1 . These scores represent the signature values assigned by a clean classifier g. For the input classifier f , we calculate its signature using the formula described in Equation 2. When the model is trojaned, its corresponding signature will be an outlier to the distribution of S i (g, D OOD ). We estimate this null distribution with a Normal distribution to find a threshold τ satisfying P rob( maxi=1,...,N -log(1 -S i (g, D OOD )) ≤ τ ) > 0.95.Solving for τ , gives the following threshold: τ = Φ -1 ( N √ 0.95), where Φ is the CDF of our estimated truncated normal distribution and we set N = 50. We refer to τ as scanning threshold. 
theoretical analysis: 
 In this section, we provide theoretical insights that underline the susceptibility of trojaned models to adversarial perturbations, particularly in near-OOD regions.Notation. In this section, L1 and L2 norms are denoted by |.| and ∥.∥ respectively. Y = Ω(X) is equivalent to Y ≥ cX for all X ≥ X0 where c, X0 ∈ R + are some constants. For vectors x = (xi) d i=1 , γ = (γi) d i=1 , and function h, we define:x γ = x γ 1 1 . . . x γ d d , ∇ γ x h = ∂ |γ| h ∂ γ 1 x 1 ...∂ γ d x d , ∇xh = [ ∂h ∂x 1 , . . . , ∂h ∂x d] ⊤ , and γ! = γ1! . . . γ d !.We aim to show that a neural network is more sensitive to adversarial perturbations when it receives a backdoor attack, especially in near-OOD data. Let h(w, x) : R dw × R dx → R be a black-box function (e.g., loss or output of a neural network) with learnable parameters w and input x.Adversarial risk of h in radius α under a distribution P is defined as follows:R P α (h, w) := Ex∼P sup ∥δ∥≤α h(w, x + δ) -h(w, x) ≈ αEx∼P ∥∇xh(w, x)∥.The approximation converges as α → 0, thus we use the last term in our analysis similar to [50, 54] .We formulate a near-OOD around P by shifting only the moments of an order k. Formally, for any k ∈ N and s ∈ R, we defineP k +s by E x∼P k +s [x v ] = Ex∼P [x v ] + s for any v ∈ N dx 0 with |v| = k , and E x∼P k +s [x u ] = Ex∼P [x u ] for any u ∈ N dx 0 with |u| ̸ = k.The following theorem shows that the adversarial risk under P k +s will increase linearly in terms of |s|. The proof is given in Appendix Section F.Theorem 1. (Adversarial risk in near-OOD)R P k +s α (h, w) ≥ α|s| max x ∥∇x |γ|=k ∇ γ x h(w, x) γ! ∥ -α∥Ex∼P ∇xh(w, x)∥.Remark 1. Theorem 1 is applicable when ∇ k+1 x i h ̸ = 0 which is usually true if h contains non-linear exponential activation functions (e.g., softmax, sigmoid, tanh, ELU, and SELU) being infinitely many times differentiable, or if it contains polynomial activation functions with total degree greater than k + 1. Under this assumption, if we consider h(w, .) as a fixed model trained on a fixed distribution P, then the only variable in the lower bound will be |s| hence we conclude RP k +s α (h, w) = Ω(|s|).We now study how the adversarial risk will increase under a backdoor attack.Let D = {(xi, yi) = w ⋆⊤ xi) : 1 ≤ i ≤ n} with xi iid ∼ P be the clean training set, D ′ = {(x ′ i + t, yc) : 1 ≤ i ≤ m} with x ′ i iid∼ P be the poisoned training set, t ∈ R dx be the trigger, and yc be the target class of the attack. We consider ŵ as the optimal solution of the least square optimization on the data D ∪ D ′ :EQUATIONWe focus on linear and two-layer networks defined as follows:h1(w, x) = w ⊤ x, h2(w, x) = 1 √ ldx l j=1 ujReLU(θ T j x),where in the latter w = [θ ⊤ j , uj] l j=1 ∈ R l(dx+1) represents the vectorized parameters of the network, with each pair [θ ⊤ j , uj] ∈ R dx+1 , and ReLU(z) = max{0, z} is the activation function. We approximate h2(w, x) using the neural tangent kernel (NTK) [63] method with first-order Taylor expansion around an initial point w0:h2(w, x) = h2(w0, x) + ∇wh2(w0, x) T (w -w0).We use the same gradient descent training process as in [54] . The following theorem shows that as the ratio of triggered samples, i.e., m n , or the norm of the trigger t increases, then the adversarial risk will also increase linearly. The proof is given in Appendix Section F. Theorem 2. (Adversarial risk after backdoor attack) for h ∈ {h1, h2}, if ŵ is learned through the Equation 4on a fixed training distribution P, we have:lim n→∞ R P α (h, ŵ) = Ω m n ∥t∥ .Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We have provided the code and the notebook to easily reproduce results. We also have provided access to our benchmark in New Assets item in the checklist. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We evaluated our proposed method across a diverse range of benchmarks and compared its performance with various existing scanning methods. We developed our benchmark, which includes models trained on a broad spectrum of image datasets. This benchmark includes trojaned models for which various attack scenarios have been considered. The results of these experiments are provided in Table 1 . Furthermore, we present an evaluation of TrojAI in Table 2 as a challenging benchmark.Baselines. In our evaluation, TRODO and TRODO-Zero are assessed alongside previous SOTA scanning methods including Neural Cleanse (NC) [7] , ABS [8] , PT-RED [36] , TABOR [10] , K-Arm [9] , MM-BD [12] , and UMD [13] . Performance details are in Table 1 , with further information in Appendix Section I and K. Implementation Details. As stated earlier, we used Tiny ImageNet as our validation set to tune our hyperparameters ϵ and τ (scanning threshold); details are provided in Table 10 . We used PGD-10 as the adversarial attack. Our experiments on our method and other baselines were conducted on a single RTX 3090 GPU.Our Designed Benchmark. We developed a benchmark to model real-world scanning scenarios, including various datasets, classifiers, trojan attacks, and label mappings. This benchmark covers both standard and adversarial training methods, ensuring a comprehensive evaluation of scanning methods. Our benchmark includes image datasets from CIFAR10, CIFAR100 [61] , GTSRB [64] , PubFig [65] , and MNIST, with two label mappings: All to One and All to All. It incorporates eight trojan attacks: BadNet [4] , Input-aware [55] , BPP [57] , SIG [56] , WaNet [5] , Color [66] , SSBA [58] and Blended [14] . Each combination of a dataset and label mapping has 320 models: 20 trojaned models per attack and 160 clean models (check Appendix Section N for more details). Both standard and adversarial training were employed. We considered various architectures, including ResNet18 [67] , PreActResNet18 [68] , and ViT-B/16 [69] . While previous works focused on CNN-based architectures, our experiments are more general. Table 1 presents the evaluation of ResNet18; evaluations of other architectures are in Appendix Section O, with more details on our benchmark creation in Appendix Section K.TrojAI Benchmark. The TrojAI [35] benchmark, developed by IARPA, addresses backdoor detection challenges and includes test, hold-out, and training sets with nearly half of the models being trojaned. These models may have various backdoor triggers, such as pixel patterns and filters, activated under specific conditions. More details are in Appendix Section K.Analysis of the Results. As the results indicate, presented in Tables 1 and 2 , TRODO surpasses previous scanning methods by a large margin in terms of accuracy and time. Specifically, TRODO achieves superior performance with an 11.4% improvement in scenarios where trojan classifiers have been trained in a standard (non-adversarial) setting and a 24.8% improvement in scenarios where trojan classifiers have been adversarially trained. Our method demonstrates superior performance in both All-to-One and All-to-All scenarios, highlighting the generality of our proposed method. Notably, TRODO-Zero, which operates without access to any training samples, preserves significant performance compared to other methods, with only a minor drop in performance compared to TRODO. The same trend holds on TrojAI, a well-known and challenging benchmark. Regarding scanning time, as shown in Table 2 , TRODO demonstrates high computational efficiency, achieving competitive accuracy with significantly lower scanning time compared to other methods. This is mainly due to the simple yet effective signature it uses to scan for trojans. Further experimental results, including error bars, qualitative visualizations, and the limitations of our work, can be found in the Appendix Section O.Adaptive Attack. In our analysis of Adaptive Attacks on TRODO, we define two strong approaches aimed at circumventing the model's defense mechanism. The first adaptive strategy trains a classifier with a custom loss function designed to equalize the confidence level (ID-Score) for both in-distribution (ID) and out-of-distribution (OOD) samples. This loss function, defined asLadaptive1 = E (x,y)∼D in [-log fy(x)] -λ1E (z,y)∼Dout [H(U ; f (z))] + λ2E (x,y)∼D in [H(U ; f (x))]where x, y are data samples and their labels, f y (x) denotes the y-th output of the classifier, U is the uniform distribution over classes, and H is the cross-entropy. The first term is the classification term (cross-entropy), while the other terms force the classifier to decrease MSP (ID-Score) for ID samples while increasing it for OOD samples. Setting λ 1 = λ 2 = 0.5, inspired by [70] , balances the importance of the first term. By this loss function, we hope the ID-Score for both OOD and ID samples will be altered, though the classifier's decisions remain fixed.Additionally, we introduce a second loss function targeting TRODO's detection signature by reducing the ID-Score gap between benign and perturbed OOD samples, making it challenging for TRODO to distinguish trojaned classifiers from clean ones. This second loss function is defined asLadaptive2 = E (x,y)∼D in [-log fy(x)] -λ3E (z,y)∼Dout [H(f (x); f (x * ))]where x * denotes the adversarially perturbed sample. Although these attacks attempt to subvert our defense, TRODO's use of random transformations in creating OOD samples provides resilience, as these transformations hinder the model's ability to learn patterns that could be exploited by an adaptive adversary. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: The implementation details of the creation procedure of our benchmark, and how we adapted baselines to the experiments and how we tune our parameters are discussed in Section 6. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Due to space constraints in the main body of the paper and to avoid overcrowding the tables, we have included the statistical significance of our method's experiments in Appendix Section O. For the baselines, we have documented the average performance across five repeated trials.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: In the "Implementation Details" subsection of the Experiments section (see Section 6), we have thoroughly outlined the computational resources utilized. Given that the efficient execution of our method is a key advantage, we have included comprehensive details about the computing resources employed to assess our method in comparison with other approaches.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
conclusion: 
 In conclusion, this study presents TRODO, a robust and general method for scanning and identifying trojaned classifiers with low time and resource complexity. TRODO's strength lies in its ability to detect trojans in diverse scenarios, including those involving adversarially trained models. Interestingly, TRODO is applicable even in scenarios where no data is available. Our experimental results demonstrate TRODO's superior performance, achieving high accuracy across various attack types and benchmark datasets. The adaptability and effectiveness of our approach mark a significant advancement in enhancing the reliability and security of deep neural networks in critical applications. x Inx In + = , f BadNet x In + = , f BPP x In + = , f Color x In + = , f Wanet x In + = , f Blended x In + = , f Sig x In + = , f SSBA x In + = , f InputAware f BadNet f BPP f Color f Wanet f Blended f Sig f SSBA f InputAwareQuestion: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In the appendix, we have allocated a section to discuss the broader impacts of our research (See Appendix Section J. Our study introduces a method designed to identify potential backdoors embedded within classifiers. As a result, our work contributes positively to societal impacts by enhancing security measures in machine learning applications and mitigating risks associated with malicious interventions.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: No safeguards are needed, as we are discussing the subject of detection of backdoor attacks, and not to incur them.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Poisoned Samples We provided images of some near-OOD data corresponding to random samples of our dataset (see Figure 5 ). Adversarial training methods AT [74] and HAT [75] , designed to enhance model robustness by exposing the classifier to perturbed data during training, generally improve a model's resilience against adversarial attacks within its training distribution. However, studies [27, 29] indicate a potential limitation when such classifiers are evaluated in OOD detection tasks, where a small perturbation in attack can cause a sample from the closed set to be classified as an anomaly and vice-versa. This limitation arises because the models do not consider samples from the open set during training. We provide Table 6 from [27] that highlights the issue. Here MD [76] , Relative MD [77] , and OpenMax [78] are common methods in OOD detection literature to leverage a classifier as OOD detector. The results reported for each outlier method correspond to the best-performing detection method. Notably, our approach has surpassed the state-of-the-art in robust out-of-distribution setting (ATD) for nearly all datasets.EQUATIONIn addition, to use RMD, one has to fit a N (µ 0 , Σ 0 ) to the whole in-distribution. Next, the distances and anomaly score for the input x ′ with pre-logits z ′ are computed as: T perm ← Randomly Permute(T )EQUATION7:for t ∈ T perm do 8:x ← t(x) x ′ ← G(x, T ) 16:D OOD ← D OOD ∪ {x ′ } 17: end for 18: ∆I ← ∅ 19: for x ∈ D OOD do 20:Adversarial Perturbation:21: x * ← PGD(f θ , x, ϵ) 22:ID score computation:EQUATIONS after ← max i=1,...,c f i θ (x * )∆ID ← S after -S before 26:Append ∆ ID to ∆I 27: end for 28: S mean ← Backdoor Attacks. Injecting pre-defined triggers into the training data is the most common approach to implement backdoor attacks. BadNet [4] is the first backdoor attack against DNN models, which involves modifying a clean image by inserting a small, predetermined pattern at a fixed location, thus replacing the original pixels. Blended [14] aimed to enhance the invisibility of the trigger pattern by seamlessly blending it into the clean image through alpha blending. SIG [56] utilized a sinusoidal waveform signal as the trigger pattern. To achieve better stealthiness, many attacks with invisible and dynamic triggers have been proposed. Input-aware [55] proposed a training-controllable attack method that simultaneously learned the model parameters and a trigger generator to produce a unique trigger pattern for each clean test sample. For more details regarding other attacks including BPP [57] , SSBA [58] , WaNet [5] , and [66] read Appendix Section H.The phenomenon of benign overfitting, where models perfectly fit noisy data without compromising generalization, was first explored in [18] . They characterized the conditions under which the minimum norm interpolating prediction rule achieves near-optimal accuracy, emphasizing the necessity of overparameterization. Subsequent studies extended these findings to various neural network architectures. Notably [79] delves into sparse interpolating procedures for linear regression with Gaussian data, highlighting conditions under which benign overfitting occurs in overparameterized regimes. Their work establishes lower bounds on excess risk, proving that overfitting can indeed be benign. Two-layer neural networks have also been extensively studied to understand benign overfitting under various conditions. Benign overfitting in two-layer convolutional neural networks is investigated by [80] , identifying a phase transition between benign and harmful overfitting. Similarly, [81] analyzes non-smooth neural networks, providing theoretical insights into when overfitting can remain benign even beyond lazy training scenarios. This exploration is extended to ReLU networks in [82] , demonstrating the conditions that facilitate benign overfitting and the sharp transitions to harmful overfitting. Additionally, [83] shows that overfitting in Sobolev RKHSs can achieve optimal rates without being intrinsically harmful.The phenomenon is also observed in more complex architectures. Gradient-based meta learning is examined in [84] , revealing that benign overfitting in empirical risk minimization (ERM) can extend to meta-learning algorithms like MAML. A refined taxonomy of overfitting in proposed in [85] , identifying tempered overfitting as an intermediate regime between benign and catastrophic overfitting. Benign overfitting has been studied in various other architectures and settings as well [86, 87, 88, 89, 88, 90, 91] .Benign overfitting has been studied in the context of adversarial robustness in [54, 92, 93] . Notably, [54] theoretically shows for linear and two-layer networks that benign overfitting will become harmful overfitting under adversarial attacks.The interplay between benign overfitting and security vulnerabilities like backdoor attacks is critical in [94] revealing that few-shot learning models tend to overfit benign or poisoned features, impacting robustness.Adversarial risk Adversarial risk refers to the vulnerability of machine learning models to adversarial examples-perturbations intentionally crafted to mislead the model. Research in this area seeks to understand and mitigate these risks. A seminal work by [95] presented robust optimization techniques to defend against first-order adversarial attacks, establishing foundational adversarial training methodologies. Following this, other studies have explored the theoretical limits of adversarial robustness. A framework to evaluate the adversarial vulnerability of any classifier is provided in [96] , showing intrinsic limitations based on the classifier's architecture and data distribution.Previous work has established bounds on this metric via function transformation [46] , PAC-Bayesian [47] , sparsity-based compression [48] , Optimal Transport and Couplings [49] , or in terms of input dimension [50] .The intersection of adversarial risk and out-of-distribution (OOD) detection has garnered increasing attention. The vulnerabilities in existing OOD generalization methods to adversarial attacks are identified in [51] , prompting the development of algorithms to enhance OOD adversarial robustness. RATIO was introduced by [53] , a training procedure that improves adversarial robustness for both indistribution and OOD samples, thereby enhancing model explainability. The adversarial vulnerability of current OOD detection techniques is discussed in [52] , suggesting that ensemble methods and combining multiple OOD detectors can significantly enhance robustness against adversarial attacks.Understanding the theoretical limits of adversarial vulnerability remains crucial for developing robust models. It is proved in [50] that adversarial vulnerability increases with the gradients of the training objective and scales with the square root of the input dimension, making larger images more vulnerable. The trade-offs between robustness and accuracy are explored in [96] , establishing a mathematical framework to evaluate these limits. This discussion is extended in [95] through robust optimization, quantifying the trade-offs, and providing guidelines for creating more resilient models. These foundational works underscore the inherent challenges in achieving robustness, emphasizing the need for innovative approaches to bridge the gap between theory and practical applications.Proof. Let µ ∈ R dx be arbitrary, by taylor series around µ, we have:h(w, x) = |γ|=k+1 ∇ γ x h(w, µ) γ! (x -µ) γ + j̸ =k+1 |γ|=j ∇ γ x h(w, µ) γ! (x -µ) γBy taking derivative we have:∇ x h(w, x) = ∇ x |γ|=k+1 ∇ γ x h(w, µ) γ! (x -µ) γ + j̸ =k+1 ∇ x |γ|=j ∇ γ x h(w, µ) γ! (x -µ) γThe distribution P k +s has the same j-th order moments as the distribution P for all j ̸ = k, therefore, by taking expectation and using the triangle inequality we have:R P k +s α (h, w) = αE x∼P k +s [∥∇ x h(w, x)∥] ≥ α∥E x∼P k +s ∇ x h(w, x)∥ ≥ α∥E x∼P k +s   ∇ x |γ|=k+1 ∇ γ x h(w, µ) γ! (x -µ) γ   -E x∼P   ∇ x |γ|=k+1 ∇ γ x h(w, µ) γ! (x -µ) γ   ∥ -α∥E x∼P k +s   ∇ x h(w, x) -∇ x |γ|=k+1 ∇ γ x h(w, µ) γ! (x -µ) γ   +E x∼P   ∇ x |γ|=k+1 ∇ γ x h(w, µ) γ! (x -µ) γ   ∥ = α∥s∇ x |γ|=k ∇ γ x h(w, µ) γ! ∥ -α∥E x∼P   ∇ x h(w, x) -∇ x |γ|=k+1 ∇ γ x h(w, µ) γ! (x -µ) γ   +E x∼P   ∇ x |γ|=k+1 ∇ γ x h(w, µ) γ! (x -µ) γ   ∥ = α|s|∥∇ x |γ|=k ∇ γ x h(w, µ) γ! ∥ -α∥E x∼P [∇ x h(w, x)] ∥Note that all moments with orders less than k are equal, hence the difference of the moment of order k around µ is equal to the difference of the moment of order k around 0. Since µ was arbitrary, we conclude:R P k +s α (h, w) ≥ α|s| max x ∥∇ x |γ|=k ∇ γ x h(w, x) γ! ∥ -α∥E x∼P ∇ x h(w, x)∥.Linear neural networkProof. We define X = [x 1 , . . . , x n ] ⊤ ∈ R n , X ′ = [x ′ 1 + t, . . . , x ′ m + t] ⊤ ∈ R m , Y = [y 1 , . . . , y n ] ⊤ ∈ R n , Y ′ = [y c , . . . , y c ] ⊤ ∈ R m , then we have: ŵ = X X ′ ⊤ X X ′ -1 X X ′ ⊤ Y Y ′ = X ⊤ X + X ′⊤ X ′ -1 X ⊤ Y + X ′⊤ Y ′Note that lim n→∞ n X ⊤ X -1 = C is a constant matrix (inverse of the covariance matrix) and lim n→∞ 1 n n i=1 x i = s is a constant vector (mean vector), and we have lim n→∞ m i=1 x ′ i = ms by the law of large numbers. If rank(X ′⊤ X ′ ) = 1, then according to the Miller's Lemma [3] :lim n→∞ ∥n X ⊤ X + X ′⊤ X ′ -1 = C - Ck 1 + tr(k) ∥ = Ω(1)wherek = X ′⊤ X ′ X ⊤ X -1 ≈ m n X ⊤ X + mst ⊤ + ms ⊤ t + mtt ⊤ X ⊤ X -1 . If rank(X ′⊤ X ′ ) ≥ 2,we can decompose it into sum matrices with rank 1 and inductively infer the same bound Ω(1). Now we have:lim n→∞ ∥ 1 n X ⊤ Y + X ′⊤ Y ′ = 1 n X ⊤ Y + m n y c (s + t ⊤ )∥ = Ω( m n ∥t∥).Therefore, we conclude:lim n→∞ R P α (h 1 , ŵ) = lim n→∞ ∥∇ x h 1 ( ŵ, x)∥ = lim n→∞ ∥ ŵ∥ = Ω m n ∥t∥ .Proof. We defineF = [h 2 (w 0 , x 1 ), . . . , h 2 (w 0 , x n )] ⊤ ∈ R n , ∇F = [∇ w h 2 (w 0 , x 1 ), . . . , ∇ w h 2 (w 0 , x n )] ⊤ ∈ R k(p+1)×n , F ′ = [h 2 (w 0 , x ′ 1 +t), . . . , h 2 (w 0 , x ′ m +t)] ⊤ ∈ R m , ∇F ′ = [∇ w h 2 (w 0 , x ′ 1 + t), . . . , ∇ w h 2 (w 0 , x ′ m + t)] ⊤ ∈ R k(p+1)×m , y = [y 1 , . . . , y n ] ⊤ ∈ R n , and y ′ = [y c , . . . , y c ] ⊤ ∈ R m .Assume the two-layer network h 2 (w, x) as described in the paper is trained using gradient descent on D ∪ D ′ with learning rates η < 1/λ max ([∇F, ∇F ′ ] ⊤ [∇F, ∇F ′ ]). According to Proposition 1 in [97] , the optimal solution will be:ŵ = ∇F ∇F ′ ⊤ ∇F ∇F ′ -1 ∇F ∇F ′ ⊤ Y -F Y ′ -F ′ = ∇F ⊤ ∇F + ∇F ′⊤ ∇F ′ -1 ∇F ⊤ (Y -F ) + ∇F ′⊤ (Y ′ -F ′ )For any i ∈ {1, . . . , m}, j ∈ {1, . . . , k} we have:∇ w h 2 (w 0 , x ′ i + t) j = 1 √ kp u 0,j ReLU ′ (θ T 0,j (x ′ i + t))(x ′ i + t), ReLU(θ T 0,j (x ′ i + t)) = 1 √ kp u 0,j (x ′ i + t), θ T 0,j (x ′ i + t) , if θ T 0,j (x ′ i + t) ≥ 0 1 √ kp 0, 0 , otherwiseTherefore, we can assume there are some constant matricesG 1 , G 2 , G ′ 1 , G ′ 2 such that F ′ = tG 1 + G 2 and ∇F ′ = tG ′ 1 + G ′ 2 . Moreover, note that lim n→∞ n ∇F ⊤ ∇F -1 = C 1 is a constant matrix(inverse of the covariance matrix). If rank(∇F ′⊤ ∇F ′ ) = 1, then according the Miller's Lemma [3] :lim n→∞ ∥n ∇F ⊤ ∇F + ∇F ′⊤ ∇F ′ -1 = C 1 - C 1 k 1 + tr(k) ∥ = Ω(1)wherek = ∇F ′⊤ ∇F ′ ∇F ⊤ ∇F -1 ≈ m n ∇F ⊤ ∇F + Ω(∥mt∥) ∇F ⊤ ∇F -1 . If rank(∇F ′⊤ ∇F ′ ) ≥ 2,we can decompose it into sum matrices with rank 1 and inductively infer the same bound Ω(1). Now we have:lim n→∞ ∥ 1 n ∇F ⊤ (Y -F ) + ∇F ′⊤ (Y ′ -F ′ ) ∥ = Ω( m n ∥t∥).Therefore, we conclude:lim n→∞ ∥ ŵ∥ = Ω m n ∥t∥ .Now we have:R P α ( h2 , ŵ) = E x sup ∥δ∥≤α h2 ( ŵ, x + δ) -h2 ( ŵ, x) = αE x ∥∇ x h2 ( ŵ, x)∥ = αE x ∥∇ x h 2 (w 0 , x) + ∂ 2 h 2 (w 0 , x) ∂w∂x ( ŵ -w 0 )∥ = Ω(∥ ŵ∥)Therefore, we conclude:lim n→∞ R P α ( h2 , ŵ) = lim n→∞ ∥ ŵ∥ = Ω m n ∥t∥ .Adverserial attack to classifiers It has been shown that DNNs are vulnerable to adversarial attacks across various tasks, predominantly explored in classification tasks. During inference, adversarial perturbations added to input data can cause classifiers to mispredict their labels. In other words, a perturbation such as δ is added to a sample x with label y to fool the model into outputting ŷ for the adversarial input sample x * , where x * = x + δ. Specifically, PGD is an iterative adversarial attack [17] that crafts adversarial samples by ensuring the noise is projected within the ℓ ∞ norm in N-step:x * 0 = x, x * t+1 = Π x+S (x * t + α • sign (∇ x J (x * t , y))), x * = x * Nwhere Π x+S denotes the projection on the norm ball S around x and J (x * t , y) denotes the targeted objective function, which for classifiers is cross entropy. Previous studies have shown that standardtrained classifiers are vulnerable to adversarial attacks, even weak attacks like FGSM. Various defense mechanisms have been proposed to address this challenge, with adversarial training on the training dataset being the most effective defense.Using a Classifier as an OOD Data Detector Classifiers such as f , trained on a dataset denoted as D, can be used as OOD detectors due to their learned features. Specifically, another dataset with separate semantics from D, denoted as D ′ , is assumed to be the source of OOD samples where D represents in-distribition samples. For instance, a classifier trained on Cifar10 is used to detect Cifar100 as OOD samples. The instructions to detect OOD samples practically involve using the output distribution of a classifier over the classes of D for a given input, offering insights into the model's prediction confidence. Specifically, classifiers tend to be more confident about ID samples compared to OOD samples. recently The Maximum Softmax Probability (MSP) has been proposed as an indicator of this confidence. formally a well-trained classifier f logits from its penultimate layer on D with k classes, for a given input sample x are represented by z = f (x), where z is a vector of unnormalized prediction scores [z 1 , z 2 , . . . , z k ] for sample x. Applying the softmax function results in:softmax(z i ) = e zi k j=1 e zj , MSP f (x) := max i∈{1,...,k} (softmax(z i ))where z i is the logit corresponding to the i-th class.intutitively MSP f (x) represents the highest probability assigned to any class by the model, reflecting the model's confidence level in its prediction.This section provides detailed descriptions of the backdoor attacks employed in our study, focusing particularly on the nature and implementation of their triggers.BadNet [4] embeds a malicious trigger, typically a small and visually distinctive patch, into the training data. This trigger is designed to be inconspicuous enough to evade detection yet recognizable by the trained model, leading it to misclassify inputs containing the trigger.Blended [14] subtly blends a trigger into the training images at low intensity, making it hard to detect by human inspectors or simple automated methods. The trigger effectively conditions the model to associate the slightly altered patterns with incorrect outputs.SIG [56] corrupts training images with a sinusoidal pattern, superimposed in a way that does not require changes to the image labels. This makes the backdoor particularly stealthy as it avoids the common detection methods that look for label inconsistencies.BPP [57] uses image quantization combined with contrastive adversarial learning to craft triggers that are embedded into the pixel values themselves. This approach alters the image at a bit-per-pixel level, making the modifications difficult to perceive or reverse-engineer.Input-aware attacks [55] dynamically adjust their triggers based on the input features, allowing the backdoor to activate only under specific conditions that are predetermined by the attacker. This adaptability makes the attack highly elusive and challenging to detect.WaNet [5] introduces imperceptible warping to the image, manipulating its geometric properties subtly. This warping acts as a trigger that is extremely hard to spot with the naked eye, ensuring the model misclassifies the warped input while appearing normal to human observers.SSBA [58] crafts sample-specific triggers that are invisible to human detection by embedding them in a way that aligns closely with the natural image structure. These triggers are tailored to each individual sample, increasing the difficulty of detecting and isolating the backdoor through general analysis.Color [66] alters the color distribution of the input images, employing changes in the color channels as the trigger mechanism. This type of manipulation can remain under the radar of typical visual inspections while effectively conditioning the model to respond to altered color cues.These descriptions underline the diversity of the backdoor mechanisms used, highlighting the challenges in detecting and mitigating such threats in machine learning models.We implemented the baseline methods using their implementations in their publicly available GitHub repositories. For methods requiring supervision to define a threshold, we followed the same approach as UMD.Review of the Methods NC Neural Cleanse [7] is a method for scanning models by reverse engineering triggers and identifying outliers with significantly smaller perturbations. NC faces significant computational overhead, sensitivity to trigger complexity, and potential false positives. Moreover it is primarily designed to handle only All-to-One attacks.ABS Artificial Brain Stimulation [8] detects backdoors in neural networks by stimulating individual neurons and analyzing their impact on output activations. Compromised neurons that substantially elevate a specific label are identified, and potential triggers are reverse-engineered to confirm the presence of backdoors. the method involves significant computational overhead and is sensitive to its underlying assumptions about compromised neurons and trigger behavior. ABS may struggle with more advanced attack scenarios and primarily handles All-to-One attacks. Additionally, benign neurons with unique features may lead to false positives, limiting its robustness in some cases.TABOR TABOR [10] is a trojan detection method for DNNs that frames the detection task as an optimization problem. It incorporates an objective function with regularization terms inspired by explainable AI techniques to guide the optimization process and reduce the adversarial search space, improving trigger identification accuracy. . However, TABOR's significant computational overhead present challenges. Additionally, it is primarily designed for geometric or symbolic triggers, potentially limiting effectiveness against irregular shapes trojans.PT-RED PT-RED [36] is a post-training backdoor detection method for DNNs. It reverse-engineers potential backdoor trigger patterns by solving an optimization problem to find minimal perturbations that cause misclassifications. However, it is primarily designed for single target class attacks with small trigger patterns. Moreover, the reverse-engineering process for identifying potential triggers is computationally intensive.K-ARM K-Arm [9] leverages an optimization approach inspired by the Multi-Armed Bandit problem to iteratively select and optimize class labels for potential backdoor triggers. This method improves detection efficiency compared to exhaustive search methods by using stochastic selection. However, the method still involves computational overhead and primarily focuses on specific and simple types of backdoor attacks. Additionally, K-Arm may face challenges in handling scenarios with more than one target label.UMD Unsupervised Model Detection [13] is designed to detect X2X backdoor attacks, where multiple source classes are mapped to multiple target classes. The method involves reverse-engineering triggers for each class pair using a small set of clean data, and then defining a transferability statistic (TR) for each class pair. TR measures how well the trigger for one class pair transfers to another. These statistics are used to select likely backdoor class pairs. An unsupervised anomaly detector then evaluates the aggregated trigger statistics to determine if a model is backdoored. However, the method involves complex optimization processes, which can be computationally intensive. Handling very large datasets or a high number of classes can pose scalability challenges. Furthermore, UMD may struggle to handle models with multiple different trigger patterns, as it relies on TR statistics that assume a single dominant trigger pattern and this method is not trigger-agnostic as its approach depends on the specific type of attack.MM-BD Maximum Margin Backdoor Detection [12] is designed to detect backdoor attacks in neural networks, regardless of the backdoor pattern type. The method operates by estimating a maximum margin statistic for each class through gradient ascent from multiple random initializations, without any clean samples, and then using these statistics in an unsupervised anomaly detection framework to identify backdoor attacks. However, the method exhibits a large false-positive rate for datasets with a small number of class, and it struggles to detect attacks with more than one target label, where each source class may be mapped to a different target class. Additionally, MM-BD's effectiveness is significantly reduced when an adaptive attacker manipulates the learning process.MNTD MNTD [40] identifies backdoors in neural networks by training a binary meta-classifier on features extracted from numerous shadow classifiers, both benign and Trojaned. However, this method struggles to generalize to types of attacks and model architecture it wasn't trained on.Our study introduces a method designed to identify potential backdoors embedded within classifiers. As a result, our work contributes positively to societal impacts by enhancing security measures in machine learning applications and mitigating risks associated with malicious interventions.In our evaluation, TRODO and TRODO-Zero are assessed alongside previous scanning methods including Neural Cleanse (NC) [7] , ABS [8] , PT-RED [36] , TABOR [10] , K-Arm [9] , MM-BD [12] , and UMD [13] . For an equitable comparison, we set the confidence threshold at 95%, corresponding to a 5% desired false positive rate for NC, PT-RED, and UMD which are based on unsupervised threshold settings. Similarly, for ABS, and K-Arm, which rely on supervised threshold adjustment, we maintained a consistent false positive rate of 5% across various tests and dataset configurations to maximize true positive outcomes. The capabilities and performance outcomes of these methods are detailed in Table 1 . Further details can be found in Appendix Section I.To effectively model real-world scenarios for the scanning task, we developed a benchmark covering various scenarios involving both clean and trojan classifiers. Specifically, to evaluate the generality of scanning methods, which is crucial in real-world applications, we included several datasets ranging from low to high resolution and different classifiers with various architectures. Moreover, different trojan attacks and label mappings were also considered, with classifiers covering both standard and adversarial training methods. More details about our developed benchmark are provided below.Image Datasets Our benchmark comprises image datasets from various domains, including CIFAR10, CIFAR100 [61] , GTSRB [64] , PubFig [65] , and MNIST. We considered two kinds of label mapping, All to One and All to All.Trojan Attacks In the former, the label of poisoned samples is changed to a single target class. In the latter, unlike All to One case, each class can be mapped to any arbitrary target class, which makes this setting more challenging. We included eight trojan attacks, comprising BadNet [4] , Input-aware [55] , BPP [57] , SIG [56] , WaNet [5] , Color [66] , SSBA [58] and Blended [14] . The test set for each combination of image dataset and label mapping consists of a total of 320 models. We trained 20 trojaned models using each type of attack. We included 160 clean models, resulting in a balanced set.Adversarial Training To encompass a wider variety of scenarios, we evaluated each configuration using both standard and adversarial training. We employed PGD-10 with ϵ = 2 255 for adversarial training.Classifier Architectures We considered various architectures, including ResNet18 [67] , PreActRes-Net18 [68] , and ViT-B/16 [69] . Previous works have solely focused on CNN-based architectures, underscoring the generality of our experiments.Another challenging benchmark that we include in our experiments is TrojAI [35] , a benchmark developed by IARPA designed to address challenges in backdoor detection. The license of this benchmark is Apache 2.0.Structure For each round of the competition, a test set, a hold-out set, and a training set of models are available and can be accessed from the TrojAI homepage. Almost half of the models in each set are trojaned. A small set of benign samples from the training dataset of each model is provided along with the model itself.Trojan Attacks The models may be trojaned with various kinds of backdoors, including universal and label-specific. The triggers could be pixel patterns and Instagram filters. Triggers can be embedded within the model such that activation occurs only under specific conditions, such as possessing a certain texture or being located in a designated area of the image. The complexity of models and trojan attacks grows from round to round. The performance of methods on this benchmark is provided in Table 2 . We evaluate methods in terms of scanning accuracy and average scanning time for a model.The Fréchet Inception Distance (FID) is a metric used to evaluate the quality of generative models, such as GANs. It compares the distribution of generated images to the distribution of real images by measuring the distance between two multivariate Gaussian distributions fitted to the feature representations of these images.Given a set of real images {x i } Nr i=1 and a set of generated images {x j } Ng j=1 , pass both sets through a pre-trained Inception v3 network to obtain feature representations. Let ϕ(x) denote the feature representation of image x from the Inception network.Calculate the mean and covariance of the feature representations for real images:µ r = 1 N r Nr i=1 ϕ(x i ) Σ r = 1 N r -1 Nr i=1 (ϕ(x i ) -µ r )(ϕ(x i ) -µ r ) TSimilarly, compute the mean and covariance for generated images:µ g = 1 N g Ng j=1 ϕ(x j ) Σ g = 1 N g -1 Ng j=1 (ϕ(x j ) -µ g )(ϕ(x j ) -µ g ) TThe FID is defined as the Fréchet distance between the two multivariate Gaussian distributions N (µ r , Σ r ) and N (µ g , Σ g ):FID = ∥µ r -µ g ∥ 2 + Tr(Σ r + Σ g -2(Σ r Σ g ) 1 2 )Here, ∥µ r -µ g ∥ 2 is the squared Euclidean distance between the mean vectors. Tr denotes the trace of a matrix. (Σ r Σ g ) 1 2 is the matrix square root of the product of the two covariance matrices. Lower FID scores indicate that the generated images have a distribution more similar to the real images, suggesting higher quality. Higher FID scores indicate greater dissimilarity between the distributions of generated and real images, suggesting lower quality.In this study, we utilized a validation set and a surrogate model to select hyperparameters, such as epsilon for the PGD attack. However, these hyperparameters are architecture-specific. Therefore, each new architecture requires a tailored approach: training on a validation set and tuning its corresponding hyperparameters for the input classifier. This process can be time-consuming when encountering new architectures, although we limited our consideration to common architectures. Additionally, our research focuses on scanning trojaned classifiers. However, this task might need to be adapted for different tasks, such as object detectors, where our method would require adjustments to handle these cases effectively. Morever, our theoretical results on adversarial risk are limited to the noiseless settings and only address the backdoor effect up to two layered networks which could be extended in future work.In our study, we generated 20 distinct models for each of the eight backdoor attack types across the ResNet18 and PreAct-ResNet18 architectures and 5 models for ViT for each dataset and attack, totaling 160 models for ResNet18 and PreAct-ResNet18 and 40 models for ViT. We utilized the Back-doorBench framework [98] (https://github.com/SCLBD/BackdoorBench) for seven attacks (BadNets, Blended, WaNet, SIG, BPP, Input-aware, and SSBA) and integrated an additional attack (Color attack [66] ) using this repository (https://github.com/lyx1224/color-backdoor).To ensure consistency and rigor, we adopted core framework settings from BackdoorBench and extended them to encompass both standard and novel attacks, adjusting parameters specifically for the COLOR attack. Our consistent methodology, following BackdoorBench's protocols, allowed us to systematically evaluate the individual effects of each backdoor strategy across the architectures, achieving a high standard of experimental reliability and detail. Performance of TRODO under different poisoning ratesIntuitively, increasing the poisoning rate enlarges the blind spots in trojaned classifiers, as these are boundary regions where the poisoned data causes the model to overfit. Consequently, this will increase the probability that TRODO detects the trojaned classifiers. However, our signature is based on the presence of blind spots in trojaned classifiers and shows consistent performance across different poisoning rates. In the paper, we considered a poisoning rate of 10% as it is common in the literature. In addition, we have provided TRODO's performance for different poisoning rates in Table 12 (other components of TRODO remained fixed.) In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification: After a thorough review of the NeurIPS Code of Ethics, we are confident that our work fully complies with all its aspects. Additionally, our research adheres to ethical standards concerning social impact and harmful consequences, based on both our proprietary code and the publicly available datasets employed.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).J(f (x)) = ID-Score f (x), x 0 * = x, x t+1 * = Πx+S (x t * + α • sign ∇xJ(f (x t * ))) , x * = x N * (1) 


Decision: The paper deals with the security of machine learning (ML) methods. Specifically, the paper proposes "TRODO", a novel defense to address the problem of "trojaned/backdoored" ML models, whose training process was compromised with adversarial manipulations that have been deliberatedly introduced by attackers. Despite being a well-known security issue, an effective solution to identifying "trojaned" models has yet to be found and TRODO seeks to overcome the limitations of prior work (such as, e.g., having poor performance when the ML model is subject to adversarial training on the poisoned dataset). TRODO is claimed to provide a "general" solution, allegedly being "agnostic" to the type of trojaning attack being employed.

The manuscript was reviewed by 6 referees with varying expertise in the application domain tackled by the paper. The reviewers praised the magnitude of the evaluation, the results, the underlying idea, the presentation/writing quality, and the fact that the paper also sheds light on the theoretical rationale behind the effectiveness of the proposed method. However, the reviewers also expressed some concerns, such as an unclear threat model, the lack of comparison with some state-of-the-art anti-trojaning methods, and the lack of consideration of an adaptive attacker.

Three reviewers, after interacting with the authors, recommend that the paper is worthy of a "6: Weak Accept", whereas another reviewer recommends a "5: Borderline Accept". In contrast, one reviewer provided a very critical review, recommending a "3: Reject" and despite actively participating in the discussion (both with the authors, as well as with other reviewers), their recommendation did not change. Finally, one reviewer recommends a "4: Borderline Reject", and is confident of the assessment, but did not acknowledge to having read the authors' response---which, in my opinion, have addressed the concerns expressed by this reviewer. 

Due to the above, I am fairly confident that this paper should be accepted: even though the _submitted manuscript_ was providing a contribution that was not yet ready to be presented at a top-tier venue such as NeurIPS, the authors' efforts in responding to the reviewers' valid remarks have shed further light on the claimed contribution, allowing it to meet the expectations of NeurIPS.

===============

To be more precise, the original submission provided a contribution that was deemed to be theoretically sound, correctly evaluated, and properly presented/motivated. Doing all of this in a (relatively) short manuscript is challenging. However, some important elements were left out, such as the effectiveness of the proposed method against an adaptive adversary (as pointed out by various reviewers), which is a necessary evaluation to carry out whenever a new "defense" is proposed. The authors have acknowledged such a shortcoming (including admitting that some attackers may completely bypass the method), and have also proceeded to carry out an evaluation considering some attacks that can be staged against TRODO.

From a security standpoint, it is well-known that "no defense is foolproof". Hence, it should not come at a surprise that TRODO cannot work against very strong attackers. The author-reviewer interactions enabled to clarify this holds also for TRODO, thereby "warning" downstream users/researchers of the limitations of TRODO in the real world. For this reason, I think that there are no grounds to reject the paper. However, the authors are encouraged to enhance their manuscript by acknowledging these limitations (potentially by toning down the overall effectiveness of TRODO) and incorporating the new experiments and clarifications that emerged during the peer review (potentially by referring to the appendix---or even to the very discussion that transpired here on OpenReview!)

Reviews: review: paper_summary: paper_summary: The paper introduces a novel trojan scanning method named TRODO (TROjan scanning by Detection of adversarial shifts in Out-of-distribution samples). TRODO leverages the concept of "blind spots," where trojaned classifiers mistakenly identify out-of-distribution (OOD) samples as in-distribution (ID). The method scans for these blind spots by adversarially shifting OOD samples towards in-distribution, using the increased likelihood of these perturbed samples being classified as ID as a signature for trojan detection. TRODO is both trojan and label mapping agnostic, effective even against adversarially trained trojaned classifiers, and applicable even when training data is absent.
strength_weakness: strength_weakness: Undefined Threat Model: The threat model is not clearly defined, and the adversary's capabilities are not explicitly listed.

Lack of Comparison with SOTA Baselines: There is an absence of comparison with SOTA baselines, such as FreeEagle. The results listed in Table 2 (baseline-NC) differ significantly from those in FreeEagle's Table 4. Additionally, many baselines from FreeEagle's Table 4, such as STRIP[2] and ANP[3], are not discussed.

[1]Fu, C., Zhang, X., Ji, S., Wang, T., Lin, P., Feng, Y., & Yin, J. (2023). {FreeEagle}: Detecting Complex Neural Trojans in {Data-Free} Cases. In 32nd USENIX Security Symposium (USENIX Security 23) (pp. 6399-6416).
[2]Gao, Y., Xu, C., Wang, D., Chen, S., Ranasinghe, D. C., & Nepal, S. (2019, December). Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th annual computer security applications conference (pp. 113-125).
[3]Wu, D., & Wang, Y. (2021). Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34, 16913-16925.
questions: questions: 1. Can you clarify your threat model?
2. Why do you have so many missing baselines?
limitations: limitations: Design is too simple
paper_summary: paper_summary: This paper points out a limitation of existing backdoor model scanning methods:
They fail to detect backdoored models trained with adversarial training. It
propose a new backdoor model scanning method by utilizing adversarial shifts in
Out-of-distribution samples. Experiments on MNIST, CIFAR-10, GTSRB, CIFAR-100,
and Pubfig show the effectiveness of the proposed method.
strength_weakness: strength_weakness: * The paper states that experiments were conducted using ResNet18, PreActResNet18,
and ViT-B/16 models. However, Table 1 only presents results for ResNet18. While
the paper claims that results for other models are included in Appendix Section
M, but this section also only contains ResNet18 results. It would be
beneficial to explicitly present the detection accuracy for each model across
various attack scenarios.

* The discussion about the adaptive attacks to the proposed method is missing,
where the attacker knows about the proposed defense strategy and actively tries
to evade or overcome it. For example, the adaptive attacker might able to add a
loss during the backdoored model construction phase to reduce the change of the
ID-Score on the backdoored models.

* While theoretical analysis of the proposed method is highly recommended, the
paper would benefit from more intuitive explanations of this analysis. There
appears to be a lack of detailed clarification regarding the connection between
the theoretical analysis and the proposed method. For instance, the relationship
between Theorem 2 and the fundamental principles of the proposed method is not
clearly stated. Providing more intuitive explanations of these theoretical
analysis would strengthen the paper.
questions: questions: See Weaknesses.
limitations: limitations: The limitations have been discussed in this paper.
paper_summary: paper_summary: The paper proposes TRODO, a method for identifying trojaned classifiers, which relies on the intuition that in presence of a backdoor it should be easier than for clean classifiers to make the model classify an out-of-distribution input as in-distribution by adding an adversarial perturbation. In practice, a PGD attack is run on OOD images to increase the confidence of the classifier predictions, and the average increase is used to detect backdoored models. In the experimental evaluation, the proposed method outperforms the baselines, being effective against several backdoor attacks even when combined with adversarial training.
strength_weakness: strength_weakness: - The proposed method seems to have brittle elements:
  - the PGD-10 attack used for computing the signature might be considered weak (only 10 iterations), then not able to fully optimize the target loss. Then, a stronger attack could further increase the confidence of the clean classifiers on OOD points, making them more similar to trojaned one (since the confidence is upper bounded, one could get the same score for all classifiers in the worst-case). Thus, the effectiveness of TRODO seems to rely to some extent on the attack not fully optimizing the target loss (which might be even exploited to bypass the detection mechanism).
  - the confidence of any classifier might be adjusted post-training by e.g. temperature rescaling without changing its decisions. In this way, it seems possible to bypass the detection by making a model under-confident.
  - adversarial training variants, e.g. on OOD data [60] or to have uniform predictions far from ID data [A], have been explored in prior works, and might be used to counter the proposed scanning scheme.

- It's not clear how the effectiveness of TRODO correlates with the strength of the backdoor attacks. For example, one can imagine that using a lower poisoning rate might make the attack less detectable (but less effective).

[A] https://arxiv.org/abs/1910.06259
questions: questions: - Why using a left-truncated normal distribution for estimating $\tau$ when the score S can take values only on a specific range (confidence is upper bounded by 1)?

- While the proposed method provides good experimental results, I think it's important to address the concerns about its robustness (see above).
limitations: limitations: Limitations are partially addressed.
paper_summary: paper_summary: This paper proposes a general strategy for distinguishing between trojaned and clean models. The generality of the approach lies in its applicability to various types of trojan attacks, different label-mapping strategies, and its ability to work with and without clean training data. The authors claim that distorted areas of the learned decision boundary in trojaned models, referred to as blind spots, serve as a consistent signature for distinguishing between trojaned and clean classifiers, regardless of the trojan attack methodology. A key characteristic of these blind spots is that samples within these regions are expected to be out-of-distribution (OOD) relative to the clean training data, yet trojaned classifiers mistakenly identify them as in-distribution (ID) samples. To leverage this characteristic, the authors propose using adversarial attacks to perturb a given OOD sample towards an ID direction, followed by computing the difference in ID score (i.e., maximum softmax probability) before and after the adversarial attack. They found that trojaned models exhibit significantly larger differences in these scores. The effectiveness of the proposed detection method was empirically validated against eight trojan attacks and under different levels of access to clean training data.
strength_weakness: strength_weakness: As mentioned in the limitation section, the selections of $\epsilon$ and $\tau$ rely on a validation set and a surrogate model. As a result, the quality of these hyperparameters depend on the choices of the validation set and the surrogate model. In particular, when detecting a trojaned model with a new model architecture or trained on a new domain, one might need to tune $\epsilon$ and $\tau$.
questions: questions: 1. Could you clarify what it means for a trojaned classifier to be adversarially trained on poisoned training data? (For example, for poisoning samples generated by a label-consistent trojan attack, what does it mean by adversarially training the trojaned model on these data?) Additionally, are there any empirical results demonstrating the effectiveness of the proposed method in this specific scenario?
2. When a portion of clean training data is not accessible, the authors propose to use Tiny-ImageNet. In addition, it comes to my mind that given the accessibility to the model, one could actually reverse engineer the training data. In that way, you could obtain a portion of fake clean training data, followed by creating near-OOD samples.Consequently, the performance without clean training data could approach that of having a portion of real clean training data, thereby diminishing the performance gap between TRODO-Zero and TRODO.
In particular, the reverse engineering of training data has been adopted in many backdoor works, such as [1] (see Figure 3).
3. Headings of paragraphs could be set in a uniform format. Eg., In Line 149, there is a ‘.’ after the paragraph heading; by comparison, in Line 168, there is not.

[1] Liu, Y., Ma, S., Aafer, Y., Lee, W. C., Zhai, J., Wang, W., & Zhang, X. (2017). Trojaning attack on neural networks.
limitations: limitations: The performance of the proposed method drops by a large margin (over 10%) when no clean training data is accessible, compared to that with a portion of clean training data.
paper_summary: paper_summary: The authors propose a trojan scanning technique that leverages the sensitivity of the network's confidence when near-OOD samples undergo an adversarial attack. The authors argue that the greater variation in confidence can be used to discriminate whether a network has been backdoored, and present extensive experiments to support the effectiveness of their method.
strength_weakness: strength_weakness: - W0.1 (On writing) The concepts are very simple, but the presentation is hard to follow. Lines 40-89 refer to figures and concepts that will be displayed in following sections. I would suggest to anticipate significantly the figures. Furthermore, I would suggest providing immediately clear definitions of concepts, that are instead introduced in a handwavy way (e.g., blind spots, benign overfitting) before being properly discussed in related works. Probably completely restructuring Sections 1 and 2 to transfer most of the contents of 1 to 2 could help, having a good 'definitions' or 'preliminaries' section in Section 2. 
- W0.2 (On writing) The formatting of the theorems in the main paper (especially with lack of assumptions) is very unusual and may e confusing. 
- W1: The scanning procedure may be vulnerable to an adaptive trojan attack. Could the authors show what happens if attackers account for the author's defense in their attack strategy? How much does the effectiveness of their technique go down? The attackers may completely elude the proposed scanning method if they account for it in the design of their attack.
- W2:  How sensitive are the hyperparameters to the choice of the validation set? How computationally expensive is it to tune them? 
- W3: Results are adequately reported exclusively for Resnet18, and are not reported exhaustively for other architectures (or their presentation is not really immediately clear).
questions: questions: - Q1: There must be a typo at line 161, I could not find information about hard transformations and therefore cannot verify claims made about them (Appendix Section 5).   The typo "Attention Section" occurs repeatedly throughout the paper. 
. Q2: When using TinyImageNet, why did the authors not simply filter out the classes overlapping with CIFAR-10? This can be easily done and show the effectiveness of the method without applying G(.).
limitations: limitations: - L1: The method seems to assume to know the training distribution of the model so that it is possible to find near-OOD samples. However this may not always be possible or easy (near-ness could depend on many factors other than class similarity, e.g. may depend on resolution of the input, color range, other forms of naturally occurring covariate-shifts, types of shortcuts taken by the models etc.). The method significantly improves when this info is accessible. 
- L2: The previous limitation spills over to the need of having a good validation set. In some cases, the performance can go below the baselines if the validation set is not representative of the training set.
paper_summary: paper_summary: The paper addresses the problem of detecting trojaned models. The paper proposes a trojaned model scanning method using out-of-distribution (OOD) samples. Specifically, it is observed that trojaned classifiers can erroneously identify adversarially attacked OOD samples as in-distribution (ID)  samples. Therefore, the increased likelihood of perturbed OOD samples being classified as ID serves as a signature for trojan detection. The proposed trojan detection method can be applied in two scenarios: avaliability of clean samples and non-avaliability of clean ID samples. Extensive experiments demonstrate that proposed can achieve state-of-the-art performance compared with other scanning trojan detection methods.
strength_weakness: strength_weakness: The motivation (especially in Introduction Section) of proposed method is not presented clearly. For example, in line 73-74, the paper introduces the concept of near-OOD samples. However, the difference of far-OOD and near-OOD samples is not clearly explained. Also, the paper writes that "see the visual demonstration in Appendix Section 5". It is obvious there is no Section 5 in Appendix.

The paper claims that current methods can not identify models trojaned using adversarial training (e.g. in line 6-7 and line 112-115). However, there is no detailed experiments to show the difference between detecting models trojaned using adversarial training and normal training.

Some ablations can be done. For example, the number of used out-of-distribution samples, and the diversity of used OOD samples.
questions: questions: Could you provide some details of used transformations? 

Could you compute FID between transformed samples and original samples?

The paper only compares with scanning methods. How about other kinds of state-of-the-art trojan detection methods?
limitations: limitations: The limitations have been addressed.

Comments: title: Response to Reviewer PnTa
Appreciation for Your Feedback and Review
Response to Reviewer PnTa
Thank You for Your Positive Feedback
Response to Reviewer PnTa
Appreciation for Your Positive Feedback
Response to rebuttal
comment: Dear Reviewer d7KJ,


As we approach the discussion period deadline, we kindly request that you review our rebuttal, in which we have aimed to thoroughly address your comments. If you have any further concerns, we would be most grateful if you could bring them to our attention, and we would be pleased to discuss them.

Sincerely,
The Authors
We thank the reviewer for their feedback. 

We believe the discussion around temperature scaling has diverged from our primary focus. Our intention was to highlight the extreme cases of a miscalibrated classifier, such as those producing one-hot or uniform outputs, which, as the reviewer suggested, are uncommon in real-world scenarios due to their lack of explainability. This is why we did not address them in our paper. While these scenarios may be relevant in theoretical contexts, they are not directly applicable to practical situations. Nonetheless, our adaptive attacks demonstrate that even in the worst-case scenarios, TRODO performs consistently. To **fully address** the reviewer's concerns, as they **agreed**, we will consider applying the softmax function ourselves rather than relying on the classifier's softmax, and we will discuss this in our paper.

>**This might be true, but should be experimentally tested and discussed in the paper.**

Our experiments were conducted under the assumption that temperature scaling was not applied ($T=1$). Therefore, applying softmax ourselves instead of using the classifier’s built-in softmax would not change the results, and the experiments would yield identical outcomes.

>**Overall, the rebuttal has confirmed that the proposed method is susceptible to.... adaptive attacks**

We believe that a fair comparison should also consider the performance of other methods under adaptive attacks for a evaluation. However, even under strong adaptive attacks, our method continues to demonstrate superior performance compared to existing scanning methods.




We remain open to further discussion to address any additional concerns the reviewer may have.

Sincerely,
The Authors
Thanks for the additional reply.

> A miscalibrated classifier [...] can lead to poor decision-making...

In classification tasks as those reported in the paper, calibration is not a factor, since confidence is not used for classification (only argmax).

> Since standard classifiers typically exhibit approximately calibrated outputs...

I think this is not in general precise, that is it's not clear whether classifiers are calibrated before applying post-processing calibration techniques (see e.g. https://arxiv.org/abs/1706.04599).

> However, we believe this issue can be addressed with minimal modifications...

This might be true, but should be experimentally tested and discussed in the paper.

&nbsp;

I think this simple approach (modifying the confidence of the model via temperature rescaling), which doesn't require designing new backdoor attacks, points to weaknesses of the proposed detection method, which can potentially be exploited by more sophisticated adaptive attacks. I think this can't be simply dismissed assuming models are well calibrated, and should be discussed in the paper.

Overall, the rebuttal has confirmed that the proposed method, in the current form, is susceptible to temperature rescaling and (at least partially) other adaptive attacks ($L_{adaptive1}$). Thus, I think the paper would require significant improvements, at least discussing the current limitations and potential countermeasures. Then, since the original main concerns remain, I will vote for rejecting the paper.
Thank you for your valuable review and positive feedback! We are pleased to hear that your concerns have been addressed.

Warm regards,
The Authors
Thanks for your detailed rebuttal. As most of my concerns are addressed, I will increase my rating to 6.
We thank the reviewer for their thoughtful review and valuable feedback.

* A key requirement for deploying deep neural networks in real-world applications is calibrated or approximately calibrated behavior on input samples. A miscalibrated classifier, particularly in cases where $T→0$,  $T>>1$, can lead to poor decision-making. For example, in medical diagnosis, overestimating disease probability can result in unnecessary treatments, while underestimating it may lead to missed diagnoses. Miscalibrated models are especially unreliable in high-stakes scenarios such as finance, healthcare, or autonomous systems, where accurate probability estimates are crucial. Furthermore, the backdoor attack/defense literature generally assumes that Trojaned classifiers behave normally on samples without triggers, similarly to standard classifiers. Since standard classifiers typically exhibit approximately calibrated outputs, we implicitly assume that the Trojaned classifier would also retain this characteristic.

* We acknowledge the challenges posed by the extreme temperature scenarios, specifically when $T→0$ or  $T>>1$, as our ID-score would converge to zero in both cases. However, we believe this issue can be addressed with minimal modifications. In the backdoor attack setup (as detailed in our threat model), it is commonly assumed that the attacker has access to the input model. To mitigate the issue, we propose a minor extension to TRODO: applying its own softmax function to the logits of the input classifier instead of relying on the input classifier's final softmax output. This approach could help counter the adversarial temperature settings highlighted by the reviewer.

* Moreover, since TRODO and TRODO-Zero both utilize an OOD  set, another minor extension could involve evaluating the classifier's softmax output on these samples. If the output significantly deviates from a uniform distribution (e.g. measured by Kullback–Leibler divergence), the model could be rejected as extremely miscalibrated due to the attacker's manipulation.



To further address your concern, we evaluated TRODO's performance on an all-to-one label mapping task using the CIFAR-10 and GTSRB datasets across different temperature values, while keeping other components unchanged. The results show that our designed detector effectively handles a reasonable range of temperature values, maintaining consistent performance throughout.


| CIFAR10 | $T=0.5$     |  $T=0.7$    | $T=1$(default)      | $T=1.2 $   |$T=1.5$         |  $T=2$     |
|-|-|-|-|-|-|-|
| ACC/ACC*            | 90.5/87.1   | 91.2/87.3   | 91.0/88.4    | 89.4/86.2   | 88.4/85.7    | 87.1/85.3   |


| GTSRB | $T=0.5$     |  $T=0.7$    | $T=1$(default)      | $T=1.2 $   |$T=1.5$         |  $T=2$     |
|-|-|-|-|-|-|-|
| ACC/ACC*            | 95.2/92.7   | 94.8/93.0 |96.6/93.2   | 95.7/92.1  |94.3/92.8    |92.7/91.5   |
> Furthermore, as indicated in Fact-A, the softmax output tends to resemble a uniform distribution for OOD samples. As a result, applying temperature rescaling will affect all logits equally and will not alter this uniform shape.

Unless the softmax output is exactly uniform (i.e. all logits are identical), which seems very unlikely, temperature will change the distribution.

> However, by perturbing these samples, the ID score can increase (up to one). Therefore, our signature would not be arbitrarily small as mentioned; it remains within the range of ($[0, 1 - 1/N]$).

I meant arbitrarily small within its range of course (as the ID-Score after the attack cannot be lower than on the clean input).

The point of using temperature >> 1, i.e. making the model under-confident, is that even when the attack is applied it will not increase much the confidence, in particularly since the hyperparameters of the attack such as $\epsilon$ and number of steps is fixed and calibrated on standard models.

Also, one can consider the other extreme case with temperature = 0 (in practice, values close to 0 are sufficient), i.e. the softmax becomes the argmax function. If the argmax of the logits is unique, then the softmax output is a one-hot vector, and even for OOD points the difference of ID-Scores will be 0. This seems to bypass the detection mechanism. Is this correct?
Thank you for your positive feedback and for considering a higher score for our work! We will ensure that these experiments are incorporated to further enhance the manuscript.

Sincerely,
The Authors
Thank you for the clarification and additional experiments. The author should consider adding this content to the article. I have decided to raise my score.
> **Am I missing something?**



* We believe there has been some misunderstanding. When we mentioned that _'the classifier remains more confident in in-distribution samples compared to OOD samples,'_ we were referring to **Fact-A**. We would like to clarify that the principle behind our signature remains valid even when temperature rescaling is applied. The logical flow of our principle is as follows: **Fact-A ⇒ Fact-B ⇒ Fact-C ⇒ Fact-D ⇒ Fact-E ⇒ Fact-F ⇒ Fact-G**.

* Furthermore, as indicated in Fact-A, the softmax output tends to resemble a uniform distribution for OOD samples. As a result, applying temperature rescaling will affect all logits equally and **will not**  alter this uniform shape. Consequently the ID score for clean OOD samples would still be very low (e.g., approximately $\frac{1}{N}$, where $N$ is the number of ID classes). However, by perturbing these samples, the ID score can increase (up to one). Therefore, our signature would not be arbitrarily small as mentioned; it remains within the range of ([0,$1-\frac{1}{N}$]). 


**Fact-A**: A classifier remains more confident in IDs compared to OODs. Specifically, the output softmax of a classifier for OODs tends to be more uniform, while for IDs, it is more concentrated.  (confidence: maximum softmax probablity) [1].


**Fact-B**: Previous research has primarily focused on perturbing IDs, shifting them, and using this as a signature to distinguish between clean and trojaned classifiers [2,3,4,5,6,7,8,9,10].


**Fact-C**: In scenarios where classifiers have been adversarially trained, perturbing ID samples for signature extraction becomes less effective. This is because adversarial training enhances the classifier's robustness to such perturbations, reducing their impact [9,10].


**Fact-D**: Instead of perturbing IDs, perturbing OODs toward the in-distribution is a viable approach, as classifiers are vulnerable to this shift. This approach remains effective even for adversarially trained classifiers, as they are still susceptible to perturbed OODs. This finding is also supported by parallel research in OOD detection [11,12,13,14].


**Fact-E**: Trojaned classifiers learn decision boundaries that include 'blind spots,' which are intuitively regions along the in-distribution boundary that have been altered to overfit on poisoned training data (data with triggers), thereby changing the geometry of the boundary.


**Fact-F**: Perturbing OODs toward the classifier's decision boundary increases their ID scores, and this effect is particularly significant in trojaned classifiers. This is because the perturbations can exploit the blind spots within the decision boundary, mimicking the triggers used during the trojan attack.

**Fact-G***: We use the difference in ID scores between a benign OOD sample (without perturbation) and an adversarially perturbed OOD sample as a signature to differentiate between clean and trojaned classifiers.




---

>**What was the OOD...?**

Following their reported setting, we used Tiny ImageNet as the out-of-distribution dataset for training. 

---

>**In general, these methods.. .**


Regardless of the adaptation strategy, existing classifiers are vulnerable to adversarial attacks on OODs, particularly when these samples are close to the in-distribution boundary. This vulnerability, as demonstrated recently by (Lorenz Deciphering, 2024), (Fort Adversarial Vulnerability, 2022), and others [11,12,13,14], limits them to act as counters to our method. To further illustrate this, we conducted additional experiments using RATIO as an OOD detector, where the MSP is employed as the ID score. In the first scenario, CIFAR-10 serves as the ID dataset and CIFAR-100 as the OOD, and vice versa in the second. While the method achieves good performance in clean scenarios , in adversarial scenarios—where perturbations are added to shift OODs to in-distribution and vice versa—its performance drops to below random detection, highlighting its vulnerability to perturbed OODs.  

| Method\Benchmark | _CIFAR-10 vs.CIFAR-100_         |_CIFAR-100 vs. CIFAR-10_|
|-|-|-|
|              | AUC/AUC*      |AUC/AUC*  |
| RATIO  | 81.3/14.8     |68.5/9.0  |

 
---

>**I think the additional ...**


In the adaptive attack scenario, we evaluated various label mappings and adaptive strategies. The worst performance decrease we observed was 12% (from 86.0% to 74.1%). Despite this, our method still outperforms previous detection methods that were not subjected to adaptive attacks. For instance, UMD achieves 57.7%, while our approach achieves 74.1%.



[1] Hendrycks, A baseline, 2016

[2] Xiang, UMD: Unsupervised  , 2023

[3] Wang, Mm-bd: , 2024

[4] Wang, Neural, 2019

[5] Liu, Abs:Scanning, 2019

[6] Shen, Backdoor scanning for, 2021

[7] Guo, Tabor, 2019

[8] Hu, Trigger hunting, 2022

[9] Edraki, Odyssey, 2021

[10]Zhang, Cassandra, 2021

[11] Chen Robust OOD 2020

[12] Azizmalayeri, Your Detector 2022

[13] Chen, Atom ,2021

[14] Mirzaei, RODEO, 2024
Thanks for the detailed response and additional experiments.

> **W1.2:** "the classifier remains more confident in in-distribution samples compared to OOD samples"

If I understand it correctly, only OOD samples are used to compute the signature. Then, I think the absolute difference $\textrm{ID-Score}_f (x_i^{OOD*}) − \textrm{ID-Score}_f (x_i^{OOD})$ could be (most likely) made arbitrarily small with temperature rescaling, so that it's in the range of the signature of a clean classifier. Similarly, since the confidence is upper bound by 1, also an over-confident classifier which has near 1 confidence on OOD data would lead to very small differences between adversarially perturbed and clean OOD points (since the ID-Score cannot increase much). Am I missing something?

> **W1.3:** "Improving robustness on adversarially out-of-distribution (OOD) samples was not their main purpose"

RATIO loss includes a term which is a robust loss on OOD data, then it directly optimizes adversarial robustness on OOD points. What was the OOD data used for training the RATIO models? In general, these methods show that it is possible to control the (adversarial) confidence on OOD data (but might require some adaptation to the specific task of bypassing the detection mechanism).

> common response

I think the additional experiments with the new losses show that adaptive attacks have the potential to bypass the detection mechanism of TRODO: in fact, $L_{adaptive1}$ improves 12% compared to $L_{default}$ in the All-to-All setup, which seems significant.
Q1: Thanks for the detailed clarification. Now I understand the motivation for the adversarial training. I also appreciate the good detection performance of your method compared with others under the adversarial training scenario.

Q2: Thanks for your clarification, especially "the row mentioning "FastDFKD" is the TRODO-Zero with the synthesized samples using FastDFKD.", which solves my confusion. I also appreciate your intuitive explanation in analyzing the bad performance of using the generated training data.

I will keep my score. Thanks!
Thanks for your feedback. Here are the answers to your questions:
Q1: 
1) Each column in the table 1, above, shows the original training data. The data is first altered by the backdoor attack by adding the backdoor to the input and changing the corresponding output labels based on all-to-one (upper half) or all-to-all (lower half) . Then, adversarial training is applied on the entire dataset including the poisoned and clean samples. Here, each training data, which might be poisoned, undergoes an adversarial attack during the training. As in the regular adversarial training, we only perturb the input and keep the ground truth label unchanged.   

2) The objective function in the adversarial training is the regular cross-entropy (line 593). 

3) The main motivation for evaluating scanning methods on such models is that many previously proposed signatures for trojan scanning experience a detection accuracy drop in the case of an adversarially trained model (lines 37-39). For instance, in the table 1 above, UMD's detection acc. drops by almost 20% points on MNIST and CIFAR-10 on the adversarially trained models compared to the standard models (compare ACC with ACC*). The possible reason behind this effect is that adversarial training makes shortcut learning difficult,  potentially leading to harder trigger reverse-engineering and identification. While previous scanning methods struggle with this issue, our proposed method does not rely on such practices and shows insignificant drop under adversarially trained models. 

Q2: 
2, 3) Please note that FastDFKD (Up to 100× Faster Data-free Knowledge Distillation by Fang et al.) is an efficient method of generating synthetic samples given access to a trained model (see Fig. 3 of that paper). Here, we use this model to create surrogates of the training samples and use them in our method TRODO-Zero to enhance it. So indeed the row mentioning "FastDFKD" *is* the TRODO-Zero with the synthesized samples using FastDFKD. As can be seen, there is still a large gap between the two TRODO and TRODO-Zero (it even got slightly worse). The first row, TrojaNN, (Trojan Signatures in DNN Weights, by Fields et al.), which also assumes no access to the training data is included as a baseline. 

1) Like we mentioned, we hypothesize that the generated samples, through methods such as FastDFKD, could contain shortcuts, potentially imperceptible, making them *unideal* OOD instances, compared to the clean tiny-ImageNet samples that lack such artifacts. Please note that such artifacts would help the classifier to classify the intended synthetic OOD sample as ID more *easily,* reducing the effectiveness of the method, because we take the gap between the ID score of the OOD sample before and after the attack as our detection score (line 179). Therefore, while your idea could generally be beneficial in other methods, it would be less so in our specific method that relies on measuring ID-ness of synthetic OOD samples before and after the attack.
W1: Thanks for the clarification.

Q1: I still have some confusions about adversarially trained models. "To clarify, the training dataset is initially compromised using a backdoor attack. The classifier is then adversarially trained using the PGD (Projected Gradient Descent) attack method."—Could you clarify: 1) the data used for this adversarial training, 2) along with the objective function, and 3) the motivation for this adversarial training? Thanks.

Q2: 1) So according to your claim, can we say that the samples generated by my mentioned's method are less effective than some randomly selected samples from a OOD dataset, e.g. Tiny ImageNet? 2) Is FastDFKD (Fang Data-free 2022) method using models to recover some training data? 2) I think my concern for using models to recover some training data when there is no clean training data arises from the large accuracy gap between TRODO and TRODO-Zero. (My intuition is that the samples generated in this way are more effective than some randomly selected samples from a OOD dataset) Thus, the comparison should be with TRODO-Zero, instead of TRODO.
Thank you for your thoughtful feedback and for taking the time to review our rebuttal! We greatly appreciate your careful evaluation and positive response.

Sincerely,
The Authors
I have read the authors' rebuttals and the other reviewers' opinions. I think several of the concernes raised by other reviewers are fair, but the authors were convincing in their rebuttals. I appreciate the clarification, additional information and experiments performed by the reviewers, especially the introduction of an analysis of their method under adaptive attacks. While this may not be a final guarantee about the robustness of the proposed technique and the limitations pointed out are quite strong, I think this is reasonable and what an attacker would try out first, and the relative robustness of the technique to it indicates a significant research effort would be required in order to further degrade the performance of this technique. Therefore, I am happy to increase my score.