Title: Scanning Trojaned Models Using Out-of-Distribution Samples
Venue: NeurIPS 2024
Accepted: yes
Keywords: Trojan Scanning Method

Trojan Post-Training Defense

Backdoor Attacks

Out-of-Distribution Samples

Adversarially Perturbed Out-of-Distribution Samples
Abstract: Scanning for trojan (backdoor) in deep neural networks is crucial due to their significant real-world applications. There has been an increasing focus on developing effective general trojan scanning methods across various trojan attacks. Despite advancements, there remains a shortage of methods that perform effectively without preconceived assumptions about the backdoor attack method. Additionally, we have observed that current methods struggle to identify classifiers trojaned using adversarial training. Motivated by these challenges, our study introduces a novel scanning method named TRODO (TROjan scanning by Detection of adversarial shifts in Out-of-distribution samples). TRODO leverages the concept of "blind spots"—regions where trojaned classifiers erroneously identify out-of-distribution (OOD) samples as in-distribution (ID). We scan for these blind spots by adversarially shifting OOD samples towards in-distribution. The increased likelihood of perturbed OOD samples being classified as ID serves as a signature for trojan detection. TRODO is both trojan and label mapping agnostic, effective even against adversarially trained trojaned classifiers. It is applicable even in scenarios where training data is absent, demonstrating high accuracy and adaptability across various scenarios and datasets, highlighting its potential as a robust trojan scanning strategy.

Scanning Trojaned Models Using Out-of-Distribution Samples

Abstract

Scanning for trojan (backdoor) in deep neural networks is crucial due to their significant real-world applications. There has been an increasing focus on developing effective general trojan scanning methods across various trojan attacks. Despite advancements, there remains a shortage of methods that perform effectively without preconceived assumptions about the backdoor attack method. Additionally, we have observed that current methods struggle to identify classifiers trojaned using adversarial training. Motivated by these challenges, our study introduces a novel scanning method named TRODO ( TRO jan scanning by D etection of adversarial shifts in O ut-of-distribution samples). TRODO leverages the concept of "blind spots"-regions where trojaned classifiers erroneously identify out-of-distribution (OOD) samples as in-distribution (ID). We scan for these blind spots by adversarially shifting OOD samples towards in-distribution. The increased likelihood of perturbed OOD samples being classified as ID serves as a signature for trojan detection. TRODO is both trojan and label mapping agnostic, effective even against adversarially trained trojaned classifiers. It is applicable even in scenarios where training data is absent, demonstrating high accuracy and adaptability across various scenarios and datasets, highlighting its potential as a robust trojan scanning strategy. The code repository is available at https://github.com/rohban-lab/TRODO .

1 Introduction

Deep Neural Network (DNN)-based models are extensively utilized in many critical applications, including image classification, face recognition [1], and autonomous driving [2]. However, the reliability of DNNs is being challenged by the emergence of various threats [3], with one of the most significant being trojan (backdoor) attacks. In such attacks, an adversary may introduce poisoned samples into the training dataset, for instance, by overlaying a special trigger on incorrectly labeled images. Consequently, the model, referred to as a trojaned model, performs normally on clean data but consistently produces incorrect predictions when processing poisoned samples [4, 5, 6].

Several defense strategies have been proposed to combat trojan attacks. Trojaned model scanning is among such remedies that deal with distinguishing between trojaned and clean models by finding a

poisoned model signature [7, 8, 9, 10, 11]. Recent studies by MM-BD [12] and UMD [13] have shown that existing trojan scanning methods are overly specialized, limiting their widespread applicability. Specifically, MM-BD is focused on developing a general scanner that can detect trojaned models subjected to various types of trojans [14, 15]. Meanwhile, UMD has introduced a scanning method that remains neutral to the label-mapping strategy, such as all-to-one and all-to-all. Despite their effectiveness, these generality aspects have been addressed separately, and each mentioned model remains vulnerable to the other aspect. Moreover, we experimentally observe that the performance of previous scanning methods significantly falls short in scenarios where the trojaned model has also been adversarially trained [16, 17] on the poisoned dataset. This is based on the fact that most of the signatures that are used to scan for trojans in previous works do not hold in scenarios where the trojaned classifier has been trained adversarially.

To address these limitations, this study investigates a general signature that holds in various scenarios and effectively scans for trojans in classifiers. Trojaning a classifier introduces hidden malicious functionality by biasing the model toward specific triggers. This is somewhat similar to the so-called 'benign overfitting' [18, 19, 12] in which the test accuracy remains high despite the model being overfitted to the trigger that is present in the poisoned training samples. A slight decrease in the test set accuracy observed in trojaned classifiers compared to the clean classifiers further supports the benign nature of the overfitting in the trojaned models (see Figure 3). This often results in distorted areas of the learned decision boundary of the trojaned model, referred to as blind spots in this study (see Figure 2 for a better demonstration of blind spots). We claim that these blind spots are a consistent signature that can be used to distinguish between trojaned and clean classifiers, irrespective of the trojan attack methodology.

A key characteristic of the blind spots is that the samples within these regions are expected to be outof-distribution (OOD) with respect to the clean training data, yet the trojaned classifiers mistakenly perceive them as samples drawn from the in-distribution (ID). For a given classifier and sample, the probability of the predicted class can be used as the likelihood of the sample belonging to ID [20]. We term this value as the ID-Score of the sample. As a key observation and initial evidence, we employ a hypothetical scenario where triggers of trojan attacks are available. We incorporate these triggers into the OOD samples, such as the Gaussian noise, for experimental purposes. Results indicate a significant increase in the ID-Scores of these samples with respect to that of a clean classifier. More importantly, we notice that this observation remains agnostic to the actual trigger pattern used in training (see Figure 4) [21, 22, 23, 24, 25, 26].

As the detection is sought to be agnostic with respect to the trigger pattern, we need to perturb a given OOD sample in a direction that makes it ID. Ideally, this perturbation would regenerate the trigger. Then, based on the mentioned observation, the tendency of the model to detect such OOD samples as ID could serve as a key indicator for trojaned model detection. Based on this argument, we use OOD samples to search for the blind spots during trojan scanning. Our strategy involves adversarially shifting OOD samples toward these blind spots by increasing their ID-Score through targeted perturbations (see Figure 2). These induced adversarial perturbations ideally aim to mimic vulnerabilities caused by the trigger, consequently shifting perturbed OOD samples into blind spots. This significantly increases their ID-Scores. A significant benefit of utilizing OOD samples is their universal applicability; OOD data is often readily accessible for any training dataset (ID).

Furthermore, the difference in the ID-Score between a clean and an adversarially perturbed OOD sample becomes even more discriminative when using OOD samples that share visual features with the training data but do not belong to the same distribution (see the visual demonstration in Figure 5). We call them near-OOD samples. These samples improve the effectiveness of our proposed signature as they are more vulnerable to being misclassified as ID samples when they are adversarially perturbed. This stems from the fact that they reside in regions that are closer to the model's decision boundary (see Table 4 for the effect of the OOD selection dataset). Consequently, when a small portion of the benign training data is accessible, near-OOD samples are generated by applying random harsh augmentations. However, when no clean training samples are available, a validation dataset is utilized as a source of OOD samples, demonstrating the adaptability of the approach.

Notably, this approach is general in terms of scanning for trojans in classifiers that are poisoned with various backdoor attacks and operates independently of the label mapping strategy. Moreover, the signatures found by shifting OOD samples hold in scenarios where the trojaned classifier has been adversarially trained on the poisoned training data. The reason is that while adversarially

robust classifiers are robust to perturbed ID samples, they are susceptible to perturbed OOD samples [27, 28, 29, 30, 31, 32, 33, 34]. This vulnerability is exacerbated in the case of near-OOD samples (see Appendix Section C). Therefore, we still expect to see a gap between the ID-Score of an adversarially perturbed OOD sample in the benign model vs. trojaned model.

Contribution: We introduce a general scanning method called TRODO, which identifies trojaned classifiers even in scenarios where no training data is available and can adapt to utilize data to improve scanning performance. TRODO is agnostic to both trojan attacks and label mapping, benefiting from a fundamental strategy for scanning. Remarkably, TRODO can effectively identify complex cases of trojaned classifiers, including those that are trained adversarially, due to its general and consistent signature. Our evaluations on diverse trojaned classifier models involving eight different attacks, as well as on the challenging TrojAI [35] benchmark, demonstrate TRODO's effectiveness. Notably, TRODO achieves 79.4% accuracy when no data is available and 90.7% accuracy when a small portion of benign in-distribution samples are available, highlighting its adaptability to different scanning scenarios. Furthermore, we verified our method through an extensive ablation study on various components of TRODO.

PGD

(

X OOD

,

)

x OOD

x OOD

*

PGD

(

X OOD

,

)

x OOD

x OOD

*

ID Score

Δ

ID Score

Δ

f

Trojaned

θ

f

Clean

θ

G

Harsh Augmentation

x OOD

x

ID

X OOD

X OOD

*

Poisoned Training Samples

G

A

B-1

B-2

Q

名

2 Related Work

Trojan Scanning. Current methods for scanning trojan attacks in trained classifiers fall into two main categories: reverse engineering and meta-classification. Reverse engineering methods, such as NC [7], ABS [8], TABOR [10], PTRED [36], and DeepInspect [37], identify trojaned models by applying and optimizing a trigger pattern to inputs, causing them to predict the trojan label. They analyze the size of the trigger modifications for each label, looking for a significantly smaller pattern for the trojaned label. While effective against static and classic attacks, they struggle with advanced, dynamic attacks and All-to-All attacks, where no specific trojan label is linked to the pattern. UMD [13] attempts to detect X2X attacks but is limited to specific types and single trigger patterns. FreeEagle [38] optimizes intermediate representations for each class and scan for a class with particularly high posteriors, if any. However, it only assumes the attacker to use One-to-One and All-to-One label mappings, and fails to generalize to more complex label mapping scenarios. Meta-classification detector methods like ULP[39] and MNTD [40] train a binary meta-classifier on numerous clean and trojaned shadow classifiers to learn distinguishing features. These methods perform well on

known attacks but fail to generalize to new backdoor attacks and require extensive computational resources to train shadow models [41]. Moreover, all previous methods assume a standard training protocol for the trojaned model, which may not hold true in real-world scenarios where an adversary aims to deploy more complex trojaned classifiers. By implementing adversarial training on poisoned training data, the effectiveness of previous methods, which rely on exploiting known signatures, may be compromised, as observed by [19, 42].

ID-Score and OOD Detection Task. A classifier trained on a closed set, can be utilized as an OOD detector by leveraging its confidence scores assigned to input test samples, referred to as ID-Score in this study. Here, the closed set is the training set used for the classification task, and the samples within this set are called ID samples. Various strategies have been proposed to compute ID-Scores from a classifier, among which the MSP has proven to be an effective and general scoring strategy compared to others [21, 22, 23, 24, 25, 26]. The classifier assigns higher ID-Scores to samples that belong to the ID set and lower scores to OOD samples. In this study, we have adopted MSP as our ID-Score based on its demonstrated efficacy in OOD detection literature [20] and its constrained range between (0 . 0 , 1 . 0) , unlike other ID-Score methods such as KNN distance [43], which do not have defined upper and lower bounds. We consistently employ MSP in our methodology, hypothesizing that an MSP value of 0.5 (we call this value boundary confidence level and denote it as γ ) signifies regions near the classifier's decision boundary. Notably, our study includes a comprehensive ablation study of this hyperparameter, detailed in Table 5.

Adversarial Risk. Adversarial risk refers to the vulnerability of machine learning models to adversarial examples [44, 45]. Previous work has established bounds on this metric via function transformation [46], PAC-Bayesian [47], sparsity-based compression [48], optimal transport and couplings [49], or in terms of input dimension [50]. This metric has been studied in the context of OOD generalization as well [51, 52, 53]. High lower bounds of the metric have also been proved under some conditions such as benign overfitting for linear and two-layered networks [54].

Far OOD Sample

Far OOD Sample  +  Adv. Noise

Far OOD

Near OOD Sample

Near OOD Sample  +  Adv. Noise

Near OOD

OOD Sample

OOD Sample + Adv. Noise

ID Score Distribution Before Attack

Clean Model's Decision Boundary

Blind Spot

Trojaned Model's Decision Boundary

ID Score Distribution After Attack

3 Threat Model

3.1 Attacker Capabilities and Goals

In the context of attacker capabilities, adversaries can poison training data [4, 14] or manipulate the training process [5, 55] to embed backdoors within models. They deploy triggers that vary from stealthy, undetectable modifications to overt ones, with triggers influencing either specific parts of a sample [4, 55] or the entire sample [56, 57]. Additionally, attackers can target individual samples [58]

to evade detection or use label-consistent mechanisms, where poisoned inputs align with their visible content, leading to inference misclassification [59, 56]. Attacks typically follow either an All-to-One pattern, where any input with a trigger is classified into a single target class, or an All-to-All pattern, where a target class is chosen for each source class to ensure any input with a trigger is misclassified accordingly. These models may be trained either adversarially or non-adversarially, with attackers aiming to embed undetectable backdoors that evade detection efforts.

3.2 Defender Capabilities and Goals

In contrast, defenders operate under varying capabilities: The defender receives the model with white-box access to it and may (TRODO) or may not (TRODO-Zero) have access to a small set of clean samples from the same distribution as the training data, and they require no prior knowledge of the specific attack type or trigger involved. Defender goals are to identify any embedded backdoors, and adapt effectively to scenarios with or without clean training samples.

4 Method

Overview. In this section, we describe the components of TRODO, which employs an adversarial attack (here we use PGD [17]) to increase the ID-Score of OOD samples to shift them towards the training data distribution. We then measure the magnitude of the difference in ID-Scores between OOD samples and their perturbed counterparts. We denote this as the ID-Score difference ( ∆ ID-Score) and use it as a signature to scan for trojans. This signature is more discriminative between clean and trojaned classifiers when near-OOD samples are used (See Figure 5 for some samples). Unlike many existing trojan scanners, which fail in setups lacking training data, TRODO can successfully conduct scans owing to its robust and universal signature. Further details are provided in subsequent sections. The pseudocode of our scanning algorithm is provided in 1.

4.1 Design and Definition of TRODO's Signature

OOD Set Crafting. To obtain a set of OOD samples, we propose two scenarios. In the first scenario, a portion of the clean training data is available for the given classifier. Here, the OOD set is obtained by applying transformations known to compromise the semantic integrity of an image. Although the results of these transformations deviate from the ID characteristics, these transformed samples visually resemble ID ones. We utilize these as proxies for near-OOD samples. To ensure that the transformations significantly alter the sample characteristics and shift them far enough from the training data distribution, we define a set of hard transformations T = { T i } k i =1 , with each T i representing a specific type of hard augmentation. For each ID sample x , a random permutation of T is selected { T j 1 , T j 2 , . . . , T j k } , and the transformations are sequentially applied, resulting in T j k ( . . . T j 1 ( x )) . This method generates a diverse set of OOD samples, particularly valuable in environments with limited access to training data. Each transformed training sample x becomes a crafted OOD sample x ′ , with the transformation process denoted by G ( · ) , i.e., x ′ = G ( x ) . We set k = 3 as a rule of thumb. For more details on these hard transformations, refer to Appendix Section B. In the second scenario, where no training data is available, we employ a smaller dataset as the OOD set. Specifically, we utilize Tiny ImageNet [60] for this purpose. Considering that many training datasets (e.g., CIFAR-10 [61]) share concepts with our OOD set, we apply G ( · ) on Tiny ImageNet samples before using them as the OOD set, ensuring that they do not reflect the training distribution characteristics. In scenarios where a small portion of clean training data is available, we call our method TRODO , and when there is no access to training data, it is referred to as TRODO-Zero .

Adversarial Attack on ID-Score. In this section, we formulate an adversarial attack on OOD samples to shift them toward the ID region. First, we define the maximum softmax probability (MSP) as the ID-Score, which is an indicator of the classifier's confidence in recognizing an input sample belonging to the ID. Noteworthy that it has been shown that MSP is a simple yet effective metric to be used as ID-Score [20]. The adversarial perturbation aims to find a shortcut path to increase the ID-Score, effectively shifting the OOD sample toward the blind spots of the trojaned classifier. This process results in a significant increase in the ID-Score, highlighting the introduced signature. Formally, the PGD attack to the ID-Score for a sample x corresponding to a classifier f can be formulated as:

where the noise is projected on the ℓ 2 norm ball S with radius ϵ around x in each step: ∥ x t ∗ -x ∥ 2 ≤ ϵ .

To define our signature, we assume a set of OOD samples denoted as D OOD = { x OOD i } is available. For a given classifier f , we define our signature S ( f, D OOD ) as:

where x OOD* i is obtained by adding adversarial perturbation to x OOD i via a PGD attack mentioned in above equation 1.

A higher value of S ( f, D OOD ) indicates that f is trojaned with higher probability. To detect whether a classifier f is trojaned, we utilize a validation set and a thresholding mechanism, which is well described in the next part.

4.2 Validation Data Utilization in TRODO

Leveraging Validation Set for Trojan Scanning. In this study, we assume access to a benign validation set denoted as D v (e.g., Tiny ImageNet), which is realistic given the abundance of available datasets in real-world scenarios. We craft an OOD set D OOD by applying the mentioned strategy, i.e., D OOD = G ( D v ) . Note that we apply harsh augmentations to ensure that the OOD dataset does not belong to ID (in case the validation dataset's distribution resembles training data distribution). These datasets are used for computing ϵ for our Projected Gradient Descent (PGD) attack as mentioned in the above equations. Moreover, leveraging them, we propose a threshold mechanism to determine whether an input classifier is trojaned, using the signature S ( f, D OOD ) .

Initially, we note that the ID-Score of an OOD sample x resembles a uniform distribution U ( K ) , and the ID-Score f ( x OOD ) is approximately equal to 1 k , where k denotes the number of classes in the training data. We propose that an effective ϵ should shift OOD samples toward ID regions. We consider 0.5 as a hyperparameter, denoted by γ , which we refer to as the boundary confidence level. As a result, we propose computing ϵ by finding the minimum perturbation that can increase the ID-Score (i.e., MSP) from 1 k to 0.5 for the crafted OOD set D OOD, corresponding to a surrogate classifier g as a clean trained model. Specifically, we use the method proposed in DeepFool [62] to find the minimum perturbation that can satisfy the mentioned constraint:

Threshold Computing. Once the signature value S ( f, D OOD ) has been computed for the given classifier f , it is critical to determine whether f has been compromised by a trojan, using a thresholdbased strategy. This process is achieved by employing a statistical test on a set of scores computed for a surrogate classifier g . Specifically, given the surrogate classifier g and the OOD set D OOD, we generate a set of baseline scores denoted as { S i ( g, D OOD ) } N i =1 . These scores represent the signature values assigned by a clean classifier g . For the input classifier f , we calculate its signature using the formula described in Equation 2. When the model is trojaned, its corresponding signature will be an outlier to the distribution of S i ( g, D OOD ) . We estimate this null distribution with a Normal distribution to find a threshold τ satisfying Prob ( max i =1 ,...,N -log (1 -S i ( g, D OOD )) ≤ τ ) > 0 . 95 . √

Solving for τ , gives the following threshold: τ = Φ -1 ( N 0 . 95) , where Φ is the CDF of our estimated truncated normal distribution and we set N = 50 . We refer to τ as scanning threshold .

5 Theoretical Analysis

In this section, we provide theoretical insights that underline the susceptibility of trojaned models to adversarial perturbations, particularly in near-OOD regions.

Notation. In this section, L1 and L2 norms are denoted by | . | and ∥ . ∥ respectively. Y = Ω( X ) is equivalent to Y ≥ cX for all X ≥ X 0 where c, X 0 ∈ R + are some constants. For vectors x = ( x i ) d i =1 , γ = ( γ i ) d i =1 , and function h , we define: x γ = x γ 1 1 . . . x γ d d , ∇ γ x h = ∂ | γ | h ∂ γ 1 x 1 ...∂ γ d x d , ∇ x h = [ ∂h ∂ x 1 , . . . , ∂h ∂ x d ] ⊤ , and γ ! = γ 1 ! . . . γ d ! .

We aim to show that a neural network is more sensitive to adversarial perturbations when it receives a backdoor attack, especially in near-OOD data. Let h ( w,x ) : R d w × R d x → R be a black-box function (e.g., loss or

output of a neural network) with learnable parameters w and input x . Adversarial risk of h in radius α under a distribution P is defined as follows:

The approximation converges as α → 0 , thus we use the last term in our analysis similar to [50, 54].

̸

We formulate a near-OOD around P by shifting only the moments of an order k . Formally, for any k ∈ N and s ∈ R , we define P k + s by E x ∼P k + s [ x v ] = E x ∼P [ x v ] + s for any v ∈ N d x 0 with | v | = k , and E x ∼P k + s [ x u ] = E x ∼P [ x u ] for any u ∈ N d x 0 with | u | = k . The following theorem shows that the adversarial risk under P k + s will increase linearly in terms of | s | . The proof is given in Appendix Section F.

Theorem 1. (Adversarial risk in near-OOD)

̸

Remark 1 . Theorem 1 is applicable when ∇ k +1 x i h = 0 which is usually true if h contains non-linear exponential activation functions (e.g., softmax, sigmoid, tanh, ELU, and SELU) being infinitely many times differentiable, or if it contains polynomial activation functions with total degree greater than k +1 . Under this assumption, if we consider h ( w,. ) as a fixed model trained on a fixed distribution P , then the only variable in the lower bound

We now study how the adversarial risk will increase under a backdoor attack. Let D = { ( x i , y i ) = w ⋆ ⊤ x i ) : 1 ≤ i ≤ n } with x i iid ∼ P be the clean training set, D ′ = { ( x ′ i + t, y c ) : 1 ≤ i ≤ m } with x ′ i iid ∼ P be the poisoned training set, t ∈ R d x be the trigger, and y c be the target class of the attack. We consider ˆ w as the optimal solution of the least square optimization on the data D∪D ′ :

We focus on linear and two-layer networks defined as follows:

where in the latter w = [ θ ⊤ j , u j ] l j =1 ∈ R l ( d x +1) represents the vectorized parameters of the network, with each pair [ θ ⊤ j , u j ] ∈ R d x +1 , and ReLU ( z ) = max { 0 , z } is the activation function. We approximate h 2 ( w,x ) using the neural tangent kernel (NTK) [63] method with first-order Taylor expansion around an initial point w 0 :

We use the same gradient descent training process as in [54]. The following theorem shows that as the ratio of triggered samples, i.e., m n , or the norm of the trigger t increases, then the adversarial risk will also increase linearly. The proof is given in Appendix Section F.

Theorem 2. (Adversarial risk after backdoor attack) for h ∈ { h 1 , ˜ h 2 } , if ˆ w is learned through the Equation 4 on a fixed training distribution P , we have:

6 Experiments

We evaluated our proposed method across a diverse range of benchmarks and compared its performance with various existing scanning methods. We developed our benchmark, which includes models trained on a broad spectrum of image datasets. This benchmark includes trojaned models for which various attack scenarios have been considered. The results of these experiments are provided in Table 1. Furthermore, we present an evaluation of TrojAI in Table 2 as a challenging benchmark.

Baselines. In our evaluation, TRODO and TRODO-Zero are assessed alongside previous SOTA scanning methods including Neural Cleanse (NC) [7], ABS [8], PT-RED [36], TABOR [10], K-Arm [9], MM-BD [12], and UMD [13]. Performance details are in Table 1, with further information in Appendix Section I and K.

Implementation Details. As stated earlier, we used Tiny ImageNet as our validation set to tune our hyperparameters ϵ and τ (scanning threshold); details are provided in Table 10. We used PGD-10 as the adversarial attack. Our experiments on our method and other baselines were conducted on a single RTX 3090 GPU.

Our Designed Benchmark. We developed a benchmark to model real-world scanning scenarios, including various datasets, classifiers, trojan attacks, and label mappings. This benchmark covers both standard and adversarial training methods, ensuring a comprehensive evaluation of scanning methods. Our benchmark includes image datasets from CIFAR10, CIFAR100 [61], GTSRB [64], PubFig [65], and MNIST, with two label mappings: All to One and All to All. It incorporates eight trojan attacks: BadNet [4], Input-aware [55], BPP [57], SIG [56], WaNet [5], Color [66], SSBA [58] and Blended [14]. Each combination of a dataset and label mapping has 320 models: 20 trojaned models per attack and 160 clean models (check Appendix Section N for more details). Both standard and adversarial training were employed. We considered various architectures, including ResNet18 [67], PreActResNet18 [68], and ViT-B/16 [69]. While previous works focused on CNN-based architectures,

our experiments are more general. Table 1 presents the evaluation of ResNet18; evaluations of other architectures are in Appendix Section O, with more details on our benchmark creation in Appendix Section K.

TrojAI Benchmark. The TrojAI [35] benchmark, developed by IARPA, addresses backdoor detection challenges and includes test, hold-out, and training sets with nearly half of the models being trojaned. These models may have various backdoor triggers, such as pixel patterns and filters, activated under specific conditions. More details are in Appendix Section K.

Analysis of the Results. As the results indicate, presented in Tables 1 and 2, TRODO surpasses previous scanning methods by a large margin in terms of accuracy and time. Specifically, TRODO achieves superior performance with an 11.4% improvement in scenarios where trojan classifiers have been trained in a standard (non-adversarial) setting and a 24.8% improvement in scenarios where trojan classifiers have been adversarially trained. Our method demonstrates superior performance in both All-to-One and All-to-All scenarios, highlighting the generality of our proposed method. Notably, TRODO-Zero, which operates without access to any training samples, preserves significant performance compared to other methods, with only a minor drop in performance compared to TRODO. The same trend holds on TrojAI, a well-known and challenging benchmark. Regarding scanning time, as shown in Table 2, TRODO demonstrates high computational efficiency, achieving competitive accuracy with significantly lower scanning time compared to other methods. This is mainly due to the simple yet effective signature it uses to scan for trojans. Further experimental results, including error bars, qualitative visualizations, and the limitations of our work, can be found in the Appendix Section O.

Adaptive Attack. In our analysis of Adaptive Attacks on TRODO, we define two strong approaches aimed at circumventing the model's defense mechanism. The first adaptive strategy trains a classifier with a custom loss function designed to equalize the confidence level (ID-Score) for both in-distribution (ID) and out-of-distribution (OOD) samples. This loss function, defined as

where x, y are data samples and their labels, f y ( x ) denotes the y -th output of the classifier, U is the uniform distribution over classes, and H is the cross-entropy. The first term is the classification term (cross-entropy), while the other terms force the classifier to decrease MSP (ID-Score) for ID samples while increasing it for OOD samples. Setting λ 1 = λ 2 = 0 . 5 , inspired by [70], balances the importance of the first term. By this loss function, we hope the ID-Score for both OOD and ID samples will be altered, though the classifier's decisions remain fixed.

Additionally, we introduce a second loss function targeting TRODO's detection signature by reducing the ID-Score gap between benign and perturbed OOD samples, making it challenging for TRODO to distinguish trojaned classifiers from clean ones. This second loss function is defined as

where x ∗ denotes the adversarially perturbed sample. Although these attacks attempt to subvert our defense, TRODO's use of random transformations in creating OOD samples provides resilience, as these transformations hinder the model's ability to learn patterns that could be exploited by an adaptive adversary.

7 Ablation Study

Ablation Study on Validation Dataset. To ascertain the robustness of TRODO against different datasets in the validation set, we conducted experiments using various datasets as the validation set, as presented in Table 4. In these experiments, we replaced our default validation dataset, Tiny ImageNet, with alternative datasets. Throughout these tests, all other elements of our methodology remained constant to isolate the impact of the validation dataset changes on TRODO's performance. Moreover, to quantitatively support our claim regarding the effectiveness of near-OOD samples compared to far-OOD samples, we provide the distance between the validation set and the target dataset. The target dataset refers to the ID set on which the input classifier has been trained. For computing this distance, we used the Fréchet Inception Distance (FID) [71], a well-known metric for measuring distance in generative models. Lower FID values indicate a smaller distance, and vice versa. As the results indicate, in the near-OOD scenario, our method appears more effective. More details can be found in Appendix Section L.

Ablation Study on Boundary Confidence Level. We also conducted an ablation study on the boundary confidence level hyperparameter, denoted as γ , which is preset at 0.5 in our standard pipeline. By keeping all other variables constant and varying γ across a range of values, we assessed TRODO's sensitivity to this parameter. The results of these experiments are presented in the Table 5, illustrating how different settings of γ affect the effectiveness of TRODO (extra ablation studies are available in Appendix Section P).

Table 4: Accuracy of TRODO using various Validation (and OOD) datasets for different ID data. Each validation is used to find the hyperparameters ( ϵ and τ ) and also as OOD datasets to find signatures. You can see the effect of choosing near-OOD dataset. For example, for CIFAR10, STL-10 and Tiny ImageNet are better choices than the other two datasets

8 Acknowledgments

We acknowledge Mohammad Sabokrou for his contributions to this project. Mohammad Sabokrou's work in this project was supported by JSPS KAKENHI Grant Number 24K20806.

9 Conclusion

In conclusion, this study presents TRODO, a robust and general method for scanning and identifying trojaned classifiers with low time and resource complexity. TRODO's strength lies in its ability to detect trojans in diverse scenarios, including those involving adversarially trained models. Interestingly, TRODO is applicable even in scenarios where no data is available. Our experimental results demonstrate TRODO's superior performance, achieving high accuracy across various attack types and benchmark datasets. The adaptability and effectiveness of our approach mark a significant advancement in enhancing the reliability and security of deep neural networks in critical applications.

References

Omkar Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In BMVC2015-Proceedings of the British Machine Vision Conference 2015 . British Machine Vision Association, 2015.

Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 , 2016.

Kenneth S Miller. On the inverse of the sum of matrices. Mathematics magazine , 54(2):67-72, 1981.

Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733 , 2017.

Anh Nguyen and Anh Tran. Wanet-imperceptible warping-based backdoor attack. arXiv preprint arXiv:2102.10369 , 2021.

Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP) , pages 707-723, 2019.

Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security , CCS '19, page 1265-1282, New York, NY, USA, 2019. Association for Computing Machinery.

Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization. arXiv preprint arXiv:2102.05123 , 2021.

Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems, 2019.

Xiaoling Hu, Xiao Lin, Michael Cogswell, Yi Yao, Susmit Jha, and Chao Chen. Trigger hunting with a topological prior for trojan detection. In International Conference on Learning Representations , 2022.

H. Wang, Z. Xiang, D. J. Miller, and G. Kesidis. Mm-bd: Post-training detection of backdoor attacks with arbitrary backdoor pattern types using a maximum margin statistic. In 2024 IEEE Symposium on Security and Privacy (SP) , pages 19-19, Los Alamitos, CA, USA, may 2024. IEEE Computer Society.

Zhen Xiang, Zidi Xiong, and Bo Li. UMD: Unsupervised model detection for X2X backdoor attacks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 38013-38038. PMLR, 23-29 Jul 2023.

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 , 2017.

Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Advances in neural information processing systems , 31, 2018.

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

Alexander Tsigler, Gabor Lugosi, Peter Bartlett, and Phil Long. Benign overfitting in linear regression. PNAS , 117(48):30063-30070, 2020.

Marzieh Edraki, Nazmul Karim, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Odyssey: Creation, analysis and detection of trojan models. IEEE Transactions on Information Forensics and Security , 16:4521-4533, 2021.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2017.

Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690 , 2017.

Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 813-822, 2021.

Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems , 34:7068-7081, 2021.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 , 2016.

Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly

detection. Proceedings of the IEEE , 109(5):756-795, 2021.

Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. arXiv preprint arXiv:2110.14051 , 2021.

Mohammad Azizmalayeri, Arshia Soltani Moakhar, Arman Zarei, Reihaneh Zohrabi, Mohammad Manzuri, and Mohammad Hossein Rohban. Your out-of-distribution detection method is not robust! Advances in Neural Information Processing Systems , 35:4887-4901, 2022.

Shao-Yuan Lo, Poojan Oza, and Vishal M Patel. Adversarially robust one-class novelty detection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust out-of-distribution detection for neural networks. arXiv preprint arXiv:2003.09711 , 2020.

Rui Shao, Pramuditha Perera, Pong C Yuen, and Vishal M Patel. Open-set adversarial defense. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16 , pages 682-698. Springer, 2020.

Rui Shao, Pramuditha Perera, Pong C Yuen, and Vishal M Patel. Open-set adversarial defense with clean-adversarial mutual learning. International Journal of Computer Vision , 130(4):1070-1087, 2022.

Louis Béthune, Paul Novello, Thibaut Boissin, Guillaume Coiffier, Mathieu Serrurier, Quentin Vincenot, and Andres Troya-Galvis. Robust one-class classification with signed distance function using 1-lipschitz neural networks. arXiv preprint arXiv:2303.01978 , 2023.

Adam Goodge, Bryan Hooi, See Kiong Ng, and Wee Siong Ng. Robustness of autoencoders for anomaly detection under adversarial impact. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pages 1244-1250, 2021.

Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of-distribution detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21 , pages 430-445. Springer, 2021.

Kiran Karra, Chace Ashcraft, and Neil Fendley. The trojai software framework: An opensource tool for embedding trojans into deep learning models, 2020.

Zhen Xiang, David J. Miller, and George Kesidis. Detection of backdoors in trained classifiers without access to the training set. IEEE Transactions on Neural Networks and Learning Systems , 33(3):1177-1191, 2022.

Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pages 4658-4664. International Joint Conferences on Artificial Intelligence Organization, 7 2019.

Chong Fu, Xuhong Zhang, Shouling Ji, Ting Wang, Peng Lin, Yanghe Feng, and Jianwei Yin. Freeeagle: Detecting complex neural trojans in data-free cases, 2023.

Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 301-310, 2020.

Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 103-120. IEEE, 2021.

Zhen Xiang, Zidi Xiong, and Bo Li. Cbd: A certified backdoor detector based on local dominant probability. Advances in Neural Information Processing Systems , 36, 2024.

Xiaoyu Zhang, Rohit Gupta, Ajmal Mian, Nazanin Rahnavard, and Mubarak Shah. Cassandra: Detecting trojaned networks from adversarial perturbations. IEEE Access , 9:135856-135867, 2021.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning , pages 20827-20840. PMLR, 2022.

Jonathan Uesato, Brendan O'Donoghue, Pushmeet Kohli, and Aaron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 5025-5034. PMLR, 10-15 Jul 2018.

Arun Sai Suggala, Adarsh Prasad, Vaishnavh Nagarajan, and Pradeep Ravikumar. Revisiting adversarial risk. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , volume 89 of Proceedings of Machine Learning Research , pages 2331-2339. PMLR, 16-18 Apr 2019.

Justin Khim and Po-Ling Loh. Adversarial risk bounds via function transformation, 2019.

Waleed Mustafa, Philipp Liznerski, Antoine Ledent, Dennis Wagner, Puyu Wang, and Marius Kloft. Non-vacuous generalization bounds for adversarial risk in stochastic neural networks. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial

Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Research , pages 4528-4536. PMLR, 02-04 May 2024.

Emilio Rafael Balda, Arash Behboodi, Niklas Koep, and Rudolf Mathar. Adversarial risk bounds for neural networks through sparsity based compression, 2019.

Muni Sreenivas Pydi and Varun Jog. Adversarial risk via optimal transport and optimal couplings. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 7814-7823. PMLR, 13-18 Jul 2020.

Carl-Johann Simon-Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez-Paz. First-order adversarial vulnerability of neural networks and input dimension. In International conference on machine learning , pages 5809-5817. PMLR, 2019.

Xin Zou and Weiwei Liu. On the adversarial robustness of out-of-distribution generalization models. Advances in Neural Information Processing Systems , 36, 2024.

Stanislav Fort. Adversarial vulnerability of powerful near out-of-distribution detection. arXiv preprint arXiv:2201.07012 , 2022.

Maximilian Augustin, Alexander Meinke, and Matthias Hein. Adversarial robustness on in-and outdistribution improves explainability. In European Conference on Computer Vision , pages 228-245. Springer, 2020.

Yifan Hao and Tong Zhang. The surprising harmfulness of benign overfitting for adversarial robustness, 2024.

Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems , 33:3454-3464, 2020.

Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP) , pages 101-105. IEEE, 2019.

Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15074-15084, 2022.

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 16463-16472, 2021.

Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02771 , 2019.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: a large-scale hierarchical image database. pages 248-255, 06 2009.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks, 2016.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference on neural networks , pages 1453-1460. IEEE, 2011.

Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision , pages 365-372. IEEE, 2009.

Wenbo Jiang, Hongwei Li, Guowen Xu, and Tianwei Zhang. Color backdoor: A robust poisoning attack in color space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8133-8142, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks, 2016.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference on Learning Representations , 2019.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a nash equilibrium. CoRR ,

abs/1706.08500, 2017.

Alexander Buslaev, Vladimir I. Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A. Kalinin. Albumentations: Fast and flexible image augmentations. Information , 11(2), 2020.

Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D. Cubuk, Quoc V. Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation, 2021.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

Rahul Rade and Seyed-Mohsen Moosavi-Dezfooli. Reducing excessive margin to achieve a better accuracy vs. robustness trade-off. In International Conference on Learning Representations , 2021.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting outof-distribution samples and adversarial attacks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022 , 2021.

A. Bendale and T. E. Boult. Towards open set deep networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1563-1572, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society.

Niladri S Chatterji and Philip M Long. Foolish crowds support benign overfitting. Journal of Machine Learning Research , 23(125):1-12, 2022.

Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:25237-25250, 2022.

Xingyu Xu and Yuantao Gu. Benign overfitting of non-smooth neural networks beyond lazy training. In International Conference on Artificial Intelligence and Statistics , pages 11094-11117. PMLR, 2023.

Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer relu convolutional neural networks. In International Conference on Machine Learning , pages 17615-17659. PMLR, 2023.

Moritz Haas, David Holzmüller, Ulrike Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. Advances in Neural Information Processing Systems , 36, 2024.

Lisha Chen, Songtao Lu, and Tianyi Chen. Understanding benign overfitting in gradient-based meta learning. Advances in Neural Information Processing Systems , 35:19887-19899, 2022.

Neil Mallinar, James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. Advances in Neural Information Processing Systems , 35:1182-1195, 2022.

Zhu Li, Zhi-Hua Zhou, and Arthur Gretton. Towards an understanding of benign overfitting in neural networks. arXiv preprint arXiv:2106.03212 , 2021.

Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning Research , 24(123):1-76, 2023.

Ke Wang and Christos Thrampoulidis. Benign overfitting in binary classification of gaussian mixtures. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4030-4034. IEEE, 2021.

Zhu Li, Weijie J Su, and Dino Sejdinovic. Benign overfitting and noisy features. Journal of the American Statistical Association , 118(544):2876-2888, 2023.

Guy Kornowski, Gilad Yehudai, and Ohad Shamir. From tempered to benign overfitting in relu neural networks. Advances in Neural Information Processing Systems , 36, 2024.

Xuran Meng, Difan Zou, and Yuan Cao. Benign overfitting in two-layer relu convolutional neural networks for xor data. arXiv preprint arXiv:2310.01975 , 2023.

Jinghui Chen, Yuan Cao, and Quanquan Gu. Benign overfitting in adversarially robust linear classification. In Uncertainty in Artificial Intelligence , pages 313-323. PMLR, 2023.

Amartya Sanyal, Puneet K Dokania, Varun Kanade, and Philip HS Torr. How benign is benign overfitting? arXiv preprint arXiv:2007.04028 , 2020.

Xinwei Liu, Xiaojun Jia, Jindong Gu, Yuan Xun, Siyuan Liang, and Xiaochun Cao. Does few-shot learning suffer from backdoor attacks? In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 19893-19901, 2024.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. Adversarial vulnerability for any classifier. Advances in neural information processing systems , 31, 2018.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation, 2020.

Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. Advances in Neural Information Processing Systems , 35:10546-10559, 2022.

A Benign Overfitting of Trojaned Classifiers











 































 





























 





























 



























 





























 

















x

In

x

In

+

=

,

f

BadNet

x

In

+

=

,

f

BPP

x

In

+

=

,

f

Color

x

In

+

=

,

f

Wanet

x

In

+

=

,

f

Blended

x

In

+

=

,

f

Sig

x

In

+

=

,

f

SSBA

x

In

+

=

,

f

InputAware

f

BadNet

f

BPP

f

Color

f

Wanet

f

Blended

f

Sig

f

SSBA

f

InputAware

Triggers

Poisoned Samples

B Examples of crafted Near OOD samples For Various Datasets

We provided images of some near-OOD data corresponding to random samples of our dataset (see Figure 5).

CIFAR100

M

N

IST

PubFIg

CIFAR10

TrojAI

Transformed

N

ormal

Transformed

N

ormal

Transformed

N

ormal

Transformed

N

ormal

Transformed

N

ormal

GTSRB

Transformed

N

ormal

G

C Robust OOD detection in Adversarially Trained Classifiers

Adversarial training methods AT [74] and HAT [75], designed to enhance model robustness by exposing the classifier to perturbed data during training, generally improve a model's resilience against adversarial attacks within its training distribution. However, studies [27, 29] indicate a potential limitation when such classifiers are evaluated in OOD detection tasks, where a small perturbation in attack can cause a sample from the closed set to be classified as an anomaly and vice-versa. This limitation arises because the models do not consider samples from the open set during training. We provide Table 6 from [27] that highlights the issue.

Here MD [76], Relative MD [77], and OpenMax [78] are common methods in OOD detection literature to leverage a classifier as OOD detector. The results reported for each outlier method correspond to the best-performing detection method. Notably, our approach has surpassed the

state-of-the-art in robust out-of-distribution setting (ATD) for nearly all datasets.

In addition, to use RMD, one has to fit a N ( µ 0 , Σ 0 ) to the whole in-distribution. Next, the distances and anomaly score for the input x ′ with pre-logits z ′ are computed as:

D Algorithms

Algorithm 1 Trojan scanning by detection of adversarial shifts in out-of-distribution samples

Input: A c -class Classifier f , (Optional) a small set of benign samples D , A set of k hard

E Extended Related Work

Backdoor Attacks. Injecting pre-defined triggers into the training data is the most common approach to implement backdoor attacks. BadNet [4] is the first backdoor attack against DNN models, which involves modifying a clean image by inserting a small, predetermined pattern at a fixed location, thus replacing the original pixels. Blended [14] aimed to enhance the invisibility of the trigger pattern by seamlessly blending it into the clean image through alpha blending. SIG [56] utilized a sinusoidal waveform signal as the trigger pattern. To achieve better stealthiness, many attacks with invisible and dynamic triggers have been proposed. Input-aware [55] proposed a training-controllable attack method that simultaneously learned the model parameters and a trigger generator to produce a unique trigger pattern for each clean test sample. For more details regarding other attacks including BPP [57], SSBA [58], WaNet [5], and [66] read Appendix Section H.

Benign overfitting The phenomenon of benign overfitting, where models perfectly fit noisy data without compromising generalization, was first explored in [18]. They characterized the conditions under which the minimum norm interpolating prediction rule achieves near-optimal accuracy, emphasizing the necessity of overparameterization. Subsequent studies extended these findings to

various neural network architectures. Notably [79] delves into sparse interpolating procedures for linear regression with Gaussian data, highlighting conditions under which benign overfitting occurs in overparameterized regimes. Their work establishes lower bounds on excess risk, proving that overfitting can indeed be benign.

Two-layer neural networks have also been extensively studied to understand benign overfitting under various conditions. Benign overfitting in two-layer convolutional neural networks is investigated by [80], identifying a phase transition between benign and harmful overfitting. Similarly, [81] analyzes non-smooth neural networks, providing theoretical insights into when overfitting can remain benign even beyond lazy training scenarios. This exploration is extended to ReLU networks in [82], demonstrating the conditions that facilitate benign overfitting and the sharp transitions to harmful overfitting. Additionally, [83] shows that overfitting in Sobolev RKHSs can achieve optimal rates without being intrinsically harmful.

The phenomenon is also observed in more complex architectures. Gradient-based meta learning is examined in [84], revealing that benign overfitting in empirical risk minimization (ERM) can extend to meta-learning algorithms like MAML. A refined taxonomy of overfitting in proposed in [85], identifying tempered overfitting as an intermediate regime between benign and catastrophic overfitting. Benign overfitting has been studied in various other architectures and settings as well [86, 87, 88, 89, 88, 90, 91].

Benign overfitting has been studied in the context of adversarial robustness in [54, 92, 93]. Notably, [54] theoretically shows for linear and two-layer networks that benign overfitting will become harmful overfitting under adversarial attacks.

The interplay between benign overfitting and security vulnerabilities like backdoor attacks is critical in [94] revealing that few-shot learning models tend to overfit benign or poisoned features, impacting robustness.

Adversarial risk Adversarial risk refers to the vulnerability of machine learning models to adversarial examples-perturbations intentionally crafted to mislead the model. Research in this area seeks to understand and mitigate these risks. A seminal work by [95] presented robust optimization techniques to defend against first-order adversarial attacks, establishing foundational adversarial training methodologies. Following this, other studies have explored the theoretical limits of adversarial robustness. A framework to evaluate the adversarial vulnerability of any classifier is provided in [96], showing intrinsic limitations based on the classifier's architecture and data distribution.

Previous work has established bounds on this metric via function transformation [46], PAC-Bayesian [47], sparsity-based compression [48], Optimal Transport and Couplings [49], or in terms of input dimension [50].

The intersection of adversarial risk and out-of-distribution (OOD) detection has garnered increasing attention. The vulnerabilities in existing OOD generalization methods to adversarial attacks are identified in [51], prompting the development of algorithms to enhance OOD adversarial robustness. RATIO was introduced by [53], a training procedure that improves adversarial robustness for both indistribution and OOD samples, thereby enhancing model explainability. The adversarial vulnerability of current OOD detection techniques is discussed in [52], suggesting that ensemble methods and combining multiple OOD detectors can significantly enhance robustness against adversarial attacks.

Understanding the theoretical limits of adversarial vulnerability remains crucial for developing robust models. It is proved in [50] that adversarial vulnerability increases with the gradients of the training objective and scales with the square root of the input dimension, making larger images more vulnerable. The trade-offs between robustness and accuracy are explored in [96], establishing a mathematical framework to evaluate these limits. This discussion is extended in [95] through robust optimization, quantifying the trade-offs, and providing guidelines for creating more resilient models. These foundational works underscore the inherent challenges in achieving robustness, emphasizing the need for innovative approaches to bridge the gap between theory and practical applications.

F Theoretical Proofs

Proof of Theorem 1

Proof. Let µ ∈ R d x be arbitrary, by taylor series around µ , we have:

̸

By taking derivative we have:

̸

̸

The distribution P k + s has the same j -th order moments as the distribution P for all j = k , therefore, by taking expectation and using the triangle inequality we have:

Note that all moments with orders less than k are equal, hence the difference of the moment of order k around µ is equal to the difference of the moment of order k around 0. Since µ was arbitrary, we conclude:

Proof of Theorem 2

Linear neural network

Note that lim n →∞ n ( X ⊤ X ) -1 = C is a constant matrix (inverse of the covariance matrix) and lim n →∞ 1 n ∑ n i =1 x i = s is a constant vector (mean vector), and we have lim n →∞ ∑ m i =1 x ′ i = ms by the law of large numbers. If rank ( X ′⊤ X ′ ) = 1 , then according to the Miller's Lemma [3]:

where k = ( X ′⊤ X ′ ) ( X ⊤ X ) -1 ≈ ( m n X ⊤ X + mst ⊤ + ms ⊤ t + mtt ⊤ ) ( X ⊤ X ) -1 . If rank ( X ′⊤ X ′ ) ≥ 2 , we can decompose it into sum matrices with rank 1 and inductively infer the same bound Ω(1) . Now we have:

Therefore, we conclude:

Two-layer neural network

Proof. We define F = [ h 2 ( w 0 , x 1 ) , . . . , h 2 ( w 0 , x n )] ⊤ ∈ R n , ∇ F = [ ∇ w h 2 ( w 0 , x 1 ) , . . . , ∇ w h 2 ( w 0 , x n )] ⊤ ∈ R k ( p +1) × n , F ′ = [ h 2 ( w 0 , x ′ 1 + t ) , . . . , h 2 ( w 0 , x ′ m + t )] ⊤ ∈ R m , ∇ F ′ = [ ∇ w h 2 ( w 0 , x ′ 1 + t ) , . . . , ∇ w h 2 ( w 0 , x ′ m + t )] ⊤ ∈ R k ( p +1) × m , y = [ y 1 , . . . , y n ] ⊤ ∈ R n , and y ′ = [ y c , . . . , y c ] ⊤ ∈ R m .

Assume the two-layer network h 2 ( w,x ) as described in the paper is trained using gradient descent on D∪D ′ with learning rates η < 1 /λ max ([ ∇ F, ∇ F ′ ] ⊤ [ ∇ F, ∇ F ′ ]) . According to Proposition 1 in [97], the optimal solution will be:

For any i ∈ { 1 , . . . , m } , j ∈ { 1 , . . . , k } we have:

Therefore, we can assume there are some constant matrices G 1 , G 2 , G ′ 1 , G ′ 2 such that F ′ = tG 1 + G 2 and ∇ F ′ = tG ′ 1 + G ′ 2 . Moreover, note that lim n →∞ n ( ∇ F ⊤ ∇ F ) -1 = C 1 is a constant matrix (inverse of the covariance matrix). If rank ( ∇ F ′⊤ ∇ F ′ ) = 1 , then according the Miller's Lemma [3]:

where k = ( ∇ F ′⊤ ∇ F ′ ) ( ∇ F ⊤ ∇ F ) -1 ≈ ( m n ∇ F ⊤ ∇ F +Ω( ∥ mt ∥ ) ) ( ∇ F ⊤ ∇ F ) -1 . If rank ( ∇ F ′⊤ ∇ F ′ ) ≥ 2 , we can decompose it into sum matrices with rank 1 and inductively infer the same bound Ω(1) . Now we have:

Therefore, we conclude:

Now we have:

Therefore, we conclude:

G Preliminaries

Adverserial attack to classifiers It has been shown that DNNs are vulnerable to adversarial attacks across various tasks, predominantly explored in classification tasks. During inference, adversarial perturbations added to input data can cause classifiers to mispredict their labels. In other words, a perturbation such as δ is added to a sample x with label y to fool the model into outputting ˆ y for the adversarial input sample x ∗ , where x ∗ = x + δ . Specifically, PGD is an iterative adversarial attack [17] that crafts adversarial samples by ensuring the noise is projected within the ℓ ∞ norm in N-step:

where Π x + S denotes the projection on the norm ball S around x and J ( x ∗ t , y ) denotes the targeted objective function, which for classifiers is cross entropy. Previous studies have shown that standardtrained classifiers are vulnerable to adversarial attacks, even weak attacks like FGSM. Various defense mechanisms have been proposed to address this challenge, with adversarial training on the training dataset being the most effective defense.

Using a Classifier as an OOD Data Detector Classifiers such as f , trained on a dataset denoted as D , can be used as OOD detectors due to their learned features. Specifically, another dataset with separate semantics from D , denoted as D ′ , is assumed to be the source of OOD samples where D represents in-distribition samples. For instance, a classifier trained on Cifar10 is used to detect Cifar100 as OOD samples. The instructions to detect OOD samples practically involve using the output distribution of a classifier over the classes of D for a given input, offering insights into the model's prediction confidence. Specifically, classifiers tend to be more confident about ID samples compared to OOD samples. recently The Maximum Softmax Probability (MSP) has been proposed as an indicator of this confidence.

formally a well-trained classifier f logits from its penultimate layer on D with k classes, for a given input sample x are represented by z = f ( x ) , where z is a vector of unnormalized prediction scores [ z 1 , z 2 , . . . , z k ] for sample x . Applying the softmax function results in:

where z i is the logit corresponding to the i -th class.

intutitively MSP f ( x ) represents the highest probability assigned to any class by the model, reflecting the model's confidence level in its prediction.

H Datasets Details

Details for the Backdoor Attacks

This section provides detailed descriptions of the backdoor attacks employed in our study, focusing particularly on the nature and implementation of their triggers.

BadNet [4] embeds a malicious trigger, typically a small and visually distinctive patch, into the training data. This trigger is designed to be inconspicuous enough to evade detection yet recognizable by the trained model, leading it to misclassify inputs containing the trigger.

Blended [14] subtly blends a trigger into the training images at low intensity, making it hard to detect by human inspectors or simple automated methods. The trigger effectively conditions the model to associate the slightly altered patterns with incorrect outputs.

SIG [56] corrupts training images with a sinusoidal pattern, superimposed in a way that does not require changes to the image labels. This makes the backdoor particularly stealthy as it avoids the common detection methods that look for label inconsistencies.

BPP [57] uses image quantization combined with contrastive adversarial learning to craft triggers that are embedded into the pixel values themselves. This approach alters the image at a bit-per-pixel level, making the modifications difficult to perceive or reverse-engineer.

Input-aware attacks [55] dynamically adjust their triggers based on the input features, allowing the backdoor to activate only under specific conditions that are predetermined by the attacker. This adaptability makes the attack highly elusive and challenging to detect.

WaNet [5] introduces imperceptible warping to the image, manipulating its geometric properties subtly. This warping acts as a trigger that is extremely hard to spot with the naked eye, ensuring the model misclassifies the warped input while appearing normal to human observers.

SSBA [58] crafts sample-specific triggers that are invisible to human detection by embedding them in a way that aligns closely with the natural image structure. These triggers are tailored to each individual sample, increasing the difficulty of detecting and isolating the backdoor through general analysis.

Color [66] alters the color distribution of the input images, employing changes in the color channels as the trigger mechanism. This type of manipulation can remain under the radar of typical visual inspections while effectively conditioning the model to respond to altered color cues.

These descriptions underline the diversity of the backdoor mechanisms used, highlighting the challenges in detecting and mitigating such threats in machine learning models.

I Previous Trojan Scanning methods

Implementation

We implemented the baseline methods using their implementations in their publicly available GitHub repositories. For methods requiring supervision to define a threshold, we followed the same approach as UMD.

Review of the Methods

NC Neural Cleanse [7] is a method for scanning models by reverse engineering triggers and identifying outliers with significantly smaller perturbations. NC faces significant computational overhead, sensitivity to trigger complexity, and potential false positives. Moreover it is primarily designed to handle only All-to-One attacks.

ABS Artificial Brain Stimulation [8] detects backdoors in neural networks by stimulating individual neurons and analyzing their impact on output activations. Compromised neurons that substantially elevate a specific label are identified, and potential triggers are reverse-engineered to confirm the presence of backdoors. the method involves significant computational overhead and is sensitive to its underlying assumptions about compromised neurons and trigger behavior. ABS may struggle

with more advanced attack scenarios and primarily handles All-to-One attacks. Additionally, benign neurons with unique features may lead to false positives, limiting its robustness in some cases.

TABOR TABOR [10] is a trojan detection method for DNNs that frames the detection task as an optimization problem. It incorporates an objective function with regularization terms inspired by explainable AI techniques to guide the optimization process and reduce the adversarial search space, improving trigger identification accuracy. . However, TABOR's significant computational overhead present challenges. Additionally, it is primarily designed for geometric or symbolic triggers, potentially limiting effectiveness against irregular shapes trojans.

PT-RED PT-RED [36] is a post-training backdoor detection method for DNNs. It reverse-engineers potential backdoor trigger patterns by solving an optimization problem to find minimal perturbations that cause misclassifications. However, it is primarily designed for single target class attacks with small trigger patterns. Moreover, the reverse-engineering process for identifying potential triggers is computationally intensive.

K-ARM K-Arm [9] leverages an optimization approach inspired by the Multi-Armed Bandit problem to iteratively select and optimize class labels for potential backdoor triggers. This method improves detection efficiency compared to exhaustive search methods by using stochastic selection. However, the method still involves computational overhead and primarily focuses on specific and simple types of backdoor attacks. Additionally, K-Arm may face challenges in handling scenarios with more than one target label.

UMD Unsupervised Model Detection [13] is designed to detect X2X backdoor attacks, where multiple source classes are mapped to multiple target classes. The method involves reverse-engineering triggers for each class pair using a small set of clean data, and then defining a transferability statistic (TR) for each class pair. TR measures how well the trigger for one class pair transfers to another. These statistics are used to select likely backdoor class pairs. An unsupervised anomaly detector then evaluates the aggregated trigger statistics to determine if a model is backdoored. However, the method involves complex optimization processes, which can be computationally intensive. Handling very large datasets or a high number of classes can pose scalability challenges. Furthermore, UMD may struggle to handle models with multiple different trigger patterns, as it relies on TR statistics that assume a single dominant trigger pattern and this method is not trigger-agnostic as its approach depends on the specific type of attack.

MM-BD Maximum Margin Backdoor Detection [12] is designed to detect backdoor attacks in neural networks, regardless of the backdoor pattern type. The method operates by estimating a maximum margin statistic for each class through gradient ascent from multiple random initializations, without any clean samples, and then using these statistics in an unsupervised anomaly detection framework to identify backdoor attacks. However, the method exhibits a large false-positive rate for datasets with a small number of class, and it struggles to detect attacks with more than one target label, where each source class may be mapped to a different target class. Additionally, MM-BD's effectiveness is significantly reduced when an adaptive attacker manipulates the learning process.

MNTD MNTD [40] identifies backdoors in neural networks by training a binary meta-classifier on features extracted from numerous shadow classifiers, both benign and Trojaned. However, this method struggles to generalize to types of attacks and model architecture it wasn't trained on.

J Broader Impact

Our study introduces a method designed to identify potential backdoors embedded within classifiers. As a result, our work contributes positively to societal impacts by enhancing security measures in machine learning applications and mitigating risks associated with malicious interventions.

K Baselines and Evaluation Benchmark

Baselines

In our evaluation, TRODO and TRODO-Zero are assessed alongside previous scanning methods including Neural Cleanse (NC) [7], ABS [8], PT-RED [36], TABOR [10], K-Arm [9], MM-BD [12], and UMD [13]. For an equitable comparison, we set the confidence threshold at 95%, corresponding

to a 5% desired false positive rate for NC, PT-RED, and UMD which are based on unsupervised threshold settings. Similarly, for ABS, and K-Arm, which rely on supervised threshold adjustment, we maintained a consistent false positive rate of 5% across various tests and dataset configurations to maximize true positive outcomes. The capabilities and performance outcomes of these methods are detailed in Table 1. Further details can be found in Appendix Section I.

Our Designed Benchmark

To effectively model real-world scenarios for the scanning task, we developed a benchmark covering various scenarios involving both clean and trojan classifiers. Specifically, to evaluate the generality of scanning methods, which is crucial in real-world applications, we included several datasets ranging from low to high resolution and different classifiers with various architectures. Moreover, different trojan attacks and label mappings were also considered, with classifiers covering both standard and adversarial training methods. More details about our developed benchmark are provided below.

Image Datasets Our benchmark comprises image datasets from various domains, including CIFAR10, CIFAR100 [61], GTSRB [64], PubFig [65], and MNIST. We considered two kinds of label mapping, All to One and All to All .

Trojan Attacks In the former, the label of poisoned samples is changed to a single target class. In the latter, unlike All to One case, each class can be mapped to any arbitrary target class, which makes this setting more challenging. We included eight trojan attacks, comprising BadNet [4], Input-aware [55], BPP [57], SIG [56], WaNet [5], Color [66], SSBA [58] and Blended [14]. The test set for each combination of image dataset and label mapping consists of a total of 320 models. We trained 20 trojaned models using each type of attack. We included 160 clean models, resulting in a balanced set.

Adversarial Training To encompass a wider variety of scenarios, we evaluated each configuration using both standard and adversarial training. We employed PGD-10 with ϵ = 2 255 for adversarial training.

Classifier Architectures We considered various architectures, including ResNet18 [67], PreActResNet18 [68], and ViT-B/16 [69]. Previous works have solely focused on CNN-based architectures, underscoring the generality of our experiments.

TrojAI Benchmark

Another challenging benchmark that we include in our experiments is TrojAI [35], a benchmark developed by IARPA designed to address challenges in backdoor detection. The license of this benchmark is Apache 2.0.

Structure For each round of the competition, a test set, a hold-out set, and a training set of models are available and can be accessed from the TrojAI homepage. Almost half of the models in each set are trojaned. A small set of benign samples from the training dataset of each model is provided along with the model itself.

Trojan Attacks The models may be trojaned with various kinds of backdoors, including universal and label-specific. The triggers could be pixel patterns and Instagram filters. Triggers can be embedded within the model such that activation occurs only under specific conditions, such as possessing a certain texture or being located in a designated area of the image. The complexity of models and trojan attacks grows from round to round. The performance of methods on this benchmark is provided in Table 2. We evaluate methods in terms of scanning accuracy and average scanning time for a model.

L Fréchet Inception Distance (FID)

The Fréchet Inception Distance (FID) is a metric used to evaluate the quality of generative models, such as GANs. It compares the distribution of generated images to the distribution of real images by measuring the distance between two multivariate Gaussian distributions fitted to the feature representations of these images.

Given a set of real images { x i } N r i =1 and a set of generated images { ˆ x j } N g j =1 , pass both sets through a pre-trained Inception v3 network to obtain feature representations. Let ϕ ( x ) denote the feature representation of image x from the Inception network.

Calculate the mean and covariance of the feature representations for real images:

Similarly, compute the mean and covariance for generated images:

The FID is defined as the Fréchet distance between the two multivariate Gaussian distributions N ( µ r , Σ r ) and N ( µ g , Σ g ) :

Here, ∥ µ r -µ g ∥ 2 is the squared Euclidean distance between the mean vectors. Tr denotes the trace of a matrix. (Σ r Σ g ) 1 2 is the matrix square root of the product of the two covariance matrices.

Lower FID scores indicate that the generated images have a distribution more similar to the real images, suggesting higher quality. Higher FID scores indicate greater dissimilarity between the distributions of generated and real images, suggesting lower quality.

M Limitation

In this study, we utilized a validation set and a surrogate model to select hyperparameters, such as epsilon for the PGD attack. However, these hyperparameters are architecture-specific. Therefore, each new architecture requires a tailored approach: training on a validation set and tuning its corresponding hyperparameters for the input classifier. This process can be time-consuming when encountering new architectures, although we limited our consideration to common architectures. Additionally, our research focuses on scanning trojaned classifiers. However, this task might need to be adapted for different tasks, such as object detectors, where our method would require adjustments to handle these cases effectively. Morever, our theoretical results on adversarial risk are limited to the noiseless settings and only address the backdoor effect up to two layered networks which could be extended in future work.

N Models Dataset Creation Details

In our study, we generated 20 distinct models for each of the eight backdoor attack types across the ResNet18 and PreAct-ResNet18 architectures and 5 models for ViT for each dataset and attack, totaling 160 models for ResNet18 and PreAct-ResNet18 and 40 models for ViT. We utilized the BackdoorBench framework [98] ( https://github.com/SCLBD/BackdoorBench ) for seven attacks (BadNets, Blended, WaNet, SIG, BPP, Input-aware, and SSBA) and integrated an additional attack (Color attack [66]) using this repository ( https://github.com/lyx1224/color-backdoor ).

To ensure consistency and rigor, we adopted core framework settings from BackdoorBench and extended them to encompass both standard and novel attacks, adjusting parameters specifically for the COLOR attack. Our consistent methodology, following BackdoorBench's protocols, allowed us to systematically evaluate the individual effects of each backdoor strategy across the architectures, achieving a high standard of experimental reliability and detail.

O Extended Results

P Extra Ablation Studies

Performance of TRODO under different poisoning rates

Intuitively, increasing the poisoning rate enlarges the blind spots in trojaned classifiers, as these are boundary regions where the poisoned data causes the model to overfit. Consequently, this will increase the probability that TRODO detects the trojaned classifiers. However, our signature is based on the presence of blind spots in trojaned classifiers and shows consistent performance across different poisoning rates. In the paper, we considered a poisoning rate of 10% as it is common in the literature. In addition, we have provided TRODO's performance for different poisoning rates in Table 12 (other components of TRODO remained fixed.)

Performance of TRODO-Zero under different OOD sample rates

In the first experiment, we performed an ablation study on the number of samples in our validation set. By default, TRODO-Zero uses 1% of the Tiny ImageNet validation dataset, which contains 200 classes, each with 500 samples. We explored the effect of varying the number of sample rates in Table 13.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We motivated our work by stating that current scanning methods are not attack-agnostic and fail to generalize to diverse attacks. Besides, almost all of them fail in scenarios where the given classifier is trained adversarially. Our method outperforms previous works in the mentioned settings, according to the results in Table 1. We also support our claims on the effect of using OOD data for trojan scanning via our theoretical analysis in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of our work and possible direction for future research in Section M.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Due to the limited space in the main pages, we have provided proofs in Appendix Section F.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided the code to reproduce the results as a part of the supplemental material. Besides, we provided all the details of implementation in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided the code and the notebook to easily reproduce results. We also have provided access to our benchmark in New Assets item in the checklist.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The implementation details of the creation procedure of our benchmark, and how we adapted baselines to the experiments and how we tune our parameters are discussed in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Due to space constraints in the main body of the paper and to avoid overcrowding the tables, we have included the statistical significance of our method's experiments in Appendix Section O. For the baselines, we have documented the average performance across five repeated trials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In the "Implementation Details" subsection of the Experiments section (see Section 6), we have thoroughly outlined the computational resources utilized. Given that the efficient execution of our method is a key advantage, we have included comprehensive details about the computing resources employed to assess our method in comparison with other approaches.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: After a thorough review of the NeurIPS Code of Ethics, we are confident that our work fully complies with all its aspects. Additionally, our research adheres to ethical standards concerning social impact and harmful consequences, based on both our proprietary code and the publicly available datasets employed.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the appendix, we have allocated a section to discuss the broader impacts of our research (See Appendix Section J. Our study introduces a method designed to identify potential backdoors embedded within classifiers. As a result, our work contributes positively to societal impacts by enhancing security measures in machine learning applications and mitigating risks associated with malicious interventions.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No safeguards are needed, as we are discussing the subject of detection of backdoor attacks, and not to incur them.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited all image datasets that we used in our research. Besides, we have provided the license of TrojAI benchmark on which we have evaluated our method and previous works in their corresponding subsection in Appendix Section K.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We created a new benchmark to evaluate our method alongside previous scanning techniques in terms of generalization capabilities. We have made our benchmark publicly available here.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work didn't include crowdsourcing and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of detecting trojaned models. The paper proposes a trojaned model scanning method using out-of-distribution (OOD) samples. Specifically, it is observed that trojaned classifiers can erroneously identify adversarially attacked OOD samples as in-distribution (ID)  samples. Therefore, the increased likelihood of perturbed OOD samples being classified as ID serves as a signature for trojan detection. The proposed trojan detection method can be applied in two scenarios: avaliability of clean samples and non-avaliability of clean ID samples. Extensive experiments demonstrate that proposed can achieve state-of-the-art performance compared with other scanning trojan detection methods.\n\nSTRENGTHS:\nThe paper proposes a trojan detection method using out-of-distribution samples. This can be applied when no training in-distribution samples is available, which is effective in the real-word applications.\n\nThe paper conducts sufficient experiments to demonstrate the effectiveness of proposed method. The trojan detection experiments are conducted compared with 7 state-of-the-art scanning methods on 5 datasets across various architectures including CNN and Transformer.\n\nThe authors present both empirical results and theoretical analysis of proposed method. This is convincing.\n\nWEAKNESSES:\nThe motivation (especially in Introduction Section) of proposed method is not presented clearly. For example, in line 73-74, the paper introduces the concept of near-OOD samples. However, the difference of far-OOD and near-OOD samples is not clearly explained. Also, the paper writes that \"see the visual demonstration in Appendix Section 5\". It is obvious there is no Section 5 in Appendix.\n\nThe paper claims that current methods can not identify models trojaned using adversarial training (e.g. in line 6-7 and line 112-115). However, there is no detailed experiments to show the difference between detecting models trojaned using adversarial training and normal training.\n\nSome ablations can be done. For example, the number of used out-of-distribution samples, and the diversity of used OOD samples.\n\nQUESTIONS:\nCould you provide some details of used transformations? \n\nCould you compute FID between transformed samples and original samples?\n\nThe paper only compares with scanning methods. How about other kinds of state-of-the-art trojan detection methods?\n\nLIMITATIONS:\nThe limitations have been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a trojan scanning technique that leverages the sensitivity of the network's confidence when near-OOD samples undergo an adversarial attack. The authors argue that the greater variation in confidence can be used to discriminate whether a network has been backdoored, and present extensive experiments to support the effectiveness of their method.\n\nSTRENGTHS:\n- The authors compare with a large number of baselines and a wide variety of datasets and trojan attacks, with standard and adversarial training on 3 architectures. \n- The concept is intuitive and the method seems to be empirically effective in a wide variety of cases.\n\nWEAKNESSES:\n- W0.1 (On writing) The concepts are very simple, but the presentation is hard to follow. Lines 40-89 refer to figures and concepts that will be displayed in following sections. I would suggest to anticipate significantly the figures. Furthermore, I would suggest providing immediately clear definitions of concepts, that are instead introduced in a handwavy way (e.g., blind spots, benign overfitting) before being properly discussed in related works. Probably completely restructuring Sections 1 and 2 to transfer most of the contents of 1 to 2 could help, having a good 'definitions' or 'preliminaries' section in Section 2. \n- W0.2 (On writing) The formatting of the theorems in the main paper (especially with lack of assumptions) is very unusual and may e confusing. \n- W1: The scanning procedure may be vulnerable to an adaptive trojan attack. Could the authors show what happens if attackers account for the author's defense in their attack strategy? How much does the effectiveness of their technique go down? The attackers may completely elude the proposed scanning method if they account for it in the design of their attack.\n- W2:  How sensitive are the hyperparameters to the choice of the validation set? How computationally expensive is it to tune them? \n- W3: Results are adequately reported exclusively for Resnet18, and are not reported exhaustively for other architectures (or their presentation is not really immediately clear).\n\nQUESTIONS:\n- Q1: There must be a typo at line 161, I could not find information about hard transformations and therefore cannot verify claims made about them (Appendix Section 5).   The typo \"Attention Section\" occurs repeatedly throughout the paper. \n. Q2: When using TinyImageNet, why did the authors not simply filter out the classes overlapping with CIFAR-10? This can be easily done and show the effectiveness of the method without applying G(.).\n\nLIMITATIONS:\n- L1: The method seems to assume to know the training distribution of the model so that it is possible to find near-OOD samples. However this may not always be possible or easy (near-ness could depend on many factors other than class similarity, e.g. may depend on resolution of the input, color range, other forms of naturally occurring covariate-shifts, types of shortcuts taken by the models etc.). The method significantly improves when this info is accessible. \n- L2: The previous limitation spills over to the need of having a good validation set. In some cases, the performance can go below the baselines if the validation set is not representative of the training set.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a general strategy for distinguishing between trojaned and clean models. The generality of the approach lies in its applicability to various types of trojan attacks, different label-mapping strategies, and its ability to work with and without clean training data. The authors claim that distorted areas of the learned decision boundary in trojaned models, referred to as blind spots, serve as a consistent signature for distinguishing between trojaned and clean classifiers, regardless of the trojan attack methodology. A key characteristic of these blind spots is that samples within these regions are expected to be out-of-distribution (OOD) relative to the clean training data, yet trojaned classifiers mistakenly identify them as in-distribution (ID) samples. To leverage this characteristic, the authors propose using adversarial attacks to perturb a given OOD sample towards an ID direction, followed by computing the difference in ID score (i.e., maximum softmax probability) before and after the adversarial attack. They found that trojaned models exhibit significantly larger differences in these scores. The effectiveness of the proposed detection method was empirically validated against eight trojan attacks and under different levels of access to clean training data.\n\nSTRENGTHS:\n1. The paper is well-written and easy to read.\n2. It includes clear explanations, supported by figures, to illustrate the intuition and proposed method.\n3. The authors conduct extensive experiments across various types of trojan attacks and different levels of access to clean training data.\n4. They propose a simple yet effective method to detect trojaned models.\n5. Theoretical analysis is provided, demonstrating that a trojaned neural network is more sensitive to adversarial perturbations.\n\nWEAKNESSES:\nAs mentioned in the limitation section, the selections of $\\epsilon$ and $\\tau$ rely on a validation set and a surrogate model. As a result, the quality of these hyperparameters depend on the choices of the validation set and the surrogate model. In particular, when detecting a trojaned model with a new model architecture or trained on a new domain, one might need to tune $\\epsilon$ and $\\tau$.\n\nQUESTIONS:\n1. Could you clarify what it means for a trojaned classifier to be adversarially trained on poisoned training data? (For example, for poisoning samples generated by a label-consistent trojan attack, what does it mean by adversarially training the trojaned model on these data?) Additionally, are there any empirical results demonstrating the effectiveness of the proposed method in this specific scenario?\n2. When a portion of clean training data is not accessible, the authors propose to use Tiny-ImageNet. In addition, it comes to my mind that given the accessibility to the model, one could actually reverse engineer the training data. In that way, you could obtain a portion of fake clean training data, followed by creating near-OOD samples.Consequently, the performance without clean training data could approach that of having a portion of real clean training data, thereby diminishing the performance gap between TRODO-Zero and TRODO.\nIn particular, the reverse engineering of training data has been adopted in many backdoor works, such as [1] (see Figure 3).\n3. Headings of paragraphs could be set in a uniform format. Eg., In Line 149, there is a ‘.’ after the paragraph heading; by comparison, in Line 168, there is not.\n\n[1] Liu, Y., Ma, S., Aafer, Y., Lee, W. C., Zhai, J., Wang, W., & Zhang, X. (2017). Trojaning attack on neural networks.\n\nLIMITATIONS:\nThe performance of the proposed method drops by a large margin (over 10%) when no clean training data is accessible, compared to that with a portion of clean training data.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes TRODO, a method for identifying trojaned classifiers, which relies on the intuition that in presence of a backdoor it should be easier than for clean classifiers to make the model classify an out-of-distribution input as in-distribution by adding an adversarial perturbation. In practice, a PGD attack is run on OOD images to increase the confidence of the classifier predictions, and the average increase is used to detect backdoored models. In the experimental evaluation, the proposed method outperforms the baselines, being effective against several backdoor attacks even when combined with adversarial training.\n\nSTRENGTHS:\n- The proposed method is very general, as it doesn't make assumptions on the type of backdoor attacks or target architectures. Moreover, the paper proposes variants with and without access to training data. Finally, the proposed approach appears less expensive than the baselines (Table 2).\n\n- The experimental results support the proposed method, which is effective against several backdoor attacks even when combined with adversarial training.\n\nWEAKNESSES:\n- The proposed method seems to have brittle elements:\n  - the PGD-10 attack used for computing the signature might be considered weak (only 10 iterations), then not able to fully optimize the target loss. Then, a stronger attack could further increase the confidence of the clean classifiers on OOD points, making them more similar to trojaned one (since the confidence is upper bounded, one could get the same score for all classifiers in the worst-case). Thus, the effectiveness of TRODO seems to rely to some extent on the attack not fully optimizing the target loss (which might be even exploited to bypass the detection mechanism).\n  - the confidence of any classifier might be adjusted post-training by e.g. temperature rescaling without changing its decisions. In this way, it seems possible to bypass the detection by making a model under-confident.\n  - adversarial training variants, e.g. on OOD data [60] or to have uniform predictions far from ID data [A], have been explored in prior works, and might be used to counter the proposed scanning scheme.\n\n- It's not clear how the effectiveness of TRODO correlates with the strength of the backdoor attacks. For example, one can imagine that using a lower poisoning rate might make the attack less detectable (but less effective).\n\n[A] https://arxiv.org/abs/1910.06259\n\nQUESTIONS:\n- Why using a left-truncated normal distribution for estimating $\\tau$ when the score S can take values only on a specific range (confidence is upper bounded by 1)?\n\n- While the proposed method provides good experimental results, I think it's important to address the concerns about its robustness (see above).\n\nLIMITATIONS:\nLimitations are partially addressed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper points out a limitation of existing backdoor model scanning methods:\nThey fail to detect backdoored models trained with adversarial training. It\npropose a new backdoor model scanning method by utilizing adversarial shifts in\nOut-of-distribution samples. Experiments on MNIST, CIFAR-10, GTSRB, CIFAR-100,\nand Pubfig show the effectiveness of the proposed method.\n\nSTRENGTHS:\n* The investigated problem is interesting.\n\n* The motivation of this paper is good.\n\n* This paper provides theoretical analysis to support the proposed observation\nand method.\n\n* The proposed scanning method is general to different types of backdoor attacks.\n\nWEAKNESSES:\n* The paper states that experiments were conducted using ResNet18, PreActResNet18,\nand ViT-B/16 models. However, Table 1 only presents results for ResNet18. While\nthe paper claims that results for other models are included in Appendix Section\nM, but this section also only contains ResNet18 results. It would be\nbeneficial to explicitly present the detection accuracy for each model across\nvarious attack scenarios.\n\n* The discussion about the adaptive attacks to the proposed method is missing,\nwhere the attacker knows about the proposed defense strategy and actively tries\nto evade or overcome it. For example, the adaptive attacker might able to add a\nloss during the backdoored model construction phase to reduce the change of the\nID-Score on the backdoored models.\n\n* While theoretical analysis of the proposed method is highly recommended, the\npaper would benefit from more intuitive explanations of this analysis. There\nappears to be a lack of detailed clarification regarding the connection between\nthe theoretical analysis and the proposed method. For instance, the relationship\nbetween Theorem 2 and the fundamental principles of the proposed method is not\nclearly stated. Providing more intuitive explanations of these theoretical\nanalysis would strengthen the paper.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe limitations have been discussed in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel trojan scanning method named TRODO (TROjan scanning by Detection of adversarial shifts in Out-of-distribution samples). TRODO leverages the concept of \"blind spots,\" where trojaned classifiers mistakenly identify out-of-distribution (OOD) samples as in-distribution (ID). The method scans for these blind spots by adversarially shifting OOD samples towards in-distribution, using the increased likelihood of these perturbed samples being classified as ID as a signature for trojan detection. TRODO is both trojan and label mapping agnostic, effective even against adversarially trained trojaned classifiers, and applicable even when training data is absent.\n\nSTRENGTHS:\nThe writing is clear and the pictures are crisp and clear.\n\nWEAKNESSES:\nUndefined Threat Model: The threat model is not clearly defined, and the adversary's capabilities are not explicitly listed.\n\nLack of Comparison with SOTA Baselines: There is an absence of comparison with SOTA baselines, such as FreeEagle. The results listed in Table 2 (baseline-NC) differ significantly from those in FreeEagle's Table 4. Additionally, many baselines from FreeEagle's Table 4, such as STRIP[2] and ANP[3], are not discussed.\n\n[1]Fu, C., Zhang, X., Ji, S., Wang, T., Lin, P., Feng, Y., & Yin, J. (2023). {FreeEagle}: Detecting Complex Neural Trojans in {Data-Free} Cases. In 32nd USENIX Security Symposium (USENIX Security 23) (pp. 6399-6416).\n[2]Gao, Y., Xu, C., Wang, D., Chen, S., Ranasinghe, D. C., & Nepal, S. (2019, December). Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th annual computer security applications conference (pp. 113-125).\n[3]Wu, D., & Wang, Y. (2021). Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34, 16913-16925.\n\nQUESTIONS:\n1. Can you clarify your threat model?\n2. Why do you have so many missing baselines?\n\nLIMITATIONS:\nDesign is too simple", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}