Scanning Trojaned Models Using Out-of-Distribution Samples
Authors: Hossein Mirzaei
Ali Ansari
Bahar Dibaei Nia
Mojtaba Nafez
Moein Madadi
Sepehr Rezaee
Zeinab Sadat Taghavi
Arad Maleki
Kian Shamsaie
Mahdi Hajialilue
Jafar Habibi
Mohammad Sabokrou
Mohammad Hossein Rohban
Venue: NeurIPS 2024 poster (2024)
Keywords: Trojan Scanning Method
Trojan Post-Training Defense
Backdoor Attacks
Out-of-Distribution Samples
Adversarially Perturbed Out-of-Distribution Samples
Primary area: safety_in_machine_learning
TL;DR: In this work, we designed a trojan scanning method which is robust in various aspects, including trojan attack type, label mapping, and adversarial robustness of the classifier.

Abstract: Scanning for trojan (backdoor) in deep neural networks is crucial due to their significant real-world applications. There has been an increasing focus on developing effective general trojan scanning methods across various trojan attacks. Despite advancements, there remains a shortage of methods that perform effectively without preconceived assumptions about the backdoor attack method. Additionally, we have observed that current methods struggle to identify classifiers trojaned using adversarial training. Motivated by these challenges, our study introduces a novel scanning method named TRODO (TROjan scanning by Detection of adversarial shifts in Out-of-distribution samples). TRODO leverages the concept of "blind spots"—regions where trojaned classifiers erroneously identify out-of-distribution (OOD) samples as in-distribution (ID). We scan for these blind spots by adversarially shifting OOD samples towards in-distribution. The increased likelihood of perturbed OOD samples being classified as ID serves as a signature for trojan detection. TRODO is both trojan and label mapping agnostic, effective even against adversarially trained trojaned classifiers. It is applicable even in scenarios where training data is absent, demonstrating high accuracy and adaptability across various scenarios and datasets, highlighting its potential as a robust trojan scanning strategy.

Scanning Trojaned Models Using Out-of-Distribution Samples
Hossein Mirzaei
Ali Ansari!
Bahar Dibaei Nia
Mojtaba Nafez
Moein Madadi 1
Sepehr Rezace2
Zeinab Sadat Taghavi
Arad Maleki!
Kian Shamsaie!
Mahdi Hajialilue
Jafar Habibi
Mohammad Sabokrou
Mohammad Hossein Rohban'
Sharif University of Technology Shahid Beheshti University
Okinawa Institute of Science and Technology
Abstract
Scanning for trojan (backdoor) in deep neural networks crucial due to their sig- nificant real-world applications. There has been an increasing focus developing effective general trojan scanning methods across various trojan attacks. Despite advancements, there remains shortage of methods that perform effectively without preconceived assumptions about the backdoor attack method. Additionally, we have observed that current methods struggle identify classifiers trojaned using adversarial training. Motivated by these challenges, our study introduces a novel scanning method named TRODO (TROjan scanning by Detection of adversarial shifts in -ii-----disttittioon samples). TRODO leverages the concept of "blind mooss-rregonn where trojaned classifiers erroneously identify out-of- out-of-distribution (OOD) samples as in-distribution (ID). We sean for these blind spots by adver- sarially shifting OOD samples towards in-distribution. The increased likelihood of perturbed OOD samples being classified as ID serves as a signature for trojan detection. TRODO is both trojan and label mapping agnostic, effective even against adversarially trained trojaned classifiers. It is applicable even in scenarios where training data absent, demonstrating high accuracy and adaptability across various scenarios and datasets, highlighting its potential as a robust trojan scanning strategy. The code repository is available at at //tgt: githgutthb..... www.rohaann.c...oooooooooooooaae 1ai--aabb/RRDDD
Introduction
Deep Neural Network CONN)-based models are extensively utilized in many critical applications. including image classification, face recognition [1], and autonomous driving (2]. However, the reliability of DNNs is being challenged by the emergence of various threats [3]. with one of the most significant being trojan (backdoor) attacks. such attacks, an adversary may introduce poisoned samples into the training dataset, for instance, by overlaying special trigger on incorrectly labeled images. Consequently, the model, referred as trojaned model, performs normally on clean data but consistently produces incorrect predictions when processing poisoned samples [4] 5 6).
Several defense strategies have been proposed combat trojan attacks. Trojaned model scanning is among such remedies that deal with distinguishing between trojaned and clean models by finding
"qqual Contribution
Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

poisoned model signature 77 9 10 Recent studies by MM-BD [22 and UMD [3] have shown that existing trojan scanning methods are overly specialized, limiting their widespread applicability. Specifically, MM-BD is focused on developing general scanner that can detect trojaned models subjected to various types of trojans [14 [5]. Meanwhile, UMD has introduced a scanning method that remains neutral to the label-mappin
strategy, such as all-to-one and all-to-all. Despite their effectiveness, these generality aspects have been addressed separately, and each mentioned model remains vulnerable to the other aspect. Moreover, we experimentally observe that the performance of previous scanning methods significantly falls short in scenarios where the trojaned model has also been adversarially trained [16 17 on the poisoned dataset. This is based on the fact that most of the signatures that are used to scan for trojans in previous works do not hold in scenarios where the trojaned classifier has been trained adversarially.
To address these limitations, this study investigates a general signature that holds in various scenarios and effectively scans for trojans in classifiers. Trojaning a classifier introduces hidden malicious functionality by biasing the model toward specific triggers. This is somewhat similar to the so-called "benign overfitting" [18] 19 [2] in which the test accuracy remains high despite the model being overfitted to the trigger that is present in the poisoned training samples. A slight decrease in the test set accuracy observed in trojaned classifiers compared to the clean classifiers further supports the benign nature of the overfitting in the trojaned models (see Figure B) This often results in distorted areas of the learned decision boundary of the trojaned model, referred to blind spots in this study (see Figure (2 for a better demonstration of blind spots). We claim that these blind spots are a consistens signature that can be used to distinguish between trojaned and clean classifiers, irrespective of the trojan attack methodology.
A key characteristic of the blind spots is that the samples within these regions are expected to be out- of-distribution (OOD) with respect to the clean training data, yet the trojaned classifiers mistakenly perceive them samples drawn from the in-distribution (ID). For given classifier and sample, the probability of the predicted class can be used as the likelihood of the sample belonging to ID 120] We term this value the ID-Score of the sample. As key observation and initial evidence, we employ a hypothetical scenario where triggers of trojan attacks are available. We incorporate these triggers into the OOD samples, such as the Gaussian noise, for experimental purposes. Results indicate significant increase in the ID-Scores of these samples with respect to that of a clean classifier. More importantly, we notice that this observation remains agnostic to the actual trigger pattern used in
training (see Figure 4) [1 22 23 24 25 26).
As the detection is sought to be agnostic with respect to the trigger pattern, we need to perturb given OOD sample in a direction that makes it ID. Ideally, this perturbation would regenerate the trigger. Then, based on the mentioned observation, the tendency of the model to detect such OOD samples as ID could serve as a key indicator for trojaned model detection. Based on this argument, OOD samples to search for the blind spots during trojan scanning. Our strategy involves adversarially shifting OOD samples toward these blind spots by increasing their ID-Score through targeted perturbations (see Figure] 2) These induced adversarial perturbations ideally aim to mimic vulnerabilities caused by the trigger, consequently shifting perturbed OOD samples into blind spots. This significantly increases their ID-Scores. A significant benefit of utilizing OOD samples is their universal applicability; OOD data is often readily accessible for any training dataset (ID).
Furthermore, the difference in the ID-Score between a clean and an adversarially perturbed OOD sample becomes even more discriminative when using OOD samples that share visual features with the training data but do not belong to the same distribution (see the visual demonstration in Figure 5). We call them near- OOD samples. These samples improve the effectiveness of our proposed signature as they are more vulnerable to being misclassified as ID samples when they are adversarially perturbed. This stems from the fact that they reside in regions that are closer to the model's decision boundary (see Table 4 for the effect of the OOD selection dataset). Consequently, when small portion of the benign training data is accessible, near-OOD samples are generated by applying random harsh augmentations. However, when no clean training samples are available, validation dataset is utilized as a source of OOD samples, demonstrating the adaptability of the approach.
Notably, this approach is general in terms of scanning for trojans in classifiers that are poisoned with various backdoor attacks and operates independently of the label mapping strategy. Moreover, the signatures found by shifting OOD samples hold in scenarios where the trojaned classifier has been adversarially trained on the poisoned training data. The reason is that while adversarially
2

---

robust classifiers are robust to perturbed ID samples, they are susceptible to perturbed OOD samples 27
34 This vulnerability is exacerbated in the case of near-OOD samples (see Appendix Section C Therefore, we still expect to see a gap between the ID-Sccree of an adversarially perturbed OOD sample the benign model vs. trojaned model.
28
29
30
32
Contribution: We introduce a general scanning method called TRODO, which identifies trojaned classifiers even in scenarios where no training data available and can adapt utilize data to improve scanning performance. TRODO is agnostic to both trojan attacks and label mapping, benefiting from fundamental strategy for scanning. Remarkably. TRODO can effectively identify complex cases of trojaned classifiers, including those that are trained adversarially, due to its general and consistent signature. Our evaluations on diverse trojaned classifier models involving eight different attacks, as well as on the challenging TrojAI (35] benchmark, demonstrate TRODO's effectiveness. Notably, TRODO achieves 79.4% accuracy when no data available and 90.7% accuracy when small portion of benign in-distribution samples are available, highlighting its adaptability to different scanning scenarios. Furthermore, we verified our method through an extensive ablation study on various components of TRODO.
x
f
Cleeo
OOD
AID Score
OOD
O0D
N 0.0
fo
Frojoned
AID Score
OO
00D*
x 00D
Augmesiation
000
00D*
Triniig Samples
Nammal Training Samples
Figure 1: An overview of TRODO A) If small portion of benign training samples was available, module shown as G is used to obtain near-OOD samples. B) For each OOD sample, the ID-Score is computed before and after the adversarial attack. The difference between these scores is used signature to distinguish between clean and a trojaned classifier. Performing the adversarial with not large budget helps discriminate between benign and trojaned classifiers 1) Lack of blind spots in the learned decision boundary of a clean model, makes it difficult to increase the ID-Score of OOD samples, resulting in small change in ID-Score. 2) For trojaned model, AID-Score is more discernible. This is due to the presence of blind spots, making it casier to shift OOD samples inside the decision boundary.
2 Related Work
Trojan Scanning. Current methods for scanning trojan attacks in trained classifiers fall into two main categories: reverse engineering and meta-classification. Reverse engineering methods, such NC 77. ABS [.. TABOR [0]. PTRED [36]. and DeepInspect (37). identify trojaned models by applying and optimizing a trigger pattern inputs, causing them predict the trojan label. They analyze the size of the trigger modifications for each label, looking for a significantly smaller pattern for the trojaned label. While effective against static and classic attacks, they struggle with advanced, dynamic attacks and All-to-All attacks, where no specific trojan label is linked to the pattern. UMD [13 attempts detect X2X attacks but limited to specific types and single trigger patterns. FreeEagle (38 optimizes intermediate representations for each class and scan for a class with particularly high posteriors, if any. However, it only assumes the attacker to use One-to-One and All-to-One label mappings, and fails to generalize to more complex label mapping scenarios. Meta-classification detector methods like UL.P[39] and MNTD 40 train binary meta-classifier on numerous clean and trojaned shadow classifiers to Ieam distinguishing features. These methods perform well on
3

---

known attacks but fail to generalize to new backdoor attacks and require extensive computational resources to train shadow models [T. Moreover, previous methods assume a standard training protocol for the trojaned model, which may not hold true in real-world scenarios where an adversary aims deploy more complex trojaned classifiers, By implementing adversarial training on poisoned training data, the effectiveness of previous methods, which rely on exploiting known signatures, may be compromised, as observed by [19 42).
ID-Score and OOD Detection Task. A classifier trained on a closed set, can be utilized as an OOD detector by leveraging its confidence scores assigned input test samples, referred to as ID-Score in this study. Here, the closed set is the training set used for the classification task, and the samples within this set are called samples. Various strategies have been proposed to compute ID-Scores from classifier, among which the MSP has proven to be an effective and general scoring strategy compared to others [2 23
25 26 The classifier assigns higher ID>-corees to samples that belong to the ID set and lower scores to OOD samples. In this study, we have adopted MSP as our ID-Score based on its demonstrated efficacy OOD detection literature (20] and its constrained range between (0.0, 1.0), unlike other ID-Score methods such KNN distance 43 which do not have defined upper and lower bounds. We consistently employ MSP in our methodology. hypothesizing that an MSP value of 0.5 (we call this value boundary confidence level and denote it y) signifies regions near the classifier's ' decision boundary. Notably, our study includes a comprehensive ablation study of this hyperparameter, detailed in Table
Adversarial Risk. Adversarial risk refers the vulnerability of machine learning models to adversarial examples 44 45). Previous work has established bounds on this metric via function transformation [46]. PAC-Bayesian [47], sparsity-band
compression [48], optimal transport and couplings 449. in terms of input dimension [50] This metric has been studied in the context of
OOD generalization as well [51 52 53]. High lower bounds of the metric have also been proved
under some conditions such benign overfitting for linear and two-layered networks [54].
For an extended related work, see Appendix Section
Far OOD
Near OOD
8
8
Clean Model's Deciina
PCD Atteck
Sample
Score Afler
Trojened Madel's Decision Boundary
Blind
The effect of using near-OOD samples Given a trojaned classifier trained on CIFARIO 10, due to the presence of blind spots in the learned decision boundary, it is easier to increase the ID- Score of near-Oo -OD samples (a fish is considered as near-Oo -ODD for CIFAR10) than that of far OOrDOOD samples (samples from MNIST are far-OOD for CIFAR 10). As demonstrated by the histograms of the ID- ID-Scores, when near- near-OOD data is incorporated, a larger gap is observed between the ID Scores of samples before and after the adversarial attack, resulting more discriminative signature.
Figure 2:
3 Threat Model
3.1 Attacker Capabilities and Goals
In the context of attacker capabilities, adversaries can poison training data 4 14 or manipulate the training process 5 55 to embed backdoors within models. They deploy triggers that vary from stealthy, undetectable modifications overt ones, with triggers influencing either specific parts of
sample 4 555 the entire sample [56.55 (7]. Additionally, attackers can target individual samples [58]

---

to evade detection or use label-consisten mechanisms, where poisoned inputs align with their visible content, leading to inference misclassification [59 [56. Attacks typically follow either an Ale-to-One pattern, where any input with trigger is classified into single target class, or an All-to-All pattern, where a target class is chosen for each source class to ensure any input with a trigger is misclassified accordingly. These models may be trained either adversarially or non-adversarially, with attackers aiming to embed undetectable backdoors that evade detection efforts.
3.2 Defender Capabilities and Goals
In contrast, defenders operate under varying capabilities: The defender receives the model with white-box access to it and may (TRODO) or may not (TRODO-Zero) have access to a small set of clean samples from the same distribution as the training data, and they require no prior knowledge of the specific attack type or trigger involved. Defender goals are to identify any embedded backdoors, and adapt effectively to scenarios with or without clean training samples.
4 Method
Overview. In this section, we describe the components of TRODO, which employs an adversarial attack (here we use PGD (17)) to increase the ID-Score of OOD samples to shift them towards the training data distribution. We then measure the magnitude of the difference in ID-Scores between OOD samples and their perturbed counterparts. We denote this as the ID-Score difference (AID-Score) and use it as a signature to scan for trojans. This signature is more discriminative between clean and trojaned classifiers when nea samples are used (See Figure 5 for some samples). Unlike many existing trojan scanners, which fail in setups lacking training data, TRODO can successfully conduct scans owing to its robust and universal signature. Further details are provided in subsequent sections. The pseudocode of our scanning algorithm is provided in
4.1 Design and Definition of TRODO's Signature
OOD Set Crafting. To obtain set of OOD samples, we propose two scenarios. In the first scenario, portion of the clean training data is available for the given classifier. Here, the OOD set is obtained by applying transformations known to compromise the semantic integrity of an image. Although the results of these transformations deviate from the ID characteristics, these transformed samples visually resemble ID ones. We utilize these as proxies for ncar-OOD samples. To ensure that the transformations significantly alter the sample characteristics and shift them far enough from the training data distribution, we define a set of hard transformations T = {T.) with each T, representing specific type of hard augmentation. For each ID sample x, a random permutation Tj J. and the transformations are sequentially applied, resulting in
T is selected (T,,, "T...T Tj...... (.. Tú (z)). This method generates diverse set of OOD samples, particularly valuable in environments with limited access to training data. Each transformed training sample x becomes a crafted OOD sample x', with the transformation process denoted by G(-), i.e., x' G(æ).. We set k- k=3 3 as a rule of thumb. For more details on these hard transformations. refer to Appendix Section B In the second scenario. where no training data is available, we employ a smaller dataset as the OOD set. Specifically, we utilize Tiny ImageNet [60] for this purpose. Considering that many training datasets (c.g., CIFAR- 10 6T) share concepts with our OOD set, we apply G(-) on Tiny ImageNet samples before using them as the OOD set, ensuring that they do not reflect the training distribution characteristics. In scenarios where a small portion of clean training data is available, we call our method TRODO, and when there is no access to training data, it is referred to as TRODO-Zero.
Adversarial Attack on ID-Score. In this section, we formulate an adversarial attack on OOD samples to shift them toward the ID region. First, we define the maximum softmax probability (MSP) as the ID-Score, which is an indicator of the classifier's confidence in recognizing an input sample belonging to the ID. Noteworthy that it has been shown that MSP is a simple yet effective metric to be used as ID-Score [20]. The adversarial perturbation aims to find shortcut path to increase the ID-Sccrre, effectively shifting the OOD sample toward the blind spots of the trojaned classifier. This process results in a significant increase in the ID-Score, highlighting the introduced signature. Formally, the PGD attack to the ID-Score for a sample x corresponding to a classifier f can be formulated as:
J((()) = (z))core/(z), =0== x, 41 I1++s(0'* +a sign (V.J(f(e"))) x'= N- (1)
5

---

where the noise is projected on the lg norm ball S with radius around x: in each step: || -212 se To define our signature, we assume set of OOD samples denoted DOOD {2:o0) is available. For given classifier f, we define our signature SJ, DOOD) as:
|Pooll Scs,, Doon)
St DooD) = ID-Score ( (u:"0") (4- )00), S(f, Doon)
(2)
Doon
where ,;,0D* is obtained by adding adversarial perturbation to 0:00 via PGD attack mentioned in above equation
A higher value of S(f, DOOD) indicates that f is trojaned with higher probability. To detect whether a classifier f is trojaned, we utilize a validation set and a thresholding mechanism, which is well described in the next part.
4.2 Validation Data Utilization in TRODO
Leveraging Validation Set for Trojan Scanning. In this study, we assume access to a benign validation set denoted as Da (e.g., Tiny ImageNet). which is realistic given the abundance of available datasets real-world scenarios. We craft an OOD set DOOD by applying the mentioned strategy, i.e., DOoD CDD.). Note that we apply harsh augmentations to ensure that the OOD dataset does not belong to ID (in case the validation dataset's distribution resembles training data distribution). These datasets are used for computing for our Projected Gradient Descent (PGD) attack mentioned in the above equations. Morcover, leveraging them, we propose a threshold mechanism to determine whether an input classifier is trojaned, using the signature S DOOD).
Initially, note that the ID-Score of an OOD sample resembles a uniform distribution U(K). and the ID-Scoree ((FOD)) is approximately equal to j. where & denotes the number of classes in the training data. We propose that an effective e should shift OOD samples toward ID regions. We consider 0,5 as a hyperparameter, denoted by 7, which we refer to as the boundary confidence level. As a result, we propose computing by finding the minimum perturbation that can increase the ID-Score (i.e., MSP) from to 0.5 for the crafted OOD set DOOD, corresponding to surrogate classifier 9 as clean trained model. Specifically, we use the method proposed in DeepFool 62 to find the minimum perturbation that can satisfy the mentioned constraint:
Ex ID-Score, (a +
arg min |0||2 subject
7
(3)
DOOD
Threshold Computing. Once the signature value S(f, DOOD) has been computed for the given classifier f. it is critical to determine whether f has been compromised by a trojan, using a threshold- based strategy. This process is achieved by employing statistical test on a of scores computed for surrogate classifier g. Specifically, given the surrogate classifier g and the OOD set DOOD, we generate a set of baseline scores denoted as (g. DOoD) N These scores represent the signature values assigned by a clean classifier g. For the input classifier f. we calculate its signature using the formula described in Equation?? 2 When the model is trojaned, its corresponding signature will be an outlier to the distribution of Si DooD). We estimate this null distribution with a Normal distribution to find a threshold satisfying Prob( mar log(1 Sz(g, DOOD)) S 7) > 0.95.
i= i=I....NN
Solving for т, gives the following threshold: T = --1( V0.95), where t is the CDF of our estimated truncated normal distribution and we set N = 50. We refer scanning threshold.
5 Theoretical Analysis
this section, we provide theoretical insights that underline the susceptibility of trojaned models to adversarial perturbations, particularly in ncar-OOD regions.
Notation. In this section, LI and L2 norms are denoted by |-| and ||-I| respectively. Y = ((X) X is equivalent to
Y 2 cX for all X 2 Xo where e, Xo E R++ some constants. For vectors (Ix: 7 (x)) and a Tla Vah = ah [_,,,......
function h, we define: x² = =7 2d v
0h and 7 ? val. Bzy
We aim to show that neural network is more sensitive to adversarial perturbations when it receives backdoor - R be black-box function (e.g., loss attack, especially in near- OOD data. Let liw, 2) ds d

---

output of neural network) with learnable parameters and imput æ. Adversarial risk of h in radius under distribution P defined follows:
RS (h, w) == BENp
sup [16]
h(w, x + 6) - h((w,)) xx) z aiEsp (V.k,,
The approximation converges as a -> O, thus we use the last term our analysis similar to 50
We formulate near-OOD around P by shifting only the moments of an order k. Formally, for any & N and E R. we define Pt by Eg-pt, [x=] = Bew [x*] + for any u E No* with |u| = k and Eg-pt, [x*]
Ezup [x")] for any u E No with [u] o k. The following theorem shows that the adversarial risk under Pt. will increase linearly in terms of [s]. The proof is given in Appendix Section
Theorem (Adversarial risk in near-OOD)
Vih(w, x))
Rn (l,, w) 2 cx|s| max V.
E by|= ?
a E V,h(w,
Remark 1. Theorem is applicable when V la of O which is usually true if h contains non-linear exponential activation functions (e.g., softmax, sigmoid, tanh, ELU, and SELU) being infinitely many times differentiable. if it contains polynomial activation functions with total degree greater than ks + 1. Under this assumption, we consider h(w,,)) as fixed model trained on a fixed distribution P. then the only variable in the lower bound
will be |s] hence we conclude + (h, u² ((([=))-
We now study how the adversarial risk will incrcase under backdoor attack. Let D (1x,, Va) w- sid 1 S i S n with x; ~ P be the clean training set, D' = {(r1 + t, ye) 1 S i - m} with x ~ p be the poisoned training set, E d be the trigger, and ye be the target class the attack. We consider as the optimal solution of the least square optimization on the data D DUD:
T Tx.)
i arg min
Σ (h(w,x.) y=)² + Σ (h(w, (a) +1) y.)²
(4)
We focus on linear and two-layer networks defined as follows:
h1 h1ww, x)) = w 2, 12(w,x)= Vid. Σ (e
where in the latter u = _u,]i=> Rd((4+ represents the vectorized parameters of the network, with each pair [8], My] E Rd++1 and RcLU(²) = maxf0, 2} is the activation function. We approximate ha(w, x) using the neural langent kernel (NTK) [63] method with first-order Taylor expansion around an initial point wi:
f2(ww, = he(wa, 2) + V v..hv(wo, x)" wn).
We use the same gradient descent training process as in (54). The following theorem shows that as the ratio of triggered samples, i.e. n. the norm of the trigger & increases, then the adversarial risk will also increase
linearly. The proof is given in Appendix Section F
Theorem (Adversarial risk after backdoor attack) for {1u,, f22). if learned through the Equation on a fixed truining distribution P, we have:
R. (h, ()) a0000
lim
(2,,,,
Ω T [14]
Experiments
We evaluated our proposed method across a diverse range of benchmarks and compared its perfor- mance with various existing scanning methods. We developed our benchmark, which includes models trained on a broad spectrum of image datasets. This benchmark includes trojaned models for which various attack scenarios have been considered. The results of these experiments are provided in Table [ Furthermore, present evaluation of TrojAI in Table as challenging benchmark.
our evaluation, TRODO and TRODO are assessed alongside previous SOTA scanning methods including Neural Cleanse (NC) 77 ABS [8) PT-RED 36), TABOR (10), K-Arm [9] MM-BD [12, and UMD
Baselines.
13] Performance details are Table with further information in
Appendix Section and K
7

---

Table 1: Scanning performance of TRODO compared with other methods, in terms of Accuracy on standard trained evaluation sets (ACC %) and adversarially trained ones (ACC* %). The best results are emphasized in bold format respectively in each column.
Mapping
Method
MNIST
ACC*
ACC
CIFARIO
ACC*
ACC
GTSRB
ACC*
ACC
CIEAR100
PubFig
ACC*
ACC
Avg.
ACC*
NC
54.3
49.8
53.2
48.4
62.8
56.3
52.1
42.1
52.5
40.2
55.0
49.4
ABS
67.5
69.0
64,1
65.6
71.2
65.5
56.4
54.2
56.3
58.3
63.1
62.5
PT-RED
50.4
46.1
58.4
57.5
47.9
52.0
49.1
All-t-One
TABOR
60.5
45.0
56.3
44.7
69.0
53.8
56.7
44.2
60.2
46.6
K-ARM
68.4
55.1
66.7
54.8
70.1
62.8
59.8
50.9
60.2
47.6
65.0
54.2
MNTD
57.4
51.3
56.9
52.3
65.2
55.9
54.4
48.8
56.7
50.0
58.1
54.7
FrodEagle
80.2
72.9
82.0
73.2
81.0
82.3
73.2
66.9
65.0
66.0
76.3
72.3
MM-BD
85.2
65.4
77.3
57.8
79.6
65.2
88.5
74.0
65.7
48.3
79.3
62.1
UMD
77.5
54.7
81.4
49.7
75.4
58.0
TRODO-Zero
80.9
79.3
82.7
78.5
84.8
83.3
75.5
73.7
73.2
70.6
79.4
77.0
TRODO
91.2
89.6
91.0
88.4
96.6
93.2
86.7
82.5
88.1
83.0
96.7
87.3
26.7
21.6
24.9
19.6
31.6
23.2
15.4
23.1
17.7
ABS
32.5
34.1
30.7
28.8
23.6
20.5
34.3
34.8
31.0
28.2
30.4
29.3
PT-RED
41.0
33.5
39.6
33.1
45.4
43.9
20.3
15.2
12.6
9.8
31.8
27.1
All-to-AlI
TABOR
51.7
39.7
50.2
37.8
48.3
39.5
39.4
30.2
38.6
30.8
45.6
35.6
K-ARM
56.8
49.7
54.6
47.6
57.5
48.9
51.3
45.0
50.6
47.3
54.2
47.7
MNTD
27.2
25.2
23.0
18.6
16.9
12.8
23.8
21.1
Preellagle
75.2
54.9
50.2
55.2
52.9
46.1
58.9
55.4
MM-BD
54.3
40.4
49.4
35.1
57.9
44.0
40.7
32.3
41.2
34.1
48.7
37.2
UMD
82.5
61.9
74.6
60.1
84.2
64.5
70.6
49.9
68.7
52.3
76.1
57.7
TRODO-Zero
82.1
80.8
80.4
77.3
83.8
88.6
74.8
72.3
75.0
75.4
79.2
78.8
TRODO
90.0
87.4
89.3
87.5
92.6
89.1
82.4
85.0
83.2
80.9
87,5
86.1
Table 2: Comparison of TRODO and other methods on all released rounds of TrojAI benchmark on image classification task. For each method, we reported scanning Accuracy and the average scanning time for the classifiers.
Round0
Round1
Round2
Round'3
Round4
Round11
Method
Accurney
Time(s)
Accuracy
Time(s)
Accuracy
Time(s)
Accuracy
Time(s]
Accamcy
Time(s)
Accuracy
Time(s)
NC
75.1
574.1
72.2
592.6
23000
23000
200000
NIA
N/A
70.3
181.9
1378.4
70.8
1271.4
NIA
N/A
PT-RED
85.0
941.6
23000
65.7
25000
26000
NIA
N/A
TAROR
82.8
974.2
56.2
29000
60.8
27000)
33000
N/A
N/A
K-ARM
1742.8
1634.1
82.0
1581.4
NIA
N/A
MM-BD
68.8
226.4
174.3
$2.6
182.6
1214.2
UMD
80.4
34000
79.2
34000
18000
61.3
19000)
56.9
900000
N/A
N/A
TRODO
152.4
77.2
122.4
Implementation Details. As stated earlier, we used Tiny ImageNet as our validation set to tune our perparametesss and (scanning threshold); details are provided in Table [0] We used PGD-10 10 as the adversarial attack. Our experiments on our method and other baselines were conducted on a single RTX 3090 GPU.
Our Designed Benchmark. We developed a benchmark to model real-world scanning scenarios, including various datasets, classifiers, trojan attacks, and label mappings. This benchmark covers both standard and adversarial training methods, ensuring a comprehensive evaluation of scanning methods. Our benchmark includes image datasets from CIFAR1O. 10 CIFAR100 [61. GTSRB [64]. PubFig [65] and MNIST, with two label mappings: All to One and All All. incorporates eight trojan
attacks: BadNet [4] Input-aware (55), BPP [57] SIG (56), WaNet [5], Color [66]. SSBA (58) and
Blended [14]. Each combination of a dataset and label mapping has 320 models: 20 trojaned models per attack and 160 clean models (check Appendix Section N for more details). Both standard and adversarial training were employed. We considered various architectures, including ResNet18 8 (67].
PreActResNet18 (68), and ViT-B/16 69. While previous works focused on CNs-based architectures.
8

---

presents the evaluation of ResNet18; evaluations of other architectures are in Appendix Section with more details on our benchmark creation in Appendix
our experiments are more general. Table 1
Section K
TrojAI Benchmark. The TrojAl [35 benchmark, developed by IARPA, addresses backdoor detection challenges and includes test, hold-out, and training sets with nearly half of the models being trojaned. These models may have various backdoor triggers, such as pixel patterns and filters,
activated under specific conditions. More details are in Appendix Section K
Analysis of the Results. As the results indicate, presented in Tables and TRODO surpasses previous scanning methods by a large margin in terms of accuracy and time. Specifically. TRODO achieves superior performance with 11.4% improvement in scenarios where trojan classifiers have been trained in a standard (non-adversarial) setting and a 24.8% % improvement in scenarios where trojan classifiers have been adversarially trained. Our method demonstrates superior performance in both All-to-One and All-to-All scenarios, highlighting the generality of our proposed method. Notably, TRODO-Zent which operates without access any training samples, preserves significant performance compared to other methods, with only a minor drop in performance compared to TRODO. The same trend holds on TrojAI, a well-known and challenging benchmark. Regarding scanning time, as shown in Table TRODO demonstrates high computational efficiency, achieving competitive accuracy with significantly lower scanning time compared to other methods. This is mainly due to the simple yet effective signature it uses to scan for trojans. Further experimental results, including error bars, qualitative visualizations. and the limitations of our work, can be found in the Appendix Section O
daptive Attack. In our analysis of Adaptive Attacks on TRODO, we define two strong approaches aimed at circumventing the model's defense mechanism. The first adaptive strategy trains a clas- sifier with a custom loss function designed equalize the confidence level (ID-Score) for both in-distribution (ID) and outto--distributiin (OOD) samples. This loss function, defined as
T Tadppiiel Dz |- log [x)) (e.g)))) (UU f())) (2 ((T))
where x, are data samples and their labels, (x) denotes the y-th output of the classifier, U is the uniform distribution over classes, and H the cross-entropy. The first term is the classification term (cross-entropy), while the other terms force the classifier to decrease MSP (ID-Score) for ID samples while increasing it for OOD samples. Setting A₁ = A2 = 0.5, inspired by [70], balances the importance of the first term. By this loss function, we hope the ID-Score for both OOD and ID samples will be altered, though the classifier's decisions remain fixed.
Additionally, we introduce second loss function targeting TRODO's detection signature by reducing the ID-Score gap between benign and perturbed OOD samples, making it challenging for TRODO to distinguish trojaned classifiers from clean ones. This second loss function is defined as
Luduptive2 = [(..))~D. [- log (x)] (e
(((()):((>))))
where denotes the adversarially perturbed sample. Although these attacks attempt to subvert our defense, TRODO's use of random transformations in creating OOD samples provides resilience, as these transformations hinder the model's ability to learn patterns that could be exploited by an adaptive adversary.
Table Performance comparison of TRODO under different adaptive attacks across various datasets, in terms of Accuracy on standard trained evaluation sets (ACC %) and adversarially trained ones (ACC* %).
Label
MNIST
CIFARIO
GTSRB
CIFARI00
PubFig
Avg.
Mapping
Loss
ACC
ACC
ACC
ACC
ACC
ACC*
ACC
ACC#
ACC
ACC#
ACC
ACC*
AIl-O-All All-to-One
91.2
89.6
91.0
88.4
96.6
93.2
86.7
82.5
83.0
90.7
87.3
Lnatetinet
87.1
84.8
87.1
84.5
91.7
89.2
79.8
78.5
81.0
79,8
85.3
83.4
87.3
86.3
88.1
86.6
93.0
90.8
83.3
81.0
83.7
81.1
87.1
85.2
90.0
87.4
89.3
87.5
92.6
89.1
82.4
85.0
83.2
80.9
87.5
86.0
Ladogiool
84.4
83.3
85.5
83.8
85.6
84.1
78.5
77.3
79.7
78.4
82.7
81.4
Indegnnnt
76.9
74.8
78.2
76.8
82.1
80.4
73.0
71.3
69.2
67.0
75.9
74.1

---

7 Ablation Study
Ablation Study on Validation Dataset. 'To ascertain the robustness of TRODO against different datasets in the validation set, we conducted experiments using various datasets as the validation set, as presented in Table In these experiments, we replaced our default validation dataset, Tiny ImageNet, with alternative datasets. Throughout these tests, all other elements of our methodology remained constant to isolate the impact of the validation dataset changes on TRODO's performance. Moreover, to quantitatively support our claim regarding the effectiveness of near-OOD samples compared to far-OOD samples, we provide the distance between the validation set and the target dataset. The target dataset refers to the ID set on which the input classifier has been trained. For computing this distance, we used the Fréchet Inception Distance (FID) [T]], well-known metric for measuring distance in generative models. Lower FID values indicate a smaller distance, and vice versa. As the results indicate, in the near-OOD scenario, our method appears more effective. More details can be found in Appendix Section L
Ablation Study Boundary Confidence Level. We also conducted an ablation study on the boundary confidence level hyperparameter, denoted as 7, which is preset at 0.5 in our standard pipeline. By keeping all other variables constant and varying y across range of values, we assessed TRODO's sensitivity to this parameter. The results of these experiments are presented in the Table [ illustrating how different settings of y affect the effectiveness of TRODO (extra ablation studies are available in Appendix Section
Table 4: Accuracy of TRODO using various Validation (and OOD) datasets for different ID data. Each validation is used to find the hyperparameters (e and r) and also as OOD datasets to find signatures. You can see the effect of choosing near-OOD dataset. For example, for CIFARIO, STL-10 and Tiny ImageNet are better choices than the other two datasets
Validation
MNIST
CIFARIO
GTSRB
CIFARI00
PubFig
Accuracy
FID
Accuracy
Accuracy
Accuracy FID Accuracy
FID
FMNIST
94.6
78.7
80.4
69.5
138
72.1
120
SVHN
92.3
118
82.6
92
84.3
105
74.3
124
73.2
137
STL-10
70.9
134
96.8
76
95.2
86
82.0
91
85.4
89
Tiny ImageNet
91.2
108
91.0
96.6
86.7
88.1
96
Table 5: Accuracy of our method with different boundary confidence level.
?
Boundary Confidence
0.2
0.3
0.4
0.5
0.6
0.7
0.8
MNIST
81.0
74,8
89.1
91.2
85.2
75.6
81.8
CIFARIO
79.7
85.6
91.0
GTSRB
94.0
91.3
92.6
96.6
90.1
88.6
92.2
CIFARTOD
77.1
80.2
86.7
PubFig
78.7
82.5
84.4
88.1
90.3
86.2
79.8
8 Acknowledgments
We acknowledge Mohammad Sabokrou for his contributions to this project. Mohammad Sabokrou's work in this project was supported by JSPS KAKENHI Grant Number 24K20806.
Conclusion
In conclusion, this study presents TRODO, robust and general method for scanning and identifying trojaned classifiers with low time and resource complexity. TRODO's strength lies its ability to detect trojans in diverse scenarios, including those involving adversarially trained models. In- terestingly, TRODO is applicable even scenarios where no data available. Our experimental results demonstrate TRODO's superior performance, achieving high accuracy across various attack types and benchmark datasets. The adaptability and effectiveness of our approach mark significant advancement in enhancing the reliability and security of deep neural networks in critical applications.
10

---

References
[1] Omkar Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In 2015 of the British Machine Vision Conference 2015. British Machine Vision Association, 2015.
[2] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firnrr, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Uns Muller, Jiakai Zhang, et al. End to end learning for self-driving
cars. arXiv preprint arXiv:1604.07316 2016.
[3] Kenneth S Miller. On the inverse of the sum of matrices. Mathematics magazine, 54(2)66772, 1981.
[4] Tianyu Gu, Brendan Dolan Davitt,,, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708. 1.0v:1770..6733: 06733, 2017.
[5] Anh Nguyen and Anh Tran. Wanet-impercepible warping-based backdoor attack. arXiv preprint arXiv:2102. 10369, 2021.
[6] Yiming Li. Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey, IEEE Transactions on Neural Networks and Learning Systems, 2022.
[7] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium Security and Privacy (SP), pages 707-723, 2019.
[8] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS '19, page 1265-1283 12822, New York, NY, USA, 2019. Association for Computing Machinery.
[9] Guangyu Shen, Yingqi Liu. Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization. arXiv preprint vrxiv:2102.0512: 2021.
[10] Wenho Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems, 2019.
[11] Xiaoling Hu, Xiao Lin, Michael Cogswell, Yi Yao. Susmit Jha, and Chao Chen. Trigger hunting with a topological prior for trojan detection. In International Conference on Learning Representations, 2022.
[12] H. Wang, Z. Xiang, D. J. Miller, and G. Kesidis. Mm-bd: Post-training detection of backdoor attacks with arbitrary backdoor pattern types using a maximum margin statistic. In 2024 IEEE Symposium Security and Privacy (SP), pages 19 19, Los Alamitos, CA, USA, may 2024. IEEE Computer Society.
[13] Zhen Xiang, Zidi Xiong, and Bo Li. UMD: Unsupervised model detection for X2X backdoor attacks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 38013 38038. PMLR, 23 23-29 Jul 2023.
[14] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 1rXin:1712.055222 2017.
[15] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Advances in neural information processing systems, 31, 2018.
[16] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamessing adversarial examples. arXiv preprint arXiv:1412.6572 2014.
[17] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint ar06v: 06083, 2017.
[18] Alexander Tsigler. Gabor Lugosi, Peter Bartlett, and Phil Long, Benign overfitting in linear regression. 2020.
PNAS, 117(48)3300333000
[19] Marzieh Edraki, Nazmul Karim, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Odyssey: Creation, analysis and detection of trojan models. IEEE Transactions on Information Forensics and Security.
16:4521-4533, 2021.
[20] Dan Hendryeks and Kevin Cimpel. A baseline for detecting misclassified and ofi-t--disttiuution examples in neural networks. In International Conference on Learning Representations, 2017.
[21] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of -uttt--distribution image detection in neural networks. arXiv preprint arXiv: /ttv:/766.020 026902 2017.
[22] Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813 822, 2021.
[23] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan Exploring the limits of out-of-distribution detection.
Advances in Neural Information Processing Systems, 34:7068-7081, 2021.
[24] Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and nutto--distribution examples
in neural networks. arXiv preprint arXiv: 1610.020.336 2016.
[25] Lukas Ruff. Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Robert Müller. A unifying review of deep and shallow anomaly
11

---

detection. Proceedings of the IEEE, 109(5):756775, 2021.
[26] Mohammadreza Salchi, Hossein Mirzaci, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohhan, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of nut-of--iitribution detection: Solutions and future challenges. arXiv preprint arXiv:2 /0./5 2021.
[27] Mohammad Azizmalayeri, Arshia Soltani Moakhar, Arman Zarei, Reihanch Zohrabi, Mohammad Manzuri, and Mohammad Hossein Rohban. Your out- out-of-distribution detection method not robust! Advances in Neural Information Processing Systems, 35:4887-4901, 2022.
[28] Shao-Yuan Lo. Poojan Oza, and Vishal Patel. Adversarially robust one-class novelty detection. JEEE Transactions on Pattern Andlysis and Machine Intelligence, 2022.
[29] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust out-o--distribution detection for neural networks. arXiv preprint arXiv:2003.097
2020.
[30] Rui Shao, Pramuditha Perera, Pong C Yuen, and Vishal M Patel. Open-sct adversarial defense. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pages 682-698. Springer, 2020.
[31] Rui Shao. Pramuditha Perera, Pong C Yuen, and Vishal M Patel. Open-set adversarial defense with dlan-adverrsaiill mutual learning. International Journal of Computer Vision, 130(4)) 2022.
[32] Louis Béthune, Paul Novello, Thibaut Boissin, Guillaume Coiffier, Mathieu Serrurier, Quentin Vincenot, and Andres Troya-Galvis. Galvis. Robust one-class classification with signed distance function using I-lipschitz neural networks. arXiv preprint arXiv:2303. 009777888.... 2023.
[33] Adam Goodge, Bryan Hooi, See Kiong Ng. and Wee Siong Ng. Robustness of autoencoders for anomaly detection under adversarial impact. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 1244-1250, 2021.
[34] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha, Atom: Robustifying out-of-distribution detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases. Research Truck: European Conference, ECML PKDD 2021, Bilbuo, Spain, September 13-17, 2021, Proceedings, Part III 21, pages 430-445. Springer, 2021.
[35] Kiran Karra, Chace Asheraft, and Neil Fendley. The trojai software framework: An opensource tool for embedding trojans into deep learning models, 2020.
[36] Zhen Xiang, David J. Miller, and George Kesidis. Detection of backdoors in trained classifiers without access to training set. IEEE Transactions on Neural Networks and Learning Systems, 3377)11177-119 2022.
[37] Huili Chen, Cheng Fu. Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In Proceedings of the Eighht International Joint Conference on Artificial Intelligence, IJCAI-19, pages 4658 4664, International Joint Conferences on Artificial Intelligence Organization, 7 2019.
[38] Chong Fu, Xuhong Zhang, Shouling Ji, Ting Wang, Peng Lin, Yanghe Feng, and Jianwei Yin. Freeeagle: Detecting complex neural trojans in data-free cases, 2023.
[39] Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash. and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 301-310, 2020.
[40] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting trojans using meta neural analysis. In 2021 IEEE Symposiun on Security and Privacy (SP), pages 103-120 IEEE, 2021.
[41] Zhen Xiang, Zidi Xiong, and Bo Li. Cbd: certified backdoor detector based on local dominant probability. Advances in Neural Information Processing Systems, 36, 2024.
[42] Xiaoyu Zhang, Rohit Gupta, Ajmal Mian, Nazanin Rahnavard, and Mubarak Shah. Cassandra: Detecting trojaned networks from adversarial perturbations. IEEF Access, 9: 1335866-135867, 2021.
[43] Yiyou Sun, Yifci Ming. Xiaojin Zhu, and Yixuan Li. -fttt-fdiistiiution detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827-20840. PMI.R, 2022.
[44] Jonathan Uesato, Brendan O' O'Donoghue, Pushmeet Kohli, and Aaron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In Jemnifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 5025 5034, PMLR, 10 15 Jul 2018.
[45] Arun Sai Suggala, Adarsh Prasad, Vaishnavh Nagarajan, and Pradeep Ravikumar. Revisiting adversarial risk. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Secone Interna- tional Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 123192339.11 PMLR, 16 18 Apr 2019.
[46] Justin Khim and Po-Ling Loh. Adversarial risk bounds via function transformation, 2019.
[47] Waleed Mustafa, Philipp Liznerski, Antoine Ledent, Dennis Wagner, Puyu Wang, and Marius Kloft. Non-vacuous generalization hounds for adversarial risk in stochastic neural networks. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on A Artificial
12

---

Intelligence and Statistics, volume 238 of Proceedings of Muchine Learning Research, pages 4528-4534 PMLR, 02-04 May 2024.
[48]
Emilio Rafacl Balda, Arash Behboodi, Niklas Koep, and Rudolf Mathar. Adversarial risk bounds for neural networks through sparsity based compression, 2019.
[49]
Muni Sreenivas Pydi and Varun Jog. Adversarial risk via optimal transport and optimal couplings. In Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7814-7823. PMIR, 13-18 Jul 2020.
[50] Carl-Johane Simon- Gabril, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez-Paz. First-order adversarial vulnerability of neural networks and input dimension. In International conference machine learning, pages
PMLR, 2019.
[51] Xin Zou and Weiwei Liu. On the adversarial robustness of out-of-distribution generalization models. Advances in Neural Information Processing Systems, 36, 2024.
[52] Stanislav Fort. Adversarial vulnerability of powerful near out-of-distribution detection. arXiv preprint arXiv:2201.00112. 2022.
[53] Maximilian Augustin, Alexander Meinke, and Matthia Hein. Adversarial robustness on in-and out- distribution improves explainability. In European Conference on Computer Vision, pages 228-245. Springer, 2020.
[54] Yifan Hao and Tong Zhang. The surprising harmfulness of benign overfitting for adversarial robustness. 2024.
[55] Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems, 33:3454-3466, 2020.
[56] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP),
pages 101-105. IEBE, 2019.
[57] Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15074-15084, 2022.
[58] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li. Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF international conference on computer vision.
pages 16463-16472, 2021.
[59] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks, arXiv preprint 112 2019.
[60] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: a large-scale hicrarchical image datahase. pages 248-255, 06 2009.
[61] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009, Seyed-Mohsen Moosavi-Dezfoole Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks, 2016.
[62]
[63]
Arthur Jacot, Franck Gahriel, and Clément Hongler. Neural tangent kemnl:: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[64] Johannes Stallkamp. Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi class classification competition. In The 2011 international joint conference on neural networks, pages 1453 1460. IEEE, 2011.
[65] Neeraj Kumar, Alexander C Berg, Peter N Bellumeuu, and Shree K Nayar. Auribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision, pages
IEEE, 2009.
[66] Wenbo Jiang. Hongwei Li, Guowen Xu, and Tianwei Zhang- Color backdoor: A robust poisoning attack in color space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8133-8142, 2023.
[67] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 TEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. Kaiming He, Xiangyu Zhang, Shaoqing Rea, and Jian Sun. Identity mappings in deep residual networks, 2016.
[68]
[69]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[70] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference Learning Representations, 2019.
[71] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-seale update rule converge nash equilibrium. CoRR,
13

---

abs/1706.08500 2017.
[72]
Alexander Buslaev, Vladimir I. Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and
Alexandr A. Kalinin. Albumentations; Fast and flexible image augmentations. Information, 11(2), 2020.
[73] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D. Cubuk, Quoc V. Le, and Barret
Zoph. Simple copy-paste is strong data augmentation method for instance segmentation, 2021.
[74] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards
deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 06083, 2017.
[75] Rahul Rade and Mored- Ahoosavi-Dezfool Reducing excessive margin achieve a better accuracy
vs. robustness trade-off. In International Conference on Learning Representations. 2021.
[76] Kimin Lee, Kibok Lee. Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. In Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31.
Curran Associates, Inc., 2018.
[77] Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.0900 2021.
[78] A. Bendale and T. F. Boult. Towards open set deep networks. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1563 1572, Los Alamitos, CA. USA, jun 2016. IEEE Computer Society.
[79] Niladri S Chatterji and Philip M Long. Foolish crowds support benign overfitting. Journal of Machine Learning Research, 23(125))1-12, 2022.
[80] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 0--25237-25200 2022.
[81] Xingyu Xu and Yuantao Gu. Benign overfitting of non-smooth neural networks beyond lazy training. In International Conference on Artificial Intelligence and Statistics, pages 1 11094-11117. PMLR, 2023.
[82] Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer relu convolutional neural networks. In International Conference on Machine Learning, pages 17615-17659.
PMLR, 2023.
[83] Moritz Haas, David Holzmüller, Ulrike Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. Advances in Neural Information Processing Systems, 36, 2024.
[84] Lisha Chen, Songtao Lu, and Tianyi Chen. Understanding benign overfitting in gradient- meta learning. Advances in Neural Information Processing Systems, 15:87-7881889 2022.
[85] Neil Mallinar. James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward refined laxonomy of overfitting. Advances in Neural Information Processing Systems, 15:1182-1199 2022.
[86] Zhu Li, Zhi-Hua Zhou, and Arthur Gretton. Towards an understanding of benign overfitting in neural networks. arXiv preprini
2021.
[87] Alexander Tsigler and Peter I. Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning
Research, 24( (1(31)):1113 1-76, 2023.
[88] Ke Wang and Christos Thrampoulidis. Benign overfitting in binary classification of gaussian mixtures. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4030 4034, IEEE, 2021.
[89] Zhu Li. Weijie J Su, and Dino Sejdinovic. Benign overfitting and noisy features. Journal of the American Statistical Association, 118(544):2876 2888, 2023.
[90] Guy Kornowski, Gilad Ychudai, and Ohad Shamir. From tempered to benign overfitting in relu neural networks. Advances in Neural Information Processing Systems, 36, 2024.
[91] Xuran Meng, Difan Zou, and Yuan Cao. Benign overfitting in two-layer relu convolutional neural networks for xor data. arXiv preprint 2.X.:233000117.. 2023.
[92] Jinghui Chen, Yuan Cao, and Quanquan Gu. Benign overfitting in adversarially robust linear classification.
In Uncertainty in Artificial Intelligence, pages 313-323. PMLR, 2023.
[93] Amartya Sanyal, Puneet K Dokania, Varun Kanade, and Philip HS Torr. How benign is benign overfitting?
arXiv preprint arXiv:2007.000 2000,04028, 2020.
[94] Xinwei Liu, Xiaojun Jia, Jindong Gu, Yuan Xun, Siyuan Liang, and Xiaochun Cao. Does few-shot learning suffer from backdoor attacks? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19893-19901, 2024.
[95] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep lcarning models resistant to adversarial attacks. arXiv preprint arXiv: (00X:v/7006.00000 2017.
[96] Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. Adversarial vulnerability for any classifier. Advances in neural information processing systems, 31, 2018.
14

---

[97] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high-dinennsioall ridgeless least squares interpolation, 2020.
[98] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: comprehensive henchmark of backdoor learning. Advances Neural Information Processing Systems, 35:10546-1055 2022.
15

---

A Benign Overfitting of Trojaned Classifiers
CIFAR-18
CIFAR-100
.006
175
15%
590
671
30L
306
SI1A
JAENS
VAhT
SBN
55%
195
154
800
504
20AA
Kalio
30W
155
Clenn
107
55%
654
300
204A
754
180 Rachal
Patul
Figure 3: Model accuracy across different architectures and datasets. Trojaned models for all backdoor attacks show consistent slight decrease in accuracy compared to clean models, suggesting benign overfitting Trojaned classifiers.
16

---

Triggers
Poisoned Samples
2
BadNer
X/n +
BadNee
2
SBPP
X
BBP
2
fColor
Xln +
fcolor
2
In
fwanet
Xln +
fwanet
2
f
XIn
Blended
fsig
fsig
XIn
2
fSSBA
X +
fSSBA
ImputAAare
X +
SmmpaAware
Figure 4: The effect of overlaying triggers OOD data, in various attacks. As demonstrated, applying the trigger (which used to poison training data) on even far-OOD samples, fools the model into identifying them ID. This is due the benign overfitting the trigger present in the training data.
B Examples of crafted Near OOD samples For Various Datasets
We provided images of some near-OOD data corresponding to random samples of our dataset (see Figure 5)
17

---

Trijil
Publlig
GUSRR
MINST
CIPARII
CIEARIO
Nureal
Tranileneed
Nernal
Traadioned
Trandfenee
Tranderuul
Nenal
Trundrack
Trandferedd
9
2
7
Figure 5: Examples of ID samples and their corresponding crafted near-OOD samples. We used
Elastic 72 random rotations, and cutpaste [73].
C Robust OOD detection in Adversarially Trained Classifiers
Adversarial training methods AT 74 and HAT 75 designed to enhance model robustness by exposing the classifier perturbed data during training, generally improve model's resilience against adversarial attacks within its training distribution. However, studies [27 29 indicate potential limitation when such classifiers are evaluated in OOD detection tasks, where small perturbation in attack can cause sample from the closed set to be classified as anomaly and vice-versa. This limitation arises because the models not consider samples from open set during training. We provide Table from 27 that highlights the issue.
8 for various methods trained with CIFAR- Table 6: OOD detection AUROC under attack with 255
10 or CIFAR- 100 as the training (closed) set. A clean evaluation indicates no attack on the data, whereas attack evaluation means that out and in data is attacked. The best and second-best results are distinguished with bold and underlined text for each column.
CIFAR-BB 10
CIFAR- 100
Method
Clean
Attack
Clean
Attack
VIT (MSP)
0.975
0.002
0.879
0.002
ViT (MD)
0.995
0.000
0.951
0.000
VIT (RMD)
0.951
0.025
0.915
0.037
VIT (OpenMax)
0.984
0.004
0.907
0.001
AT (MSP)
0.735
0.174
0.603
0.085
AT (MD)
0.771
0.232
0.649
0.108
AT (RMD)
0.836
0.151
0.700
0.136
AT (OpenMax)
0.805
0.208
0.650
0.132
HAT (MSP)
0.770
0.325
0.612
0.176
HAT (MD)
0.789
0.369
0.810
0,363
HAT (RMD) HAT (OpenMax)
0.878
0,258
0.730
0.191
0.821
0.415
0.703
0.263
are common methods in OOD detection literature leverage classifier as OOD detector. The results reported for each outlier method correspond to the best-performing detection method. Notably, approach has surpassed the
Here MD 76
Relative MD 77 and OpenMax
[78
18

---

slate-of-the-ar in robust out- out-of--istrioo of-distribbttin setting (ATD) for nearly all datasets.
[1 1 Σ Σ
Σ (z+ Ma) (z+ uux) = K k= Fyn
(5)
In addition, to use RMD, one has to fit a (no, Ea) to the whole in-distribution. Next, the distances and anomaly score for the input x' with pre-logits z' computed
DR (x') = (2' - -ma)) Σ D (x' pa). RM (z') MDR (2) - MD (z'), D (a') =-min (MDs (z')}, score RMD (x') = - min (RMDR (z))
(6)
19

---

D Algorithms
Algorithm Trojan scanning by detection of adversarial shifts in out-of-distribution samples
Input: A c-class Classifier fo. (Optional) a small of benign samples P.. A of hard transformations T, Adversarial perturbation budget c, scanning threshold r Output: Decision (Trojaned/ / Clean)
1: if "D, is not provided then
2: D, < TinylmageNet
3: end if
4: Applies a random permutation transformations to
5: procedure G(z, T)
6: Tpem f- -Randomly Permute(T)
7: for & TTerm do
- ((x) 9: end for
8:
10: return æ
11: end procedure
12: Obtain DOOD by applying hard augmentations on each sample of D':
13: DOOD (0
14: for I E DD do
G',
16: DOOD t DOOD (
end for
18: AI +0
for DOOD do
20:
Adversarial Perturbation:
r² PGD c)
22:
ID score computation:
Stefure + max;= (x)
Saleer max;=1.... f0(x') (x*)
25: AID + Satter Sbetone
Append AND to AI
27: end for
Smeun
[Poun] EsDe 6TD
29: if Smean <
then
30:
return Clean
31: else
32:
return Trojaned
33: end if
E Extended Related Work
Backdoor Attacks. Injecting pre-defined triggers into the training data is the most common approach to implement backdoor attacks. BadNet [4] is the first backdoor attack against DNN models, which involves modifying clean image by inserting small, predetermined pattern at a fixed location, thus replacing the original pixels. Blended [14] aimed to enhance the invisibility of the trigger pattern by scamlessly blending it into the clean image through alpha blending. SIG [56) utilized sinusoidal waveform signal the trigger pattern. To achieve better stealthiness, many attacks with invisible and dynamic triggers have been proposed. Input-aware (55) proposed a chining-controllabee attack method that simultaneously learned the model parameters and trigger generator to produce unique trigger pattern for each clean test sample. For more details regarding other attacks including BPP [57], SSBA (38), WaNet (5), and (66] read Appendix Section H
Benign overfitting The phenomenon benign overfitting, where models perfectly fit noisy data without compromising generalization, was first explored in [18). They characterized the condi- tions under which the minimum norm interpolating prediction rule achieves near-optimal accuracy, emphasizing the necessity of overparameterizatton.. Subsequent studies extended these findings to
20

---

various neural network architectures. Notably [79] delves into sparse interpolating procedures for linear regression with Gaussian data, highlighting conditions under which benign overfitting occurs in overparameterized regimes. Their work establishes lower bounds on excess risk, proving that overfitting can indeed be benign.
Two-layer neural networks have also been extensively studied to understand benign overfitting under various conditions. Benign overfitting in two-layer convolutional neural networks is investigated by [8), identifying a phase transition between benign and harmful overfitting. Similarly, [8]] analyzes non-smooth neural networks, providing theoretical insights into when overfitting can remain benign even beyond lazy training scenarios. This exploration is extended to ReLU networks in [82]. demonstrating the conditions that facilitate benign overfitting and the sharp transitions to harmful overfitting. Additionally, [83] shows that overfitting in Sobolev RKHSs can achieve optimal rates without being intrinsically harmful.
The phenomenon is also observed in more complex architectures. Gradient-based meta learning examined in [84], revealing that benign overfitting in empirical risk minimization (ERM) can extend to meta-learning algorithms like MAML. A refined taxonomy of overfitting in proposed in 885] identifying tempered overfitting as intermediate regime between benign and catastrophic overfitting. Benign overfitting has been studied in various other architectures and settings as well
86
90 91
Benign overfitting has been studied in the context of adversarial robustness in 54 Notably,
54 theoretically shows for linear and two-layer networks that benign overfitting will become harmful overfitting under adversarial attacks.
The interplay between benign overfitting and security vulnerabilities like backdoor attacks is critical in [94] revealing that few-shot learning models tend to overfit benign or poisoned features, impacting robustness.
Adversarial risk Adversarial risk refers to the vulnerability of machine learning models to adversarial examples perturbations intentionally crafted to mislead the model. Research in this area seeks to understand and mitigate these risks. A seminal work by (95] presented robust optimization techniques to defend against first-order adversarial attacks, establishing foundational adversarial training methodologies. Following this, other studies have explored the theoretical limits of adversarial robustness. A framework to evaluate the adversarial vulnerability of any classifier is provided in 196). showing intrinsic limitations based on the classifier's architecture and data distribution.
Previous work has established bounds on this metric via function transformation [46), PAC-Bayesian 47]. sparsity-based compression 488, Optimal Transport and Couplings 4 in terms of input dimension [50].
The intersection of adversarial risk and out-of-distribution (OOD) detection has gamerrd increasing attention. The vulnerabilities existing OOD generalization methods to adversarial attacks are identified in [51], prompting the development of algorithms to enhance OOD adversarial robustness. RATIO was introduced by [53]. a training procedure that improves adversarial robustness for both in- distribution and OOD samples, thereby enhancing model explainability. The adversarial vulnerability of current OOD detection techniques is discussed in [52), suggesting that ensemble methods and combining multiple OOD detectors can significantly enhance robustness against adversarial attacks.
Understanding the theoretical limits of adversarial vulnerability remains crucial for developing robust models. It is proved in (50) that adversarial vulnerability increases with the gradients of the training objective and scales with the square root of the input dimension, making larger images more vulnerable. The trade-offs between robustness and accuracy are explored in [96]. establishing mathematical framework to evaluate these limits. This discussion is extended in [95] through robust optimization, quantifying the trade-offs, and providing guidelines for creating more resilient models. These foundational works underscore the inherent challenges achieving robustness, emphasizing the need for innovative approaches to bridge the gap between theory and practical applications.
F
Theoretical Proofs
Proof of Theorem
Proof. Let μ E 12 be arbitrary, by taylor series around μ4 we have:
21

---

V2h(w, 14)
Vzh(w, (u)
h(wxx)) =
Σ
(x 11) + Σ
(x - 11)
7!
jpk+1 1=|=5
By taking derivative we have:
v2h(w,1) 7!
V2h(w, p2)
vah(w, Vv
E h1=
(x- 11) +
Σ j#b+1
V
(x - (4)
y!
hn|jj
The distribution P has the same j-th order moments as the distribution P for all j + k, therefore, by taking expectation and using the triangle inequality we have:
Ra pt.
(h, w) = =oREppt VIIV2 (w, 2 a Byupp tx
Vxh(w,))]
V2h(w, p))
Vih(w, p))
2 a E pppp
V
(I - 11)
- -EENP
V
(x 11)
?
7!
bn|=k+1
|+/=
Vih(w, u))
Vih(w,
-a Ey Bynpt
Vyh(w, x) V
Σ
(x p2)
+EEOP
(x 12)7 11
7
?!
17 mm+1
17/===+1
Vzh(w,,))
=a|lsV= Σ
17 7
-a Boxu
V x)
-V
(xc - 11)²
+Brup V
Vih(w, n))
(xr -1)-
117 11
7
7!
h+=k+1
In/=k11
vzh(w, 11-a|lll al|EEND (V
=a|s||/V>
Σ
(Va((w, x|]
7
In|=
Note that all moments with orders less than are equal, hence the difference of the moment of order k around μ is equal to the difference of the moment of order & around 0. Since j was arbitrary, conclude:
R.t.t (h, w) 2 als) max V
V2h(w, ?
|| - allEa V ( x)]l
|r|=
Proof of Theorem
Linear neural network
Proof. We define
X [T1,..., Fn
T E R", X' = [-| + b I,...,
+ T E Rm, Y =
[y...... ynl E R2, Y Ber..., Mel
E RT, then we have:
X
X
xx (XT
+ +XTr)
X
X'
X
Y
=
22

---

Note that lim (xTx)= C is constant matrix (inverse of the covariance matrix) and lim - I is a constant vector (mean vector), and we have lim E 4 ms by the law of large numbers. If rank(XX" X') 1, then according to the Miller's Lemma [3] Ch lim ||ma (x x X x² -1 = trkk) -1 2((1) =
where k (XTXX) xx
( x mst mtst + mtt (xTx) If
rank(X' 2 2. we can decompose into sum matrices with rank and inductively infer
the same bound ((1)). Now have:
| + = x + n n
m Ve +J) 22 ||t|))
lim 12 11000
71
n
Therefore, we conclude:
lim (hi, n-000
lim V 22,,(00,
lim 2-000
||0|||
-2 (0=11
20000
n
Two-layer neural network
Proof. We define F
(wg 21) 21)...., h2 (w
R
VF
IV wh (1,,,1)). ......., Vwhg(wo, xr,)] E 2k(++))xn F = [ Wo, x +t),... h2
Wp J
+t]]
R... VF = V w2 (wo, x4 +t), V Wyhoy onm + +t)l E R jmm, y [y1 and y' = Wee................ .........0............... ve]
Vn
E R".
E E..
Assume the two-layer network (w) as described the paper trained using gradient descent on D' with learning rates 17 1//mar (VV.. / F, VF')). According to Proposition 1 in 97]. the optimal solution will be:
VF VF
VF
VF VF
Y - F y' Fr
VF'
= (VF VFF VF (VFT (Y (VF
F) + V -Y' (X')FFF
For any i E m..... E (1,...,,)) }}} we have:
V Ww/22woo 24 + t) = Vkp
[wo. ReLU + ()4 + t). ReI. (2f + t)]]
[u) + t).
00 (2) +t]] if o (2) + 2 0
Vxp
Vxp 0.0].
otherwise
Therefore, we can assume there are some constant matrices G1, G2, Gi, G22 such that F" = LG + G2 and V tG G²-- Moreover, note that lim (VF VF) C₁ is constant matrix (inverse of the covariance matrix). If rank(VF" VF') 1. then according the Miller's Lemma 3 C₁k
lim Il VF C 1 trtk())
|| = Ω(1)
where A (VFTTTF > we can decompose into sum matrices with rank and inductively in- fer the same bound Q(1). Now we have:
(VF
( VF + (([ (VF) If
lim _ (VFF (Y 200 7
-F F')) 2 m n
|+|l
23

---

Therefore, we conclude:
lim n-0
||0||| = 52 m
(m ||e||
Now we have:
R.P (lig, 10) ERa
sup Sa
22 x + +6 -hig(u; x2)
0 122(2wo x)) (+ - wo)ll = 2(]]@]]
= aB, /2 x)||
aLE,, VV x))
Owdxx
Therefore, we conclude:
lim R. (1ig, ú0) lim || = Ω (m [|t|] n-000
x-000
G
Preliminaries
Adverserial attack to classifiers has been shown that DNNs are vulnerable to adversarial attacks across various tasks, predominantly explored in classification tasks. During inference, adversarial perturbations added to input data can cause classifiers to mispredict their labels. other words, a perturbation such as 8 is added to sample with label y to fool the model into outputting ý for the adversarial input sample x², where x² = x + 6. Specifically, PGD an iterative adversarial attack 17 that crafts adversarial samples by ensuring the noise is projected within the foo norm in N-step:
T = x, # ( + a sign (V,J (xt ui)),
xx ====r; x
where Is denotes the projection on the norm ball S around x and J (x7,9) denotes the targeted objective function, which for classifiers is cross entropy. Previous studies have shown that standard- trained classifiers are vulnerable to ad adversarial attacks, even weak attacks like FGSM. Various defense mechanisms have been proposed to address this challenge, with adversarial training on the training dataset being the most effective defense.
Using Classifier as OOD Data Detector Classifiers such f, trained dataset denoted as D, can be used as OOD detectors due to their learned features. Specifically, another dataset with separate semantics from D. denoted as D', is assumed to be the source of OOD samples where D represents in-distribition samples. For instance, a classifier trained on Cifar10 is used to detect 10 OOD samples. The instructions detect OOD samples practically involve using the output distribution of a classifier over the classes of D for a given input, offering insights into the model's prediction confidence. Specifically, classifiers tend be more confident about ID samples compared to OOD samples. recently The Maximum Softmax Probability (MSP) has been proposed as an indicator of this confidence.
formally a well-trained classifier f logits from its penultimate layer on D with classes, for a given input sample represented by 2 = fz)), where is vector of unnormalized prediction scores [21, 22.... =x] for sample v. Applying the softmax function results in:
e
softmax(2))
MSP M (z) max (softmax (2))
se (e(1....k))
where z; is the logit corresponding the i-th class.
intutitively MSP,(2) represents the highest probability assigned to any class by the model, reflecting the model's confidence level in its prediction.
24

---

H
Datasets Details
Details for the Backdoor Attacks
This section provides detailed descriptions the backdoor attacks employed in our study, focusing particularly on the nature and implementation of their triggers.
BadNet 4 embeds malicious trigger, typically a small and visually distinctive patch, into the training data. This trigger is designed to be inconspicuous enough to evade detection yet recognizable by the trained model, leading it to misclassify imputs containing the trigger.
Blended [4] subtly blends a trigger into the training images at low intensity, making it hard to detect by human inspectors simple automated methods. The trigger effectively conditions the model to associate the slightly altered patterns with incorrect outputs.
SIG [56] corrupts training images with sinusoidal pattern, superimposed in way that does not require changes to the image labels. This makes the backdoor particularly stealthy as avoids the common detection methods that look for label inconsistencies.
uses image quantization combined with contrastive adversarial learning craft triggers that are embedded into the pixel values themselves. This approach alters the image at a level, making the modifications difficult to perceive or reverse-engineer.
BPP [57
Input-aware attacks [55] dynamically adjust their triggers based on the input features, allowing the backdoor activate only under specific conditions that are predetermined by the attacker. This adaptability makes the attack highly elusive and challenging to detect.
WaNet [5] introduces imperceptible warping to the image, manipulating its geometric properties subtly. This warping acts as a trigger that is extremely hard to spot with the naked eye, ensuring the model misclassifies the warped input while appearing normal to human observers.
SSBA (58) crafts sample-specific triggers that are invisible human detection by embedding them in way that aligns closely with the natural image structure. These triggers are tailored to each individual sample, increasing the difficulty of detecting and isolating the backdoor through general analysis.
Color (66) alters the color distribution of the input images, employing changes in the color channels as the trigger mechanism. This type of manipulation can remain under the radar of typical visual inspections while effectively conditioning the model to respond altered color cues.
These descriptions underline the diversity of the backdoor mechanisms used, highlighting the chal- lenges in detecting and mitigating such threats in machine learning models.
I Previous Trojan Scanning methods
Implementation
We implemented the baseline methods using their implementations in their publicly available GitHub repositories. For methods requiring supervision define threshold, followed the same approach as UMD.
Review of the Methods
NC Neural Cleanse 7 is method for scanning models by reverse engineering triggers and identifying outliers with significantly smaller perturbations. NC faces significant computational overhead, sensitivity to trigger complexity, and potential false positives. Moreover it is primarily designed to handle only All-to-One attacks.
ABS Artificial Brain Stimulation g detects backdoors in neural networks by stimulating individual neurons and analyzing their impact on output activations. Compromised neurons that substantially elevate a specific label are identified, and potential triggers are reverse-engineered to confirm the presence of backdoors. the method involves significant computational overhead and is sensitive to its underlying assumptions about compromised neurons and trigger behavior. ABS may struggle
25

---

with more advanced attack scenarios and primarily handles All-to-One attacks. Additionally, benign neurons with unique features may lead false positives, limiting its robustness some cases.
TABOR TABOR [T] is a trojan detection method for DNNs that frames the detection task as an optimization problem. It incorporates objective function with regularization terms inspired by explainable AI techniques to guide the optimization process and reduce the adversarial search space, improving trigger identification accuracy. However, TABOR's significant computational overhead present challenges, Additionally, it is primarily designed for geometric or symbolic triggers, potentially limiting effectiveness against irregular shapes trojans.
PT-RED PT-RED [36] is a post-training backdoor detection method for DNNs. reverse- ceverse-engineers potential backdoor trigger pattems by solving an optimization problem find minimal perturbations that cause misclassifications. However, it is primarily designed for single target class attacks with small trigger patterns. Moreover, the reverse-engineering process for identifying potential triggers is computationally intensive.
K-ARM K-Arm [ leverages an optimization approach inspired by the Multi-Armed Bandit problem to iteratively select and optimize class labels for potential backdoor triggers. This method improves detection efficiency compared to exhaustive search methods by using stochastic selection. However, the method still involves computational overhead and primarily focuses on specific and simple types of backdoor attacks. Additionally, K-Arm may face challenges in handling scenarios with more than one target label.
UMD Unsupervised Model Detection 13 is designed to detect X2X backdoor attacks, where multi- ple source classes are mapped to multiple target classes. The method involves eevrrs-engineeering triggers for each class pair using small set of clean data, and then defining transferability statistic (TR) for each class pair. TR measures how well the trigger for one class pair transfers to another. These statistics are used to select likely backdoor class pairs. An unsupervised anomaly detector then evaluates the aggregated trigger statistics to determine if a model backdoored. However, the method involves complex optimization processes, which can be computationally intensive. Handling very large datasets or a high number of classes can pose scalability challenges. Furthermore, UMD may struggle to handle models with multiple different trigger patterns, as it relies TR statistics that assume a single dominant trigger pattern and this method is not trigger-agnostic as its approach depends the specific type of attack.
MM-BD Maximum Margin Backdoor Detection 12 designed detect backdoor attacks in neural networks, regardless of the backdoor pattern type. The method operates by estimating maximum margin statistic for each class through gradient ascent from multiple random initializations, without any clean samples, and then using these statistics in an unsupervised anomaly detection framework to identify backdoor attacks. However, the method exhibits large false calse-positive rate for datasets with small number of class, and it struggles to detect attacks with more than one target label, where each source class may be mapped to a different target class. Additionally, MM-BD's effectiveness is significantly reduced when an adaptive attacker manipulates the learning process.
identifies backdoors in neural networks by training binary meta-classifier on features extracted from numerous shadow classifiers, both benign and Trojaned. However, this method struggles to generalize to types of attacks and model architecture it wasn't 't trained on.
MNTD MNTD 40
J Broader Impact
Our study introduces a method designed to identify potential backdoors embedded within classifiers. As result, our work contributes positively to societal impacts by enhancing security measures in machine learning applications and mitigating risks associated with malicious interventions.
K Baselines and Evaluation Benchmark
Baselines
In our evaluation, TRODO and TRODO-Zero are assessed alongside previous scanning methods including Neural Cleanse (NC) 7 ABS R PT-RED (66), TABOR [10], K-Arm (9, MM-BD [12. and UMD [13. For an equitable comparison, we set the confidence threshold at 95%, corresponding
26

---

to 5% desired false positive rate for NC, PT-RED, and UMD which are based on unsupervised threshold settings. Similarly, for ABS, and K-Arm, which rely on supervised threshold adjustment, we maintained consistent false positive rate of 5% across various tests and dataset configurations to maximize true positive outcomes. The capabilities and performance outcomes of these methods are detailed in Table T Further details can be found in Appendix Section !
Our Designed Benchmark
To effectively model real-world scenarios for the scanning task, we developed a benchmark covering various scenarios involving both clean and trojan classifiers. Specifically, to evaluate the generality of scanning methods, which is crucial in real-world applications, we included several datasets ranging from low to high resolution and different classifiers with various architectures. Moreover, different trojan attacks and label mappings were also considered, with classifiers covering both standard and adversarial training methods. More details about our developed benchmark are provided below.
Image Datasets Our benchmark comprises image datasets from various domains, including CIFAR1O. CIFAR100 [61], GTSRB [4]] PubFig 665 and MNIST. We considered two kinds of label mapping, All to One and All All.
Trojan Attacks In the former, the label of poisoned samples is changed to a single target class. In the latter, unlike All to One case, each class can be mapped to any arbitrary target class, which makes this setting more challenging. We included eight trojan attacks, comprising BadNet [4], Input-aware [55], BPP (57). SIG (56). WaNet (5), Color (66). SSBA (58] and Blended [14. The test set for each combination of image dataset and label mapping consists of a total of 320 models. We trained 20 trojaned models using each type of attack. We included 160 clean models, resulting in balanced set.
Adversarial Training To encompass a wider variety scenarios, we evaluated each configuration using both standard and adversarial training. We employed PGD-10 with 2 for adversarial 255
training.
Classifier Architectures We considered various architectures, including ResNer18 677, PreActRes- Net18 68). and ViT-B/16 [69] Previous works have solely focused on CNN-based architectures, underscoring the generality of our experiments.
TrojAl Benchmark
Another challenging benchmark that we include in our experiments is TrojAI (35), benchmark developed by IARPA designed to address challenges in backdoor detection. The license of this benchmark is Apache 2.0.
Structure For each round of the competition, test set, a hold-out set, and a training set of models are available and can be accessed from the TrojAl homepage. Almost half of the models in each set are trojaned. small set of benign samples from the training dataset of each model provided along with the model itself.
Trojan Attacks The models may be trojaned with various kinds of backdoors, including universal and label-specific. The triggers could be pixel patterns and Instagram filters. Triggers can be embedded within the model such that activation occurs only under specific conditions, such as possessing a certain texture or being located in designated area of the image. The complexity of models and trojan attacks grows from round round. The performance of methods on this benchmark provided in Table 2 We evaluate methods in terms of scanning accuracy and average scanning time for model.
L Fréchet Inception Distance (FID)
The Fréchet Inception Distance (FID) is a metric used to evaluate the quality of generative models, such as GANs. It compares the distribution of generated images to the distribution of real images by measuring the distance between two multivariate Gaussian distributions fitted to the feature representations of these images.
27

---

Given a set of real images {xi) N. and a set of generated images { pass both sets through pre-trained Inception v3 network to obtain feature representations. Let ((x) denote the feature representation of image z from the Inception network.
Calculate the mean and covariance of the feature representations for real images:
((ri)
Σ, N (0(x,) ux) ur)
Similarly, compute the mean and covariance for generated images:
No
Mg = Σ ((t))
Ng j=1
Eg = N (d(b,) (g)()( - ks))
The FID is defined the Fréchet distance between the multivariate Gaussian distributions N (M,, Σ,) and N ((gg Eg):
FID = ||, - Myll² Tr(E, + +. 2(),),))
Here, ||,,- - Mgll² 2 is the squared Euclidean distance between the mean vectors. Tr denotes the trace of matrix. E.)) is the matrix square root of the product of the two covariance matrices.
Lower FID scores indicate that the generated images have a distribution more similar to the real images, suggesting higher quality. Higher FID scores indicate greater dissimilarity between the distributions of generated and real images, suggesting lower quality.
M Limitation
In this study, we utilized a validation set and surrogate model to select hyperparameters, such as epsilon for the PGD attack. However, these hyperparameters are architecture-speiffe Therefore, each new architecture requires tailored approach: training validation set and tuning its corresponding hyperparameters for the input classifier. This process can time-consuming when encountering new architectures, although we limited our consideration to common architectures. Additionally, our research focuses on scanning trojaned classifiers. However, this task might need to be adapted for different tasks, such as object detectors, where our method would require adjustments to handle these cases effectively. Morever, our theoretical results on adversarial risk are limited to the noiseless settings and only address the backdoor effect up two layered networks which could be extended in future work.
N Models Dataset Creation Details
In our study, we generated 20 distinct models for each of the eight backdoor attack types across the ResNet18 and PreAct-ResNetle architectures and models for ViT for each dataset and attack. totaling 160 models for ResNet18 and PreAct- 88 and 40 models for VIT. We utilized the Back- doorBench framework [98] (https://gi com http///CC//B/ac////ee// /BccchoooooBBcch for seven attacks (BadNets, Blended, WaNet, SIG. BPP, Input-aware, and SSBA) and integrated additional attack (Color attack (66)) using this repository (https://gi thwbb com http://x11////oooooaaaaoooooo///
To ensure consistency and rigor, we adopted core framework settings from BackdoorBench and extended them encompass both standard and novel attacks, adjusting parameters specifically for the COLOR attack. Our consistent methodology, following BackdoorBench's protocols, allowed us to systematically evaluate the individual effects each backdoor strategy across the architectures, achieving high standard of experimental reliability and detail.
28

---

Extended Results
Table 7: Accuracy of our scanning method across various trojan attacks. For each attack, the trojaned
models in the evaluation set are backdoored only with that attack. The number of clean and trojaned models is balanced. The architecture of all the models is resnet18
Attacks
CIFARIO
MNIST
GTSRB
CIFARI00
PubFig
BadNet
93.5
99.7
87.0
83.5
87.2
WaNet
100.0
83.5
99.8
82.2
85.4
Blended
97.0
100.0
75.8
95.0
95.7
SIG
78.8
99.0
89.1
80.6
82.3
SSBA
80.5
68.1
86.2
88.6
90.4
Input-aware
92.3
93.0
99.2
82.9
83.4
BPP
96.0
99.7
98.2
99.5
100
Color
89.7
87.8
91.1
95.2
80.4
Table 8: Scanning performance of TRODO compared with other methods using PreAct ResNet-18 as
the backbone
Mapping
Label
Method
MNIST
CIFAR10
GTSRB
CIFAR100
PubFig
Avg.
ACC
ACC*
ACC
ACC*
ACC
ACC*
ACC
ACC*
ACC
ACC*
ACC
ACC*
68.2
52.0
62.8
65.9
53.2
One
MM-BD
81.5
67.3
75.6
57.1
758
61.1
86.1
72.7
62.4
47.6
76.3
61.2
UMD
79.5
58.8
76.4
51.5
79.2
64.7
67.5
54.6
64.5
47.0
73.4
55.3
TRODO-Zero
85.0
78.5
81.2
79.1
85.2
83.9
78.2
78.9
73.6
72.4
80.6
78.6
TRODO
88.7
90.5
90.2
93.4
90.1
85.6
80.2
78,8
88.5
86.2
K-ARM
56.8
49.7
54.6
47.6
57.5
48.9
51.3
45.0
50.6
47.3
54.2
47.7
MM-BD
52.7
42.3
49.3
35.1
57.0
44.2
41.3
32.0
40.0
34.0
48.1
37.5
UMD
61.5
75.5
55.9
83.5
47.5
74.8
56.0
TRODO
90.8
87.9
88.3
89.8
92.0
87.9
82.6
82.1
81.0
76.6
86.9
84.9
Table 9: Scanning performance TRODO compared with other methods using ViT-B-16 as the
backbone
Label Mapping
Method
MNIST
CIFAR10
GTSRB
CIFAR100
PubFig
Arg.
ACC
ACC*
ACC
ACC*
ACC
ACC*
ACC
ACC*
ACC
ACC*
ACC
ACC*
K-ARM
54.3
68.2
52.0
72.0
62.8
48.6
65.9
53.2
MM-BD
72.9
58.4
67.6
49.8
67.5
52.5
78.0
62.7
54.4
39.5
68.1
52.6
UMD
75.2
55.0
69.0
45.4
71.5
59.6
62,5
48.6
58.3
40.7
67.3
49.8
TRODO-Zero
78.5
71.8
72.9
71.5
76.5
76.1
71.6
70.2
68.9
65.1
73.7
71.0
TRODO
87.9
85.6
82.5
84.4
85.2
84.3
80.3
79.6
78.2
76.1
82.8
82.0
K-ARM
$4.9
43.3
50.5
43.1
51.5
47.6
46.4
39.9
49.4
41.9
50.5
43.2
MM-BD
51.9
40.7
44.3
33.1
57.8
41.6
41.0
29.6
40.4
28.0
47.1
34.6
UMD
73.9
56.4
69.4
48.6
77.7
58.6
43.4
40.1
67.5
49.4
TRODO-Zero
70.4
73.3
72.4
TRODO
87.6
82.3
82.6
8488
83.5
83.0
79,8
77.3
76.0
73.1
81.9
80.0
29

---

Table 10: Value of and 7 for different validation sets and backbone architectures.
ResNet-18
PreAct ResNet-18
VTT-b-16
Validution
FMNIST
0.0491
1.1625
0.0538
1.0007
0.0621
0.9341
SVHN
0.0476
1.1338
0.0524
1,0025
0.0598
0.9106
STL-10
0.0488
1.1571
TinylmageNi
0.0483
1.1523
0.0527
1.0179
0.0609
0.9150
Table 11: Detailed results of statistical significance of our method's performance over 10 runs, in terms of variance of accuracy.
Mapping
Mediod
XNITT
CIRARI
GlaRB
CIARRI
Puhlu
AR-ta-lix
TRIDO
91.3
MA211
9L8:15
194001
W6104
132
167281
825014
111-05
808118
913-14
873:15
TRUDD0-Zom 831013
113-11
253107
254111
TRO0K1
"0166"
Extra Ablation Studies
Performance of TRODO under different poisoning rates
Intuitively, increasing the poisoning rate enlarges the blind spots in trojaned classifiers, as these boundary regions where the poisoned data causes the model to overfit. Consequently. this will increase the probability that TRODO detects the trojaned classifiers. However, our signature is based on the presence of blind spots in trojaned classifiers and shows consistent performance across different poisoning rates. In the paper, we considered poisoning rate of 10% as it is common in the literature. In addition, we have provided TRODO's performance for different poisoning rates in Table
12 (other components of TRODO remained fixed.)
Table 12: Performance comparison of TRODO-Zero and TRODO under different poisoning rates
across datasets.
Label Mapping
Method
Puisoning Rase
MNIST
CIFARIO ACCHACC
CTSRB
CIEARI00
Publiig
Ang.
$0.1/78.2
81.3/77.0
83.58818
746672.7
72.1/69.8
78.3/75.9
All-to-Dmne
TRODO-7e50
3%
82.0779.3
82,5/78.1
85.483.2
76.774.4
74.07111
$0.1/77.2
84.0779.5
77.375.3
75.1/72.
10% (detault)
3I19/79.3
X2.7/78.5
$4.8233.
75.573377
73.2711.
79.4/77.8
1%
$9.5/87.4
8% 7/83.6
94 9/91
84.5/81.2
4/80.3
$8.8M5.1
86.3/82.7
All to One
TRODO
5%
92.899066
91.7/88.9
97.0994.1
87.5
$9.1/84.55
91.6/88.3
10%
91.2/99.6
91 91.0088.4
96.6993.2
86.782255
Performance TRODO-Zero under different OOD sample rates
In the first experiment, we performed an ablation study on the number of samples in our validation set. By default, TRODO-Zero uses 1% of the Tiny ImageNet validation dataset, which contains 200 classes, each with 500 samples. We explored the effect of varying the number of sample rates in Table 1
30

---

Table 13: Performance comparison of TRODO-Zero under different OOD sample rates across
datasets.
LabelMapping
Mechod
OOD- HODDSaamle-Raee
CIFARIO ACC/ACC
GTSRB ACCACCC
CIFARIOO ACC/ACC
Avg.
ACC/ACC
ACC/ACC*
ACC/ACC
0.1%
78.677711
80.5/16.
$2.68111
13,2/71,4
71,268.5
77.575.0
79,37779
81.2/7.1
53,38188
73.9/72.1
71,009.2
78.1/5.77
All-t Oze
TRODO Zero
0.3%
80.078.6
81.9777
S4.0/22.5
74.6/7288
72.69999
7N.8/6.4
80.6/9922
82.5/7844
54.6/3.11
15.2/73.4
79.377.0
1% (defanl))
80,9799.3
82.7/7855
S4.0/3..
75.5/73.7
73.2/70.6
79,57771
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We motivated our work by stating that current scanning methods are not attack-agnostic and fail to generalize to diverse attacks. Besides, almost all of them fail in scenarios where the given classifier is trained adversarially. Our method outperforms previous works in the mentioned settings, according to the results in Table [ We also support our claims the effect of using OOD data for trojan scanning via our theoretical analysis in Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations of our work and possible direction for future research in Section M
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
32

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Due to the limited space in the main pages, we have provided proofs in Appendix Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the code reproduce the results part of the supplemental material. Besides, we provided all the details of implementation Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
33

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have provided the code and the notebook easily reproduce results. We also have provided access to our benchmark in New Assets item in the checklist.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/ mublic/guidess otte///eemmmiiissioooooooioooioe
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. http/uullcggguucee http///ooeSuuiiissseeeeeee asiionooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The implementation details of the creation procedure our benchmark, and how we adapted baselines to the experiments and how we tune parameters discussed in Section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Due to space constraints the main body of the paper and to avoid overcrowd- ing the tables, we have included the statistical significance of our method's experiments in Appendix Section For the baselines, we have documented the average performance across five repeated trials.
34

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the "Impiementation Details" subsection of the Experiments section (see Section (). we have thoroughly outlined the computational resources utilized. Given that the efficient execution of our method is key advantage, have included comprehensive details about the computing resources employed to assess our method comparison with other approaches.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttii/ggii/////////o/ooo//////a
Answer: [Yes]
Justification: After thorough review of the NeurlPS Code of Ethics, we are confident that our work fully complies with all its aspects. Additionally, our research adheres ethical standards concerning social impact and harmful consequences, based on both our proprietary code and the publicly available datasets employed.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
35

---

discussing the subject of detection of The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the appendix, we have allocated a section discuss the broader impacts of our research (See Appendix Section Our study introduces method designed to identify potential backdoors embedded within classifiers. As result, our work contributes positively to societal impacts by enhancing security measures in machine learning applications and mitigating risks associated with malicious interventions.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No safeguards needed, backdoor attacks, and not to incur them.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
36

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all image datasets that we used in our research. Besides, we have provided the license of TrojAI benchmark on which we have evaluated our method and previous works in their corresponding subsection in Appendix Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, satiaaetsspppesswiti paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We created new benchmark to evaluate our method alongside previous scamning techniques in terms of generalization capabilities. We have made our benchmark publicly available here
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work didn't include crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
37

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "7O3lpnjdJq", "reviewer": "Reviewer_3HiF", "summary": "The paper introduces a novel trojan scanning method named TRODO (TROjan scanning by Detection of adversarial shifts in Out-of-distribution samples). TRODO leverages the concept of \"blind spots,\" where trojaned classifiers mistakenly identify out-of-distribution (OOD) samples as in-distribution (ID). The method scans for these blind spots by adversarially shifting OOD samples towards in-distribution, using the increased likelihood of these perturbed samples being classified as ID as a signature for trojan detection. TRODO is both trojan and label mapping agnostic, effective even against adversarially trained trojaned classifiers, and applicable even when training data is absent.", "questions": "1. Can you clarify your threat model?\n2. Why do you have so many missing baselines?", "limitations": "Design is too simple", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The writing is clear and the pictures are crisp and clear.", "weaknesses": "Undefined Threat Model: The threat model is not clearly defined, and the adversary's capabilities are not explicitly listed.\n\nLack of Comparison with SOTA Baselines: There is an absence of comparison with SOTA baselines, such as FreeEagle. The results listed in Table 2 (baseline-NC) differ significantly from those in FreeEagle's Table 4. Additionally, many baselines from FreeEagle's Table 4, such as STRIP[2] and ANP[3], are not discussed.\n\n[1]Fu, C., Zhang, X., Ji, S., Wang, T., Lin, P., Feng, Y., & Yin, J. (2023). {FreeEagle}: Detecting Complex Neural Trojans in {Data-Free} Cases. In 32nd USENIX Security Symposium (USENIX Security 23) (pp. 6399-6416).\n[2]Gao, Y., Xu, C., Wang, D., Chen, S., Ranasinghe, D. C., & Nepal, S. (2019, December). Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th annual computer security applications conference (pp. 113-125).\n[3]Wu, D., & Wang, Y. (2021). Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34, 16913-16925.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Our responses to each point are provided below:\n\n\n\n>**Q1&Undefined Threat Model**\n\nSorry for the confusion. We have briefly stated our threat model in lines 212-226 of the paper. We further present our threat model more clear in depth here. We assure the reviewer that we will improve the clarity of this section in our final manuscript.\n\n### **Attacker Capabilities**\n- **Data Poisoning and Training Influence:** Attackers can poison training data [1, 3] and influence the training process [2, 4] to embed backdoors into models.\n- **Trigger Visibility and Coverage:**\n  - **Stealthy to Overt Modifications:** Attackers can deploy triggers that range from undetectable to more noticeable modifications.\n  - **Local and Global Coverage:** Triggers can affect specific parts of a sample [1, 4] or the entire sample [5, 6].\n  - **Sample-Specific Attacks:** Attacks can be tailored to specific samples [7], complicating detection.\n- **Label-Consistent Mechanisms:** Attackers can use poisoned inputs labeled according to their visible content, which leads to misclassification during inference [8, 5].\n- **Attack Types:** Attacks can be either All-to-One or All-to-All. In the former, a single target class is selected and whenever the input contains the trigger, the classifier outputs the selected target class. In the latter however, there is more control over the attack. For each source class (the class to which the clean input actually belongs), an arbitrary target class can be chosen so that the presence of the trigger causes the model to classify the input as that target class.\n- **Model Training:** Models can be trained adversarially or non-adversarially.\n\n### **Attacker Goals**\n- **Embed Backdoors:** Ensure the model contains backdoors that lead to misclassification during inference.\n- **Maintain Stealthiness:** Create triggers that may be undetectable or hard to detect, even under scrutiny.\n- **Evade Detection:** Implement attacks that complicate detection efforts, especially those that are sample-specific or label-consistent.\n\n### **Defender Capabilities**\n- **Model-Only Detection:** The defender receives the model and may (TRODO) or may not (TRODO-Zero) have access to a small set of clean samples from the same distribution as the training data.\n- **No Prior Knowledge Required:** The detection mechanism operates without any prior knowledge of the specific type of attack or the nature of the trigger involved.\n\n### **Defender Goals**\n- **Detect Backdoors:** Identify if the model has been compromised with a backdoor.\n- **Adapt to Various Scenarios:** Effectively scan and detect backdoors both in scenarios with and without clean training samples.\n\n---\n\n>**Q2&Lack of Comparison with SOTA Baselines**\n\nWe have compared TRODO with eight strong Trojan detector methods in our paper. To further address your concerns regarding the comparison with other baselines, we have conducted additional experiments, including comparisons with FreeEagle, STRIP, ANP, Ex-Ray, and DF-TND. The results of these comparisons are presented in the following table:\n\n|**LabelMapping**|**Method**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|\n|||ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|***ACC/ACC\\****|\n|***All-to-One***|||||||||\n||DF-TND|56.6/54.8|49.1/49.7|60.2/57.5|49.6/45.5|47.8/42.2|***52.7/49.9***|\n||STRIP|63.3/71.0|68.4/64.4|71.2/61.2|59.4/53.5|57.2/55.6|***63.9/61.1***|\n||ANP|78.5/59.7|66.5/51.9|73.4/65.8|76.2/69.3|61.8/47.7|***71.3/58.9***|\n||Ex-Ray+ABS|66.6/46.5|63.3/51.4|76.9/56.5|60.5/46.4|60.5/50.2|***65.6/50.2***|\n||FreeEagle|80.2/72.9|82.0/73.2|81.0/82.3|73.2/66.9|65.0/66.0|***76.3/72.3***|\n||**TRODO-Zero**|80.9/79.3|82.7/78.5|84.8/83.3|75.5/73.7|73.2/70.6|***79.4/77.0***|\n|***All-to-All***|||||||||\n||DF-TND|23.8/20.7|28.9/26.7|30.9/26.6|13.8/10.1|15.2/12.5|***22.5/19.3***|\n||STRIP|33.1/28.2|26.5/24.3|22.8/21.1|33.8/29.9|29.4/27.6|***29.1/26.2***|\n||ANP|52.4/47.9|44.5/40.8|52.7/48.4|42.5/38.2|36.1/32.5|***45.6/41.6***|\n||Ex-Ray+ABS|47.4/45.4|48.3/44.7|52.6/50.9|39.3/35.7|38.2/33.3|***45.2/42.0***|\n||FreeEagle|79.8/75.2|54.9/50.2|55.2/52.9|56.5/52.7|48.0/46.1|***58.9/55.4***|\n||**TRODO-Zero**|82.1/80.8|80.4/77.3|83.8/88.6|74.8/72.3|75.0/75.4|***79.2/78.8***|\n\nAlthough FreeEagle aims to be effective against various types of trojan attacks and performs better than other baselines in this table, it is particularly vulnerable in All-to-All settings, where samples from each class can be mapped to different target classes in the presence of a trigger. FreeEagle primarily addresses the scenario where a single source class is mapped to a single target class and attempts to identify such pairs, if they exist. Thus, it can only perform well in All-to-One scenarios.\n\n---\n>**The results listed in Table 2 (baseline-NC) differ significantly ...**\n\nRegarding the results of NC on the TrojAI benchmark (Table 2 in our paper), it is important to note that our results are specific to the TrojAI benchmark, which is not covered in FreeEagle’s Table 4. We have included baseline results from K-arm's [9] Table 1, which is a baseline primarily focusing on this benchmark.\nWe will ensure these details and the additional comparative analysis are explicitly included in the revised version to enhance clarity and comprehensibility.\n\n\n[1] Gu et al. Badnets 2017\n\n[2] Nguyen et al. Wanet 2021\n\n[3] Chen et al. Targeted 2017\n\n[4] Nguyen et al. Input-aware 2020\n\n[5] Barni et al. New backdoor 2019\n\n[6] Wang et al. Bppattack 2022\n\n[7] Li et al. Invisible 2021\n\n[8] Turner et al. Label-consistent 2019\n\n[9] Shen et al. Backdoor scanning 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ZqUtUQSca", "reviewer": "Reviewer_eToF", "summary": "This paper points out a limitation of existing backdoor model scanning methods:\nThey fail to detect backdoored models trained with adversarial training. It\npropose a new backdoor model scanning method by utilizing adversarial shifts in\nOut-of-distribution samples. Experiments on MNIST, CIFAR-10, GTSRB, CIFAR-100,\nand Pubfig show the effectiveness of the proposed method.", "questions": "See Weaknesses.", "limitations": "The limitations have been discussed in this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The investigated problem is interesting.\n\n* The motivation of this paper is good.\n\n* This paper provides theoretical analysis to support the proposed observation\nand method.\n\n* The proposed scanning method is general to different types of backdoor attacks.", "weaknesses": "* The paper states that experiments were conducted using ResNet18, PreActResNet18,\nand ViT-B/16 models. However, Table 1 only presents results for ResNet18. While\nthe paper claims that results for other models are included in Appendix Section\nM, but this section also only contains ResNet18 results. It would be\nbeneficial to explicitly present the detection accuracy for each model across\nvarious attack scenarios.\n\n* The discussion about the adaptive attacks to the proposed method is missing,\nwhere the attacker knows about the proposed defense strategy and actively tries\nto evade or overcome it. For example, the adaptive attacker might able to add a\nloss during the backdoored model construction phase to reduce the change of the\nID-Score on the backdoored models.\n\n* While theoretical analysis of the proposed method is highly recommended, the\npaper would benefit from more intuitive explanations of this analysis. There\nappears to be a lack of detailed clarification regarding the connection between\nthe theoretical analysis and the proposed method. For instance, the relationship\nbetween Theorem 2 and the fundamental principles of the proposed method is not\nclearly stated. Providing more intuitive explanations of these theoretical\nanalysis would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful review. Here is our detailed response:\n\n>**W1:**\n\nWe sincerely apologize for this oversight. There were some issues with the command that removed these tables from our submitted paper. Here, we have provided those tables and assure you that they will be included in the final manuscript. We should note that in the following experiments, all components of TRODO are fixed except for the architectures of the classifiers.\n\n_Below are the detection results in terms of accuracy on standard trained evaluation sets (ACC %) and adversarially trained ones (ACC* %). Due to character limits, we have included only the top competitors from the methods considered in Table 1 of our paper._\n\n**PreAct ResNet-18 Architecture:**\n\n|**LabelMapping**|**Method**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|\n|||ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|***ACC/ACC\\****|\n|***All-to-One***|||||||||\n||K-ARM|69.9/54.3|68.2/52.0|72.0/62.8|58.3/48.6|61.3/48.6|***65.9/53.2***|\n||MM-BD|81.5/67.3|75.6/57.1|75.8/61.1|86.1/72.7|62.4/47.6|***76.3/61.2***|\n||UMD|79.5/58.8|76.4/51.5|79.2/64.7|67.5/54.6|64.5/47.0|***73.4/55.3***|\n||**TRODO-Zero**|85.0/78.5|81.2/79.1|85.2/83.9|78.2/78.9|73.6/72.4|***80.6/78.6***|\n||**TRODO**|92.6/88.7|90.5/90.2|93.4/90.1|85.6/83.2|80.2/78.8|***88.5/86.2***|\n|***All-to-All***|||||||||\n||K-ARM|56.8/49.7|54.6/47.6|57.5/48.9|51.3/45.0|50.6/47.3|***54.2/47.7***|\n||MM-BD|52.7/42.3|49.3/35.1|57.0/44.2|41.3/32.0|40.0/34.0|***48.1/37.5***|\n||UMD|80.7/61.5|75.5/55.9|83.5/64.9|67.7/50.0|66.7/47.5|***74.8/56.0***|\n||**TRODO-Zero**|83.9/76.8|77.2/78.4|87.5/83.7|78.4/76.8|76.0/70.3|***80.6/77.2***|\n||**TRODO**|90.8/87.9|88.3/89.8|92.0/87.9|82.6/82.1|81.0/76.6|***86.9/84.9***|\n\n**ViT-B-16 Architecture:**\n\n|**LabelMapping**|**Method**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|\n|||ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|***ACC/ACC\\****|\n|***All-to-One***|||||||||\n||K-ARM|69.8/54.3|68.2/52.0|72.0/62.8|58.3/48.6|61.3/48.6|***65.9/53.2***|\n||MM-BD|72.9/58.4|67.6/49.8|67.5/52.5|78.0/62.7|54.4/39.5|***68.1/52.6***|\n||UMD|75.2/55.0|69.0/45.4|71.5/59.6|62.5/48.6|58.3/40.7|***67.3/49.8***|\n||**TRODO-Zero**|78.5/71.8|72.9/71.5|76.5/76.1|71.6/70.2|68.9/65.1|***73.7/71.0***|\n||**TRODO**|87.9/85.6|82.5/84.4|85.2/84.3|80.3/79.6|78.2/76.1|***82.8/82.0***|\n|***All-to-All***|||||||||\n||K-ARM|54.9/43.3|50.5/43.1|51.5/47.6|46.4/39.9|49.4/41.9|***50.5/43.2***|\n||MM-BD|51.9/40.7|44.3/33.1|57.8/41.6|41.0/29.6|40.4/28.0|***47.1/34.6***|\n||UMD|73.9/56.4|69.4/48.6|77.7/58.6|58.8/43.4|58.0/40.1|***67.5/49.4***|\n||**TRODO-Zero**|80.2/76.0|70.4/73.3|81.6/77.0|74.2/69.8|71.7/65.8|***75.6/72.4***|\n||**TRODO**|87.6/82.3|82.6/84.3|83.5/83.0|79.8/77.3|76.0/73.1|***81.9/80.0***|\n\n-----\n> **W2:**\nThank you for suggesting this evaluation, and we apologize for missing it in our submitted version. To address the mentioned concern, we have provided different scenarios of adaptive attacks in our _common response_, which we kindly ask you to review.\n---\n> **W3:**\n\nWe acknowledge that the paper lacks an intuitive explanation connecting the theoretical analysis to the proposed method.\nThe theory section aims to provide high-level intuition for the method's core principle: _trojaned models are more susceptible to adversarial perturbations, especially in near-OOD regions_.\n\nAs deducible from our experiments, there is a clear performance gap between TRODO and TRODO-Zero, and the main reason for this gap is the utilization of near-OOD samples instead of arbitrary (mostly far-OOD) ones. We have also illustrated this phenomenon on Figure 2 of the paper. Using near-OOD samples enhances the change of ID-Score, resulting in more recognizable signature for trojan scanning. To further validate this intuition, we have provided Theorem 1, in which we prove that the adversarial risk on near-OOD data is more evident.\n\nIn Theorem 2, we analyzed a simplified scenario using a least-square loss and two-layer networks. Despite the simplicity, these cases are indicative of the general scenario because other complex losses used in practice yield similar optimization outcomes. Additionally, two-layer networks are known to be universal approximators capable of learning any function, analogous to more complex architectures like ResNet with MSE loss. This theoretical foundation helps to explain the observed phenomena in more intricate setups, thereby reinforcing the validity of our proposed method. Regarding the connection of this Theorem with our work, it is noteworthy that the core principle of TRODO is to use the difference of ID-Score of OOD samples before and after attack as the scanning signature. This change of ID-Score is equivalent to the adversarial risk which we have defined in Section 4. According to this theorem, this risk is linearly bounded by the trigger norm ($||t||$), which is non-zero for backdoored models and 0 for clean ones, making them distinguishable by our signature.\n\nIn future revisions, we will elaborate on this connection to provide readers with a clearer understanding of how our theoretical analysis supports and motivates the proposed method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dss70XLH32", "reviewer": "Reviewer_PnTa", "summary": "The paper proposes TRODO, a method for identifying trojaned classifiers, which relies on the intuition that in presence of a backdoor it should be easier than for clean classifiers to make the model classify an out-of-distribution input as in-distribution by adding an adversarial perturbation. In practice, a PGD attack is run on OOD images to increase the confidence of the classifier predictions, and the average increase is used to detect backdoored models. In the experimental evaluation, the proposed method outperforms the baselines, being effective against several backdoor attacks even when combined with adversarial training.", "questions": "- Why using a left-truncated normal distribution for estimating $\\tau$ when the score S can take values only on a specific range (confidence is upper bounded by 1)?\n\n- While the proposed method provides good experimental results, I think it's important to address the concerns about its robustness (see above).", "limitations": "Limitations are partially addressed.", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is very general, as it doesn't make assumptions on the type of backdoor attacks or target architectures. Moreover, the paper proposes variants with and without access to training data. Finally, the proposed approach appears less expensive than the baselines (Table 2).\n\n- The experimental results support the proposed method, which is effective against several backdoor attacks even when combined with adversarial training.", "weaknesses": "- The proposed method seems to have brittle elements:\n  - the PGD-10 attack used for computing the signature might be considered weak (only 10 iterations), then not able to fully optimize the target loss. Then, a stronger attack could further increase the confidence of the clean classifiers on OOD points, making them more similar to trojaned one (since the confidence is upper bounded, one could get the same score for all classifiers in the worst-case). Thus, the effectiveness of TRODO seems to rely to some extent on the attack not fully optimizing the target loss (which might be even exploited to bypass the detection mechanism).\n  - the confidence of any classifier might be adjusted post-training by e.g. temperature rescaling without changing its decisions. In this way, it seems possible to bypass the detection by making a model under-confident.\n  - adversarial training variants, e.g. on OOD data [60] or to have uniform predictions far from ID data [A], have been explored in prior works, and might be used to counter the proposed scanning scheme.\n\n- It's not clear how the effectiveness of TRODO correlates with the strength of the backdoor attacks. For example, one can imagine that using a lower poisoning rate might make the attack less detectable (but less effective).\n\n[A] https://arxiv.org/abs/1910.06259", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We have provided the following response:\n\n>**W1.1:**\n\nWe believe the epsilon value of the attack plays a key role compare to steps in our setup. If epsilon is large, as you mentioned, our signature for both clean and Trojan classifiers would be the same. This is the main reason we carefully estimate it using the validation set and Boundary Confidence Level to avoid such scenarios (see line 199). The number of steps in the attack has less effect as the attack radius has already been determined (Madry et al. Toward 2017). To further address your concern, we have provided an additional experiment where every component of our pipeline remains fixed except for the number of steps of the attack: \n\n\n\n|**Label Mapping**|**Method**|**n-step**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|-|\n||||ACC/ACC*|ACC/ACC*|ACC/ACC*|ACC/ACC*|ACC/ACC*|***ACC/ACC****|\n|**All-to-One**|||||||||\n||TRODO|PGD-10 **(default)**|91.2/89.6|91.0/88.4|96.6/93.2|86.7/82.5|88.1/83.0|***90.7/87.3***|\n|||PGD-100|91.0/89.0|90.2/87.6|95.9/92.7|86.0/81.9|87.6/82.1|***89.9/86.6***|\n|||PGD-1000|90.5/89.1|90.5/87.6|96.3/92.7|86.1/81.7|87.9/82.4|***90.5/86.4***|\n\n---\n\n\n> **W1.2:**\n\n\nIt is possible that temperature rescaling affects the regular softmax function, but the classifier remains more confident in in-distribution samples compared to OOD samples (Tajwar et al., No True 2021). This is consistent with the rationale and principle behind our method. As a result TRODO, would also detect trojaned classifiers with temperature scaling.  To further clarify this concern, we kindly ask you to refer to our _common response_, where we considered worst-case scenarios involving adaptive attackers. Even when the attacker is aware of our defense mechanism, TOROD demonstrated robust and consistent performance.\n\n\n---\n> **W1.3:** \n\nThe mentioned papers, CCAT (Stutz et al., 2020) [A] and RATIO (Augustin et al., 2020) [60], aimed to enhance robust performance on in-distribution samples. Improving robustness on adversarially out-of-distribution (OOD) samples was not their main purpose. Although they considered the issue of overconfidence on OOD samples, their primary goal was robust in-distribution classification. They do not weaken principle behind TRODO, which is based on the vulnerability of existing robust methods, including CCAT, RATIO, and other SOTA methods, to perturbed OOD samples, especially when they are close to the in-distribution, referred to as near OOD in our paper (see line 88 and Table 5). This observation has also been demonstrated by Fort (Adversarial, **2022**). Moreover, recently, RODEO (ICML, **2024**) has shown that in the CIFAR10 vs. CIFAR100 OOD detection challenge, there is no better performance than random detection (i.e. less than 50% AUROC), which further supports our claim (see their Table 4-a of our paper). Additionally, we kindly ask you to check our _common response_, where we show that even in the worst-case scenario (adaptive attack), TRODO demonstrates promising performance. \n\n_Finally, to further address your concern, we trained input trojan models using the method proposed by RATIO instead of common adversarial training and reported TRODO's performance on them here. Other components of TRODO, including the training pipeline, remain fixed._\n\n\n\n|Label Mapping|Method|MNIST|CIFAR10|GTSRB|CIFAR100|Pubfig|***Avg.***|\n|-|-|-|-|-|-|-|-|\n|||ACC/ACC*/ACC**|ACC/ACC*/ACC**|ACC/ACC*/ACC**|ACC/ACC*/ACC**|ACC/ACC*/ACC**|***ACC/ACC*/ACC\\*\\****|\n|All-to-One|||||||||\n||TRODO-Zero|80.9/79.3/78.4|82.7/78.5/79.1|84.8/83.3/82.8|75.5/73.7/74.0|73.2/70.6/70.2|***79.4/77.1/76.9***|\n||TRODO|91.2/89.6/89.4|91.0/88.4/88.7|96.6/93.2/92.4|86.7/82.5/81.5|88.1/83.0/83.4|***90.7/87.3/87.1***|\n\n\n \n\n\n---\n\n>**W2:**\n\nIntuitively, increasing the poisoning rate enlarges the blind spots in trojaned classifiers, as these are boundary regions where the poisoned data causes the model to overfit. Consequently, this will increase the probability that TRODO detects the trojaned classifiers. However, our signature is based on the presence of blind spots in trojaned classifiers and shows consistent performance across different poisoning rates. In the paper, we considered a poisoning rate of 10% as it is common in the literature. In response to your comment, we have provided TRODO's performance for different rates below: _(Other components of TRODO remained fixed.)_\n\n|**Label Mapping**|**Method**|**Poisoning-Rate**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|-|\n||||ACC/ACC*|ACC/ACC*|ACC/ACC*|ACC/ACC*|ACC/ACC*|***ACC/ACC\\****|\n|**All-to-One**|||||||||\n||TRODO-Zero|1%|80.1/78.2|81.3/77.0|83.5/81.8|74.6/72.7|72.1/69.8|***78.3/75.9***|\n|||3%|82.0/79.3|82.5/78.1|85.4/83.2|76.7/74.4|74.0/71.1|***80.1/77.2***|\n|||5%|83.6/80.4|84.0/79.5|86.3/84.6|77.5/75.3|75.1/72.6|***81.3/78.5***|\n|||10% **(default)**|80.9/79.3|82.7/78.5|84.8/83.3|75.5/73.7|73.2/70.6|***79.4/77.0***|\n||TRODO|1%|89.5/87.4|88.7/85.6|94.9/91.0|84.5/81.2|86.4/80.3|***88.8/85.1***|\n|||3%|91.0/89.2|90.5/87.8|96.5/93.1|86.3/82.7|87.8/82.4|***90.4/87.0***|\n|||5%|92.8/90.6|91.7/88.9|97.0/94.1|87.5/83.6|89.1/84.5|***91.6/88.3***|\n|||10% **(default)**|91.2/89.6|91.0/88.4|96.6/93.2|86.7/82.5|88.1/83.0|***90.7/87.3***|\n\n\n----\n\n> **Q1:** \n\nThank you for your insightful question. You are correct that the score $S_i$ can take values only within a specific range, as confidence is upper bounded by $1$. We apologize for the oversight in the paper where we did not mention that we apply a transformation to the score $S_i$.\nIn our method, we use $-log(1 - S_i)$ instead of $S_i$ directly. This transformation maps the original score to a new range $[0, ∞)$, making it suitable for fitting with a left-truncated normal distribution.\nWe will include this clarification in the final manuscript to ensure there is no confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Eqh0SXSupO", "reviewer": "Reviewer_T6nS", "summary": "This paper proposes a general strategy for distinguishing between trojaned and clean models. The generality of the approach lies in its applicability to various types of trojan attacks, different label-mapping strategies, and its ability to work with and without clean training data. The authors claim that distorted areas of the learned decision boundary in trojaned models, referred to as blind spots, serve as a consistent signature for distinguishing between trojaned and clean classifiers, regardless of the trojan attack methodology. A key characteristic of these blind spots is that samples within these regions are expected to be out-of-distribution (OOD) relative to the clean training data, yet trojaned classifiers mistakenly identify them as in-distribution (ID) samples. To leverage this characteristic, the authors propose using adversarial attacks to perturb a given OOD sample towards an ID direction, followed by computing the difference in ID score (i.e., maximum softmax probability) before and after the adversarial attack. They found that trojaned models exhibit significantly larger differences in these scores. The effectiveness of the proposed detection method was empirically validated against eight trojan attacks and under different levels of access to clean training data.", "questions": "1. Could you clarify what it means for a trojaned classifier to be adversarially trained on poisoned training data? (For example, for poisoning samples generated by a label-consistent trojan attack, what does it mean by adversarially training the trojaned model on these data?) Additionally, are there any empirical results demonstrating the effectiveness of the proposed method in this specific scenario?\n2. When a portion of clean training data is not accessible, the authors propose to use Tiny-ImageNet. In addition, it comes to my mind that given the accessibility to the model, one could actually reverse engineer the training data. In that way, you could obtain a portion of fake clean training data, followed by creating near-OOD samples.Consequently, the performance without clean training data could approach that of having a portion of real clean training data, thereby diminishing the performance gap between TRODO-Zero and TRODO.\nIn particular, the reverse engineering of training data has been adopted in many backdoor works, such as [1] (see Figure 3).\n3. Headings of paragraphs could be set in a uniform format. Eg., In Line 149, there is a ‘.’ after the paragraph heading; by comparison, in Line 168, there is not.\n\n[1] Liu, Y., Ma, S., Aafer, Y., Lee, W. C., Zhai, J., Wang, W., & Zhang, X. (2017). Trojaning attack on neural networks.", "limitations": "The performance of the proposed method drops by a large margin (over 10%) when no clean training data is accessible, compared to that with a portion of clean training data.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and easy to read.\n2. It includes clear explanations, supported by figures, to illustrate the intuition and proposed method.\n3. The authors conduct extensive experiments across various types of trojan attacks and different levels of access to clean training data.\n4. They propose a simple yet effective method to detect trojaned models.\n5. Theoretical analysis is provided, demonstrating that a trojaned neural network is more sensitive to adversarial perturbations.", "weaknesses": "As mentioned in the limitation section, the selections of $\\epsilon$ and $\\tau$ rely on a validation set and a surrogate model. As a result, the quality of these hyperparameters depend on the choices of the validation set and the surrogate model. In particular, when detecting a trojaned model with a new model architecture or trained on a new domain, one might need to tune $\\epsilon$ and $\\tau$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback on our paper. We greatly appreciate your insights and suggestions.\n\n> **W1:**\n\nWe acknowledge that extracting hyperparameters may be considered a limitation of our method, as previously discussed in our limitations section. However, it is important to note that these hyperparameters must be selected based on the architecture where the validation is fixed. Moreover, our method demonstrates robustness with respect to the validation set, as evidenced in Table 3 of the Ablation Study. Specifically, using small datasets like FMNIST or SVHN for validation, our method outperforms others by up to 10%.\n\n---\n\n> **Q1:**\n\nWe sincerely apologize for any confusion caused. To clarify, the training dataset is initially compromised using a backdoor attack. The classifier is then adversarially trained using the PGD (Projected Gradient Descent) attack method. As mentioned in the caption of Table 1, all columns labeled ACC* pertain to adversarially trained models. Additionally, lines 287-290 states: \"Specifically, TRODO achieves superior performance with an 11.4% improvement in scenarios where trojan classifiers have been trained in a standard (non-adversarial) setting and a 24.8% improvement in scenarios where trojan classifiers have been adversarially trained,\" indicating a significant improvement in adversarially trained models. Furthermore, Table 2 includes columns for rounds 3, 4, and 11 of TrojAI, which involve adversarially trained models.\n\nTable 1: Scanning performance of TRODO compared with other methods, in terms of Accuracy on\nstandard trained evaluation sets (ACC %) and adversarially trained ones (ACC* %).\n\n|**Label Mapping**|**Method**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|\n|||ACC/ACC*|ACC/ACC*|ACC/ACC*|ACC/ACC*|ACC/ACC*|ACC/ACC*|\n|**All-to-One**|||||||||\n||NC|54.3/49.8|53.2/48.4|62.8/56.3|52.1/42.1|52.5/40.2|55.0/49.4|\n||ABS|67.5/69.0|64.1/65.6|71.2/65.5|56.4/54.2|56.3/58.3|63.1/62.5|\n||PT-RED|51.0/48.8|50.4/46.1|58.4/57.5|50.9/45.3|49.1/47.9|52.0/49.1|\n||TABOR|60.5/45.0|56.3/44.7|69.0/53.8|56.7/45.5|58.6/44.2|60.2/46.6|\n||K-ARM|68.4/55.1|66.7/54.8|70.1/62.8|59.8/50.9|60.2/47.6|65.0/54.2|\n||MNTD|57.4/51.3|56.9/52.3|65.2/55.9|54.4/48.8|56.7/50.0|58.1/54.7|\n||MM-BD|85.2/65.4|77.3/57.8|79.6/65.2|88.5/74.0|65.7/48.3|79.3/62.1|\n||UMD|81.1/61.2|77.5/54.7|81.4/68.2|69.0/56.3|67.9/49.7|75.4/58.0|\n||**TRODO-Zero**|80.9/79.3|82.7/78.5|84.8/83.3|75.5/73.7|73.2/70.6|79.4/77.0|\n||**TRODO**|91.2/89.6|91.0/88.4|96.6/93.2|86.7/82.5|88.1/83.0|***90.7/87.3***|\n|**All-to-All**|||||||||\n||NC|26.7/21.6|24.9/19.6|31.6/23.2|15.4/11.8|16.8/12.3|23.1/17.7|\n||ABS|32.5/34.1|30.7/28.8|23.6/20.5|34.3/34.8|31.0/28.2|30.4/29.3|\n||PT-RED|41.0/33.5|39.6/33.1|45.4/43.9|20.3/15.2|12.6/9.8|31.8/27.1|\n||TABOR|51.7/39.7|50.2/37.8|48.3/39.5|39.4/30.2|38.6/30.8|45.6/35.6|\n||K-ARM|56.8/49.7|54.6/47.6|57.5/48.9|51.3/45.0|50.6/47.3|54.2/47.7|\n||MNTD|27.2/25.2|23.0/18.6|16.9/12.8|29.8/31.0|22.3/17.9|23.8/21.1|\n||MM-BD|54.3/40.4|49.4/35.1|57.9/44.0|40.7/32.3|41.2/34.1|48.7/37.2|\n||UMD|82.5/61.9|74.6/60.1|84.2/64.5|70.6/49.9|68.7/52.3|76.1/57.7|\n||**TRODO-Zero**|82.1/80.8|80.4/77.3|83.8/88.6|74.8/72.3|75.0/75.4|79.2/78.8|\n||**TRODO**|90.0/87.4|89.3/87.5|92.6/89.1|82.4/85.0|83.2/80.9|***87.5/86.1***|\n\n\nTable 2: Comparison of TRODO and other methods on all released rounds of TrojAI benchmark on\nimage classification task. For each method, we reported scanning Accuracy and the average time\nscanning time for a given classifier.\n|**Method**|**Round0 Accuracy/Time(s)**|**Round1 Accuracy/Time(s)**|**Round2 Accuracy/Time(s)**|**Round3 Accuracy/Time(s)**|**Round4 Accuracy/Time(s)**|**Round11 Accuracy/Time(s)**|\n|-|-|-|-|-|-|-|\n|NC|75.1/574.1|72.2/592.6|-/>23000|-/>23000|-/>20000|N/A/N/A|\n|ABS|70.3/481.9|66.8/492.5|62.0/1378.4|70.8/1271.4|76.3/443.2|N/A/N/A|\n|PT-RED|85.0/941.6|84.3/962.7|58.2/>23000|65.7/>25000|66.1/>28000|N/A/N/A|\n|TABOR|82.8/974.2|80.3/992.5|56.2/>29000|60.8/>27000|58.3/>32000|N/A/N/A|\n|K-ARM|91.3/262.1|90.0/283.7|_76.0_/1742.8|**79.0**/1634.1|_82.0_/1581.4|N/A/N/A|\n|MM-BD|68.8/_226.4_|73.2/_231.3_|55.8/_174.3_|52.6/_182.6_|54.1/_178.1_|_51.3_/_1214.2_|\n|UMD|80.4/>34000|79.2/>34000|75.2/>18000|61.3/>19000|56.9/>90000|N/A/N/A|\n|**TRODO**|_86.2_/152.4|_85.7_/194.3|78.1/107.2|_77.2_/122.4|82.8/117.8|**61.3**/**984.3**|\n\n-----\n\n> **Q2:**\n\nWe thank the reviewer for the suggestion. We should note that access to a portion of the training data would enable us to create near-OOD samples with higher quality compared to similar methods. Specifically, the mentioned method's strategy for exploiting data from a classifier leads to samples with artifacts and shortcuts, reducing its effectiveness as it becomes more apparent to the model that they do not belong to ID (considering them far OOD). However, to further explore your suggestion, we evaluated both the proposed method and the FastDFKD (Fang Data-free 2022) method for crafting data in situations where there is no access to training data:\n\n|**LabelMapping**|**Method**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|\n||||ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|***ACC/ACC\\****|\n|All-to-One|TrojaNN|70.2/68.0|72.3/70.1|73.1/71.3|65.5/63.1|63.9/60.8|***69.0/66.7***|\n||FastDFKD|77.4/75.0|79.2/75.5|80.4/78.0|72.9/70.1|71.2/68.0|***76.2/73.3***|\n||**TRODO**|91.2/89.6|91.0/88.4|96.6/93.2|86.7/82.5|88.1/83.0|***90.7/87.3***|\n\n---\n\n\n> **Q3:**\n\nThank you for pointing out the formatting inconsistency. We apologize for this oversight and will ensure that all paragraph headings are uniformly formatted in the final manuscript.\n\n---\n\n> **L1:**\n\nWe acknowledge that the limitation you mentioned is valid. However, the scenario of having no clean training data is a special case. Most research studies, including ours, typically utilize at least a small amount of clean data to validate their methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iOTZ2h9hTe", "reviewer": "Reviewer_CctU", "summary": "The authors propose a trojan scanning technique that leverages the sensitivity of the network's confidence when near-OOD samples undergo an adversarial attack. The authors argue that the greater variation in confidence can be used to discriminate whether a network has been backdoored, and present extensive experiments to support the effectiveness of their method.", "questions": "- Q1: There must be a typo at line 161, I could not find information about hard transformations and therefore cannot verify claims made about them (Appendix Section 5).   The typo \"Attention Section\" occurs repeatedly throughout the paper. \n. Q2: When using TinyImageNet, why did the authors not simply filter out the classes overlapping with CIFAR-10? This can be easily done and show the effectiveness of the method without applying G(.).", "limitations": "- L1: The method seems to assume to know the training distribution of the model so that it is possible to find near-OOD samples. However this may not always be possible or easy (near-ness could depend on many factors other than class similarity, e.g. may depend on resolution of the input, color range, other forms of naturally occurring covariate-shifts, types of shortcuts taken by the models etc.). The method significantly improves when this info is accessible. \n- L2: The previous limitation spills over to the need of having a good validation set. In some cases, the performance can go below the baselines if the validation set is not representative of the training set.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The authors compare with a large number of baselines and a wide variety of datasets and trojan attacks, with standard and adversarial training on 3 architectures. \n- The concept is intuitive and the method seems to be empirically effective in a wide variety of cases.", "weaknesses": "- W0.1 (On writing) The concepts are very simple, but the presentation is hard to follow. Lines 40-89 refer to figures and concepts that will be displayed in following sections. I would suggest to anticipate significantly the figures. Furthermore, I would suggest providing immediately clear definitions of concepts, that are instead introduced in a handwavy way (e.g., blind spots, benign overfitting) before being properly discussed in related works. Probably completely restructuring Sections 1 and 2 to transfer most of the contents of 1 to 2 could help, having a good 'definitions' or 'preliminaries' section in Section 2. \n- W0.2 (On writing) The formatting of the theorems in the main paper (especially with lack of assumptions) is very unusual and may e confusing. \n- W1: The scanning procedure may be vulnerable to an adaptive trojan attack. Could the authors show what happens if attackers account for the author's defense in their attack strategy? How much does the effectiveness of their technique go down? The attackers may completely elude the proposed scanning method if they account for it in the design of their attack.\n- W2:  How sensitive are the hyperparameters to the choice of the validation set? How computationally expensive is it to tune them? \n- W3: Results are adequately reported exclusively for Resnet18, and are not reported exhaustively for other architectures (or their presentation is not really immediately clear).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your useful comments. Please find our responses below:\n\n\n>**W0.1&W0.2:**\n\nWe appreciate your suggestions and will implement them to improve clarity and logical flow in the final manuscript based on your recommendations.\n\n----\n\n>**W1:**\n\nWe conducted additional experiments to establish an adaptive Trojan attack. We have detailed these in our _common response_, which we kindly ask you to review.\n\n---\n\n> **W2:**\n\nAs stated in limitation section of our paper, for each new architecture, we have to firstly tune $\\epsilon$ and then $\\tau$. For each selection of architecture and validation dataset, first we have to train a surrogate classifier $g$ on the selected validation set and then find $\\epsilon$ using DeepFool [1]. As the final stage for computing $\\tau$ is not time consuming, we only report the computational cost of the first two stages of hyperparameters tuning in the tables below (the values are in terms of hours):\n\n||ResNet18|PreActResNet18|ViT-B/16|\n|-|-|-|-|\n|**Training surrogate classifier $g$**|3|5|10|\n|**DeepFool for $\\epsilon$**|0.3|0.7|2|\n\n* The experiments have been conducted on a RTX 3090.\n\nRegarding the sensitivity of TRODO to the validation set and hyperparameters, we have provided the values of hyperparameters for various selections of architecture and validation dataset:\n\n**ResNet18**\n||ϵ|τ|\n|-|-|-|\n|**FMNIST**|0.0491|1.1625|\n|**SVHN**|0.0476|1.1338|\n|**STL-10**|0.0488|1.1571|\n|**TinyImageNet**|0.0483|1.1523|\n\n**PreActResNet18**\n||ϵ|τ|\n|-|-|-|\n|**FMNIST**|0.0538|1.0407|\n|**SVHN**|0.0524|1.0025|\n|**STL-10**|0.0530|1.0462|\n|**TinyImageNet**|0.0527|1.0179|\n\n**ViT-B/16**\n||ϵ|τ|\n|-|-|-|\n|**FMNIST**|0.0621|0.9341|\n|**SVHN**|0.0598|0.9106|\n|**STL-10**|0.0611|0.9246|\n|**TinyImageNet**|0.0609|0.9150|\n\nAdditionally, the performance results of our method using these hyperparameters can be found in Table 3 in the Ablation Study (Section 6) of the paper. This section demonstrates the robustness and effectiveness of our method across different datasets in the validation set. \n\n---\n\n> **W3:**\n\nWe apologize for the oversight and any confusion it may have caused. It seems there was an issue that resulted in the exclusion of results for PreActResNet18 and ViT-B/16 models from Appendix Section M. We will ensure that these results are included in the final updated version of the paper. Below are the detection results in terms of accuracy on standard trained evaluation sets (ACC %) and adversarially trained ones (ACC* %).  \n\n**PreAct ResNet-18 Architecture:**\n\n|**LabelMapping**|**Method**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|\n|||ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|***ACC/ACC\\****|\n|***All-to-One***|||||||||\n||K-ARM|69.9/54.3|68.2/52.0|72.0/62.8|58.3/48.6|61.3/48.6|***65.9/53.2***|\n||MM-BD|81.5/67.3|75.6/57.1|75.8/61.1|86.1/72.7|62.4/47.6|***76.3/61.2***|\n||UMD|79.5/58.8|76.4/51.5|79.2/64.7|67.5/54.6|64.5/47.0|***73.4/55.3***|\n||**TRODO-Zero**|85.0/78.5|81.2/79.1|85.2/83.9|78.2/78.9|73.6/72.4|***80.6/78.6***|\n||**TRODO**|92.6/88.7|90.5/90.2|93.4/90.1|85.6/83.2|80.2/78.8|***88.5/86.2***|\n|***All-to-All***|||||||||\n||K-ARM|56.8/49.7|54.6/47.6|57.5/48.9|51.3/45.0|50.6/47.3|***54.2/47.7***|\n||MM-BD|52.7/42.3|49.3/35.1|57.0/44.2|41.3/32.0|40.0/34.0|***48.1/37.5***|\n||UMD|80.7/61.5|75.5/55.9|83.5/64.9|67.7/50.0|66.7/47.5|***74.8/56.0***|\n||**TRODO-Zero**|83.9/76.8|77.2/78.4|87.5/83.7|78.4/76.8|76.0/70.3|***80.6/77.2***|\n||**TRODO**|90.8/87.9|88.3/89.8|92.0/87.9|82.6/82.1|81.0/76.6|***86.9/84.9***|\n\n**ViT-B-16 Architecture:**\n\n|**LabelMapping**|**Method**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|\n|||ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|***ACC/ACC\\****|\n|***All-to-One***|||||||||\n||K-ARM|69.8/54.3|68.2/52.0|72.0/62.8|58.3/48.6|61.3/48.6|***65.9/53.2***|\n||MM-BD|72.9/58.4|67.6/49.8|67.5/52.5|78.0/62.7|54.4/39.5|***68.1/52.6***|\n||UMD|75.2/55.0|69.0/45.4|71.5/59.6|62.5/48.6|58.3/40.7|***67.3/49.8***|\n||**TRODO-Zero**|78.5/71.8|72.9/71.5|76.5/76.1|71.6/70.2|68.9/65.1|***73.7/71.0***|\n||**TRODO**|87.9/85.6|82.5/84.4|85.2/84.3|80.3/79.6|78.2/76.1|***82.8/82.0***|\n|***All-to-All***|||||||||\n||K-ARM|54.9/43.3|50.5/43.1|51.5/47.6|46.4/39.9|49.4/41.9|***50.5/43.2***|\n||MM-BD|51.9/40.7|44.3/33.1|57.8/41.6|41.0/29.6|40.4/28.0|***47.1/34.6***|\n||UMD|73.9/56.4|69.4/48.6|77.7/58.6|58.8/43.4|58.0/40.1|***67.5/49.4***|\n||**TRODO-Zero**|80.2/76.0|70.4/73.3|81.6/77.0|74.2/69.8|71.7/65.8|***75.6/72.4***|\n||**TRODO**|87.6/82.3|82.6/84.3|83.5/83.0|79.8/77.3|76.0/73.1|***81.9/80.0***|\n\n---\n\n> **Q1:**\n\nWe apologize for the missing details about hard augmentations/transformations and kindly ask you to review our common response, where we have provided detailed answers to this question.\n\n-----\n\n>**Q2:**\n\nWe appreciate your insightful question. Our primary objective is to develop a general method with a consistent pipeline regardless of the training data distribution. We aimed to avoid making dataset-specific modifications, such as filtering out overlapping classes for CIFAR-10 but not for others, such as GTSRB. By not removing common classes, we ensure that our approach remains uniform and applicable to various datasets without additional preprocessing steps.\n\n\n-----\n\n>**L1&L2:**\n\nAlthough our method can operate without access to any training data, we confirm that having access to such data from the input classifier can improve our detection performance. However, it should be noted that all existing detection methods, except for MM-BD, strongly rely on training data for operation. On the other hand, our method works in scenarios where training data is unavailable by just leveraging a small validation set. As shown in Table 3, by using only the FMNIST or SVHN dataset, we achieve superior performance, outperforming other methods by up to 10%.\n\n[1] Moosavi-Dezfooli et al. DeepFool 2015", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DSRwfqgcuj", "reviewer": "Reviewer_d7KJ", "summary": "The paper addresses the problem of detecting trojaned models. The paper proposes a trojaned model scanning method using out-of-distribution (OOD) samples. Specifically, it is observed that trojaned classifiers can erroneously identify adversarially attacked OOD samples as in-distribution (ID)  samples. Therefore, the increased likelihood of perturbed OOD samples being classified as ID serves as a signature for trojan detection. The proposed trojan detection method can be applied in two scenarios: avaliability of clean samples and non-avaliability of clean ID samples. Extensive experiments demonstrate that proposed can achieve state-of-the-art performance compared with other scanning trojan detection methods.", "questions": "Could you provide some details of used transformations? \n\nCould you compute FID between transformed samples and original samples?\n\nThe paper only compares with scanning methods. How about other kinds of state-of-the-art trojan detection methods?", "limitations": "The limitations have been addressed.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper proposes a trojan detection method using out-of-distribution samples. This can be applied when no training in-distribution samples is available, which is effective in the real-word applications.\n\nThe paper conducts sufficient experiments to demonstrate the effectiveness of proposed method. The trojan detection experiments are conducted compared with 7 state-of-the-art scanning methods on 5 datasets across various architectures including CNN and Transformer.\n\nThe authors present both empirical results and theoretical analysis of proposed method. This is convincing.", "weaknesses": "The motivation (especially in Introduction Section) of proposed method is not presented clearly. For example, in line 73-74, the paper introduces the concept of near-OOD samples. However, the difference of far-OOD and near-OOD samples is not clearly explained. Also, the paper writes that \"see the visual demonstration in Appendix Section 5\". It is obvious there is no Section 5 in Appendix.\n\nThe paper claims that current methods can not identify models trojaned using adversarial training (e.g. in line 6-7 and line 112-115). However, there is no detailed experiments to show the difference between detecting models trojaned using adversarial training and normal training.\n\nSome ablations can be done. For example, the number of used out-of-distribution samples, and the diversity of used OOD samples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments. Responses to specific points are provided below:\n\n> **W1:**\n\n* We apologize for the referencing error. We intended to refer to Figure5 in SectionE.\n\n* We mentioned that near-OODs are those that share semanti/stylistic features with the IDs making them harder to distinguish (e.g., CIFAR10 vs. CIFAR100). On the other hand, far-OODs do not share any semantic or stylistic similarity with IDs (e.g., CIFAR10 vs. MNIST), making them easier to identify as OODs. For a more formal definition this Concepts, we kindly refer the reviewer to [1], as this is not the primary focus of our study; instead, we leverage these concepts for detecting trojaned classifiers.\n\n* The main motivation behind TRODO is to define and find a general and robust signature that distinguishes trojaned and clean classifiers, independent of architecture, label mapping strategy in trojaning, and the training strategy of the classifier (clean/adversarial).\n---\n> **W2:**\n \nAs stated in the caption of Table 1, all ACC* columns are related to adversarially trained models. Additionally, lines 287-290 mention, \"Specifically, TRODO achieves superior performance with an 11.4% improvement in scenarios where trojan classifiers have been trained in a standard (non-adversarial) setting and a 24.8% improvement in scenarios where trojan classifiers have been adversarially trained,\" indicating a greater improvement on adversarially trained models. Moreover, Table 2 in our paper includes columns for rounds 3, 4, and 11 of TrojAI, which contain adversarially trained models.\n\nTable 1: Performance of TRODO compared with other methods, in terms of Accuracy on\nstandard trained evaluation sets (ACC%) and adversarially trained ones (ACC*%).\n\n|**LabelMapping**|**Method**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|\n|||ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|***ACC/ACC\\****|\n|**All-to-One**|NC|54.3/49.8|53.2/48.4|62.8/56.3|52.1/42.1|52.5/40.2|***55.0/49.4***|\n||ABS|67.5/69.0|64.1/65.6|71.2/65.5|56.4/54.2|56.3/58.3|***63.1/62.5***|\n||PT-RED|51.0/48.8|50.4/46.1|58.4/57.5|50.9/45.3|49.1/47.9|***52.0/49.1***|\n||TABOR|60.5/45.0|56.3/44.7|69.0/53.8|56.7/45.5|58.6/44.2|***60.2/46.6***|\n||K-ARM|68.4/55.1|66.7/54.8|70.1/62.8|59.8/50.9|60.2/47.6|***65.0/54.2***|\n||MNTD|57.4/51.3|56.9/52.3|65.2/55.9|54.4/48.8|56.7/50.0|***58.1/54.7***|\n||MM-BD|85.2/65.4|77.3/57.8|79.6/65.2|88.5/74.0|65.7/48.3|***79.3/62.1***|\n||UMD|81.1/61.2|77.5/54.7|81.4/68.2|69.0/56.3|67.9/49.7|***75.4/58.0***|\n||**TRODO-Zero**|80.9/79.3|82.7/78.5|84.8/83.3|75.5/73.7|73.2/70.6|***79.4/77.0***|\n||**TRODO**|91.2/89.6|91.0/88.4|96.6/93.2|86.7/82.5|88.1/83.0|***90.7**/**87.3***|\n|**All-to-All**|NC|26.7/21.6|24.9/19.6|31.6/23.2|15.4/11.8|16.8/12.3|***23.1/17.7***|\n||ABS|32.5/34.1|30.7/28.8|23.6/20.5|34.3/34.8|31.0/28.2|***30.4/29.3***|\n||PT-RED|41.0/33.5|39.6/33.1|45.4/43.9|20.3/15.2|12.6/9.8|***31.8/27.1***|\n||TABOR|51.7/39.7|50.2/37.8|48.3/39.5|39.4/30.2|38.6/30.8|***45.6/35.6***|\n||K-ARM|56.8/49.7|54.6/47.6|57.5/48.9|51.3/45.0|50.6/47.3|***54.2/47.7***|\n||MNTD|27.2/25.2|23.0/18.6|16.9/12.8|29.8/31.0|22.3/17.9|***23.8/21.1***|\n||MM-BD|54.3/40.4|49.4/35.1|57.9/44.0|40.7/32.3|41.2/34.1|***48.7/37.2***|\n||UMD|82.5/61.9|74.6/60.1|84.2/64.5|70.6/49.9|68.7/52.3|***76.1/57.7***|\n||**TRODO-Zero**|82.1/80.8|80.4/77.3|83.8/88.6|74.8/72.3|75.0/75.4|***79.2/78.8***|\n||**TRODO**|90.0/87.4|89.3/87.5|92.6/89.1|82.4/85.0|83.2/80.9|***87.5/86.1***|\n---\n> **W3:**\n\nWe aimed to provide similar ablations in Table 3 of the paper. However, to address your concerns, we conducted additional experiments.\n\nIn the first experiment, we performed an ablation study on the number of samples in our validation set. By default, TRODO Zero uses the Tiny ImageNet validation dataset, which contains 200 classes, each with 500 samples. We explored the effect of varying the number of samples per class and investigated the ratio [0.0, 1.0].\n\n|**LabelMapping**|**Method**|**OOD-Sample-Rate**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|\n|-|-|-|-|-|-|-|-|-|\n||||ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|ACC/ACC\\*|***ACC/ACC\\****|\n|All-to-One|TRODO-Zero|0.1%|78.6/77.2|80.5/76.3|82.6/81.1|73.2/71.4|71.2/68.5|***77.3/75.0***|\n|||0.2%|79.3/77.9|81.2/77.1|83.3/81.8|73.9/72.1|71.9/69.2|***78.1/75.7***|\n|||0.3%|80.0/78.6|81.9/77.8|84.0/82.5|74.6/72.8|72.6/69.9|***78.8/76.4***|\n|||0.5%|80.6/79.2|82.5/78.4|84.6/83.1|75.2/73.4|73.1/70.5|***79.3/77.0***|\n|||1% **(default)**|80.9/79.3|82.7/78.5|84.8/83.3|75.5/73.7|73.2/70.6|***79.5/77.1***|\n\nHere, we utilized the DC metric [2] to measure the diversity based on your suggestion. A higher value indicates greater density and coverage of the validation set with respect to the evaluation set:\n\n|Validation|MNIST|CIFAR10|GTSRB|CIFAR100|PubFig|**DC Average**|\n|-|-|-|-|-|-|-|\n|FMNIST|0.45|0.41|0.52|0.54|0.47|0.48|\n|SVHN|0.50|0.53|0.60|0.59|0.57|0.56|\n|STL-10|0.55|0.64|0.63|0.65|0.62|0.62|\n|TinyImageNet(default)|0.60|0.75|0.70|0.78|0.80|0.73|\n\n \n---\n\n> **Q1:** \n\nWe apologize for the missing details about hard augmentations/transformations and kindly ask you to review our common response, where we have provided detailed answers to this question.\n\n---\n\n> **Q2:**\n\nHere  for each validation set, we crafted the OOD samples with hard transformations and then computed their distance using the FID metric:\n\n  FMNIST|SVHN|STL-10|TinyImageNet|PubFig|\n|-|-|-|-|-|\n|43|68|59|84|75|\n\n---\n\n> **Q3:**\n\nWe apologize for any ambiguity/inconvenience caused by using different terms. However, we would like to clarify that our method is a post-training defense approach aimed at detecting whether an input model/classifier has been trojaned (backdoored). After reviewing the literature, we understand that researchers refer to this as both 'trojan detection methods' and 'scanning methods,' as the task involves scanning an input model. \n \n\n[1]  Winkens Contrastive 2021\n\n[2] Naeem Reliable  2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
One of the common concerns raised by the reviewers was TRODO's performance against adaptive attacks. To address this concern, we conducted additional experiments and present the results here. Also there were concerns on the details of hard transformations, which we have also addressed in the following. We assure the reviewers that we will discuss these in our final manuscript and include the provided experiments.

>**TRODO under Adaptive Attack:**

We define two types of adaptive attacks targeting TRODO, which we will describe below:

TRODO operates on the implicit assumption that a classifier is more confident on in-distribution (ID) samples than on out-of-distribution (OOD) samples, as shown in the OOD detection literature. The maximum softmax probability (MSP) assigned by the classifier to an input sample is considered as its confidence level, referred to as the ID-Score in our paper.

An adaptive attacker knowing our defense mechanism and aiming to undermine the core principle of TRODO, we believe training a classifier with a new loss function that equalizes the ID-Score for both ID and OOD samples would be an adversarial approach to diminish our defense. As a result, we propose the following loss function for training a $k$-class classifier $f$:

$L_{adaptive1}$=$E_{(x, y) ∼ D_{\text{in}}}$  [ $- \log f_{y}(x) $]  $- λ_1 E_{(x, y) ∼ D_{\text{out}}}$  [ $H(U; f(x))  $] $+ λ_2 E_{(x, y) ∼ D_{\text{in}}}$  [ $H(U; f(x))  $]

Here, $x, y$ are data samples and their labels, $f_{y}(x)$ denotes $y$-th output of the classifier, $U$ is the uniform distribution over $k$ classes, and $H$ is the cross-entropy. The first term of $L_{adaptive1}$ is the classification term (cross-entropy), while the other terms force the classifier to decrease MSP (ID-Score) for ID samples while increasing it for OOD samples. We set $λ_1=λ_2 = 0.5$, inspired by [1], to balance the importance of the first term. By this loss function, we hope the ID-Score for both OOD and ID samples will change while the classifier's decisions for them remain fixed.

Moreover, we define another loss function for adaptive attacking. Our study relies on the difference in ID-Score between a benign OOD sample and a perturbed OOD sample as the scanning signature, where a higher signature indicates that the classifier is trojaned. In this adaptive attack, we explicitly target this by defining a new loss function that aims to decrease the signature for trojan classifiers, making it harder for our defense to distinguish them from clean ones:

$L_{adaptive2}$=$E_{(x, y) ∼ D_{\text{in}}}$  [ $- \log f_{y}(x) $]  $ -  λ_3 E_{(x, y) ∼ D_{\text{out}}}$  [ $H(f(x); f(x^*))  $]

Here, $x^*$ denotes the adversarially perturbed sample.

We recreated our benchmark evaluation set, including both clean and trojaned classifiers, and trained trojan models using $L_{adaptive1}$ and $L_{adaptive2} $ and examined TRODO's performance in distinguishing them.

We should note that in $L_{adaptive1}$ although the second and third terms act as regularizers to disrupt our core principle, the trojaned classifier still assigns relatively high confidence to ID samples as it is forced to classify IDs properly. Moreover, the designed loss functions $L_{adaptive1}$ and $L_{adaptive2}$ make the classifiers overfit to utilized OODs, as there is no pattern behind that loss function for to be learned by the model. Furthermore, we use a random process to create OODs in TRODO (selecting random hard transformations to obtain $G(.)$ ). This offers some remedy against such adaptive attacks.

Here are the results for these experiments:

|**LabelMapping**|**Loss**|**MNIST**|**CIFAR10**|**GTSRB**|**CIFAR100**|**Pubfig**|***Avg.***|
|-|-|-|-|-|-|-|-|
|||ACC/ACC\*|ACC/ACC\*|ACC/ACC\*|ACC/ACC\*|ACC/ACC\*|**ACC/ACC\***|
|**All-to-One**|||||||
||$L_{default}$|91.2/89.6|91.0/88.4|96.6/93.2|86.7/82.5|88.1/83.0|**90.7/87.3**|
||$L_{adaptive1}$|87.1/84.8|87.1/84.5|91.7/89.2|79.8/78.5|81.0/79.8|**85.3/83.4**|
||$L_{adaptive2}$|87.3/86.3|88.1/86.6|93.0/90.8|83.3/81.0|83.7/81.1|**87.1/85.2**|
|**All-to-All**|||||||
||$L_{default}$|90.0/87.4|89.3/87.5|92.6/89.1|82.4/85.0|83.2/80.9|**87.5/86.0**|
||$L_{adaptive1}$|76.9/74.8|78.2/76.8|82.1/80.4|73.0/71.3|69.2/67.0|**75.9/74.1**|
||$L_{adaptive2}$|84.4/83.3|85.5/83.8|85.6/84.1|78.5/77.3|79.7/78.4|**82.7/81.4**|

----
>**Details of Hard Transformations:**

We define a set $\mathcal{T} = $ { $ \{T_i\}$ } , with each $T_i$ representing a specific type of hard augmentation. For each ID sample $x$, a random subset of $k$ members from $\mathcal{T}$ is selected and permuted, $\{T_{j_1}, T_{j_2}, \ldots, T_{j_k}\}$, and the transformations are sequentially applied, resulting in $T_{j_k}(\ldots T_{j_1}(x))$.

Each transformed training sample $x$ becomes a crafted OOD $x'$, with the transformation process denoted by $G(\cdot)$, i.e., $x' = G(x)$, where $G(\cdot) = T_{j_k}(\ldots T_{j_1}(x))$. We avoid using $k=1$, which would apply only a single hard transformation, because in some cases, the hard transformation does not significantly alter the semantics. For instance, applying rotation on a "Car" image yields an OOD as it is rare in natural images. However, some semantics are rotation invariant, such as "Flower" images. Therefore, we use $k>1$, and by rule of thumb, $k=3$ has been used. This ensures that the output of $G(\cdot)$ is sufficiently shifted from the ID.

For creating the set of hard transformations, we use techniques that have been demonstrated to be harmful in the literature, including Jigsaw, Random Erasing, CutPaste, Rotation, Extreme Blurring, Intense Random Cropping, Noise Injection, Extreme Cropping, Mixup, Cutout, CutMix, and Elastic Transform [2,3,4,5]. These are the methods investigated in the literature for crafting OODs. Fig5 provides some examples of crafted OODs.

[1] Hendrycks et al. Deep Anomaly 2019

[2] Miyai Rethinking 2023

[3] Kalantidis Hard 2020

[4] Li Cutpaste 2021

[5] Sinha Negative 2021

Meta-review: The paper deals with the security of machine learning (ML) methods. Specifically, the paper proposes "TRODO", a novel defense to address the problem of "trojaned/backdoored" ML models, whose training process was compromised with adversarial manipulations that have been deliberatedly introduced by attackers. Despite being a well-known security issue, an effective solution to identifying "trojaned" models has yet to be found and TRODO seeks to overcome the limitations of prior work (such as, e.g., having poor performance when the ML model is subject to adversarial training on the poisoned dataset). TRODO is claimed to provide a "general" solution, allegedly being "agnostic" to the type of trojaning attack being employed.

The manuscript was reviewed by 6 referees with varying expertise in the application domain tackled by the paper. The reviewers praised the magnitude of the evaluation, the results, the underlying idea, the presentation/writing quality, and the fact that the paper also sheds light on the theoretical rationale behind the effectiveness of the proposed method. However, the reviewers also expressed some concerns, such as an unclear threat model, the lack of comparison with some state-of-the-art anti-trojaning methods, and the lack of consideration of an adaptive attacker.

Three reviewers, after interacting with the authors, recommend that the paper is worthy of a "6: Weak Accept", whereas another reviewer recommends a "5: Borderline Accept". In contrast, one reviewer provided a very critical review, recommending a "3: Reject" and despite actively participating in the discussion (both with the authors, as well as with other reviewers), their recommendation did not change. Finally, one reviewer recommends a "4: Borderline Reject", and is confident of the assessment, but did not acknowledge to having read the authors' response---which, in my opinion, have addressed the concerns expressed by this reviewer. 

Due to the above, I am fairly confident that this paper should be accepted: even though the _submitted manuscript_ was providing a contribution that was not yet ready to be presented at a top-tier venue such as NeurIPS, the authors' efforts in responding to the reviewers' valid remarks have shed further light on the claimed contribution, allowing it to meet the expectations of NeurIPS.

===============

To be more precise, the original submission provided a contribution that was deemed to be theoretically sound, correctly evaluated, and properly presented/motivated. Doing all of this in a (relatively) short manuscript is challenging. However, some important elements were left out, such as the effectiveness of the proposed method against an adaptive adversary (as pointed out by various reviewers), which is a necessary evaluation to carry out whenever a new "defense" is proposed. The authors have acknowledged such a shortcoming (including admitting that some attackers may completely bypass the method), and have also proceeded to carry out an evaluation considering some attacks that can be staged against TRODO.

From a security standpoint, it is well-known that "no defense is foolproof". Hence, it should not come at a surprise that TRODO cannot work against very strong attackers. The author-reviewer interactions enabled to clarify this holds also for TRODO, thereby "warning" downstream users/researchers of the limitations of TRODO in the real world. For this reason, I think that there are no grounds to reject the paper. However, the authors are encouraged to enhance their manuscript by acknowledging these limitations (potentially by toning down the overall effectiveness of TRODO) and incorporating the new experiments and clarifications that emerged during the peer review (potentially by referring to the appendix---or even to the very discussion that transpired here on OpenReview!)
Decision: Accept (poster)