The Dark Side of AutoML: Towards Architectural Backdoor Search
Authors: Ren Pang
Changjiang Li
Zhaohan Xi
Shouling Ji
Ting Wang
Venue: ICLR 2023 poster (2023)
Keywords: backdoor attack and defense
neural architecture search

Abstract: This paper asks the intriguing question: is it possible to exploit neural architecture search (NAS) as a new attack vector to launch previously improbable attacks? Specifically, we present EVAS, a new attack that leverages NAS to find neural architectures with inherent backdoors and exploits such vulnerability using input-aware triggers. Compared with existing attacks, EVAS demonstrates many interesting properties: (i) it does not require polluting training data or perturbing model parameters; (ii) it is agnostic to downstream fine-tuning or even re-training from scratch; (iii) it naturally evades defenses that rely on inspecting model parameters or training data. With extensive evaluation on benchmark datasets, we show that EVAS features high evasiveness, transferability, and robustness, thereby expanding the adversary's design spectrum. We further characterize the mechanisms underlying EVAS, which are possibly explainable by architecture-level ``shortcuts'' that recognize trigger patterns. This work showcases that NAS can be exploited in a harmful way to find architectures with inherent backdoor vulnerability. The code is available at https://github.com/ain-soph/nas_backdoor.

THE DARK SIDE OF AUTOML: TOWARDS ARCHITEC- TURAL BACKDOOR SEARCH
Ren Pang! Changjiang Lit Zhaohan Xit
Shouling Ji Ting Wang!
iPennsylvania State University, (rbp5354, cb15583, zxx5113, 13, ting @psu.edu 'Zhejiang University, sji zju.edu.cn
ABSTRACT
This paper asks the intriguing question: is it possible to exploit neural architecture search (NAS) as a new attack vector to launch previously improbable attacks? Specifically, we present EVAS, a new attack that leverages NAS to find neural archi- tectures with inherent backdoors and exploits such vulnerability using input-aware triggers. Compared with existing attacks, EVAS demonstrates many interesting properties: (i) it does not require polluting training data or perturbing model pa- rameters; (ii) it agnostic to downstream fine-tuning or even re-training from scratch; (iii) it naturally evades defenses that rely inspecting model parameters or training data. With extensive evaluation on benchmark datasets, we show that EVAS features high evasiveness, transferability, and robustness, thereby expanding the adversary's s design spectrum. We further characterize the mechanisms under- lying EVAS, which are possibly explainable by architecture-level "shortcuts" that recognize trigger patterns. This work showcases that NAS can be exploited in a harmful way to find architectures with inherent backdoor vulnerability. The code is available at httpp:::/ /ttps:///tttttt github. com/ ain soph/ /oph/nas,.....oooo..pa nas backdoor
INTRODUCTION
As new paradigm of applying ML techniques in practice, automated machine learning (AutoML) automates the pipeline from raw data to deployable models, which covers model design, optimizer selection, and parameter tuning. The use of AutoML greatly simplifies the ML development cycles and propels the trend of ML democratization. In particular, neural architecture search (NAS). one primary AutoMI. task, aims to find performant deep neural network (DNN) arche tailored to given datasets. In many cases, NAS is shown to find models remarkably outperforming manually designed ones (Pham et al. 2018 Liu et al. 2019 Li et al. 2020).
In contrast to the intensive research improving the capability of NAS, its security implications are largely unexplored. As ML models are becoming the new targets malicious attacks (Biggio Roli 2018), the lack of understanding about the risks of NAS is highly concerning, given its surging popularity in security-sensitivv domains (Pang et al. 2022). Towards bridging this striking gap, we pose the intriguing yet critical question:
possible for the adversary to exploit NAS to launch previously improbable attacks?
This work provides an affirmative answer to this question. We present exploitable and vulnerable arch search (EVAS), new backdoor attack that leverages NAS to find neural arches with inherent, exploitable vulnerability. Conventional backdoor attacks typically embed the malicious functions ("backdoors") into the space of model parameters. They often assume strong threat models, such as polluting training data (Gu et al., 2017. et al. 2018: Pang et al. 2020) perturbing model Qi et al. 2022), and are thus subject to defenses based on model inspection (Wang al. 2019 Liu et al. 2019) and data filtering (Gao et al. 2019). In EVAS, however, as the backdoors are carried in the space of model arches, even if the victim trains the models using clean data and operates them in black-box manner, the backdoors are still retained. Moreover, due
parameters (Ji et al.) 2018
In the following, use "arch" for short of "architecture".

---

Published as a conference paper at ICLR 2023
to its independence of model parameters or training data, EVAS is naturally robust against defenses such as model inspection and input filtering.
To realize EVAS, define novel metrie based on neural tangent kernel (Chen et al. 202 which effectively indicates the exploitable vulnerability of a given arch; further, we integrate this metric into the
trainnnnttttgggtaaan framework (Mellor et al. [021 Chen et al.) (002T). The resulting search method is able to efficiently identify candidate arches without requiring model training or backdoor testing. To verify EVAS's empirical effectiveness, we evaluate EVAS on benchmark datasets and show: (i) EVAS successfully finds arches with exploitable vulnerability, (ii) the injected backdoors may be explained by arch-level "shortcuts" that recognize trigger patterns, and (iii) EVAS demonstrates high evasiveness, transferability, and robustness against defenses. Our findings show the feasibility of exploiting NAS as a new attack vector to implement previously improbable attacks, raise concerns about the current practice of NAS in security-sensitive domains, and point to potential directions to develop effective mitigation.
RELATED WORK
Next, we survey the literature relevant this work.
Neural arch search. The existing NAS methods can be categorized along search space, search strategy, and performance measure. Search space - early methods focus on the chain-of-layer structure (Baker et al. (017)), while recent work proposes to search for motifs of cell structures (Zoph)
al. 2018: et al. 2018 Liu et al. 2019). Search strategy - early methods rely on either random
search (Jozefowicz et al. 22155) or Bayesian optimization (Bergstra et al.) 2013). which are limited in model complexity; recent work mainly uses the approaches of reinforcement learning (Baker et al., 2017) or neural evolution (Liu et al. 2019). Performance measure- one-shot NAS has emerged as popular performance measure. It considers all candidate arches as different sub-graphs of a super-net (i.e., the one-shot model) and shares weights between candidate arches (Liu et al., 2019). Despite the intensive research on NAS, its security implications are largely unexplored. Recent work shows that NAS- MAS--enerattd models tend to be more vulnerable to various malicious attacks than manually designed ones (Pang et al. 2022] Devaguptapu et al.] 202I). This work explores another dimension: whether it can be exploited an attack vector launch new attacks, which complements the existing studies on the security of NAS.
Backdoor attacks and defenses. Backdoor attacks inject malicious backdoors into the victim's model during training and activate such backdoors at inference, which can be categorized along attack
targets input-specific (Shafahi et al., 2018), class-specific (Tang al. 2020), or any-input (Gu
et al.
2017), attack vectors polluting training data (Liu et al. 2018) or releasing infected models (Ji
ct al.
2018 and optimization metrics attack effectiveness (Pang et al.) 2020), transferability (Yao
et al.
2019 or attack evasiveness(Cher et al. 2017). To mitigate such threats, many defenses have
also been proposed, which can be categorized according to their strategies (Pang et al., (022):: input
filtering purges poisoning samples from training data (Tran et al.) 2018); model inspection determines whether a given model is backdoored(Liu et al.) 2019 Wang et al. 2019), and input inspection detects trigger inputs at inference time (Gao et al. 2019). Most attacks and defenses above focus on backdoors implemented in the space of model parameters. Concurrent to this work, (Bber- Irizar et al. (2022) explore using neural arches to implement backdoors by manually designing "trigger detectors" in the arches and activating such detectors using poisoning data during training. This work investigates using NAS to directly search for arches with exploitable vulnerability, which represents new direction of backdoor attacks.
3 EVAS
Next, we present EVAS, a new backdoor attack leveraging NAS to find neural arches with exploitable vulnerability. We begin by introducing the threat model.
3.1 THREAT MODEL.
backdoor attack injects a hidden malicious function ("backdoor") into a target model (Pang et al., 2022). The backdoor is activated once pre-defined condition ("trigger") is present, while the model
2

---

Published as a conference paper at ICLR 2023
vulnerable arch
FVAS search
fa
adversary
clean dataset
2
Training
A
clean
trigger input
input
generator
infected model
maliunction
T
((:())
(a(:0)
Figure 1: Attack framework of EvAS. (/) The adversary applies NAS search for arches with exploitable vulnerability; (2) such vulnerability is retained even the models are trained using clean data; (3) the adversary exploits such vulnerability by generating trigger-embedded inputs.
behaves normally otherwise. In a predictive task, the backdoor is often defined classifying a given input to a class desired by the adversary, while the trigger can be defined as a specific perturbation applied to the input. Formally, given input x and trigger r = (m, p) in which ra a mask and p is a pattern, the trigger-embedded input is defined as:
m) p
Let be the backdoor-infected model. The backdoor attack implies that for given input-label pair (x,, y), f(x)= and f(ø) = t with high probability, where t is the adversary's 's target class.
The conventional backdoor attacks typically follow two types of threat models: (i) the adversary backdoor-embeddde
directly trains a backdoor embedded model, which is then released to and used by the victim user (Liu jt al. 2018, Pang et al. 2020 Ji et al. [018); or (ii) the adversary indirectly pollutes the training data or manipulate the training process (Gu et al.) 2017 Qi et al. (022) to inject the backdoor into the target model. As illustrated Figure T. in EVAS, we assume a more practical threat model in which the adversary only releases the exploitable arch to the user, who may choose to train the model from scratch using clean data or apply various defenses (e.g. model inspection or data filtering) before or during using the model. We believe this represents more realistic setting: due to the prohibitive computational cost of NAS, users may opt to use performant model arches provided by third parties, which opens the door for the adversary launch the EVAS attack.
However, realizing EVAS represents non-trivial challenges including (i) how to define the trigger patterns? (ii) how to define the exploitable, vulnerable arches? and (ii) how search for such arches efficiently? Below we elaborate on each of these key questions.
3.2 INPUT-AWARE TRIGGERS
Most conventional backdoor attacks assume universal triggers: the same trigger is applied to all the inputs. However, universal triggers can be easily detected and mitigated by current defenses (Wang et al. 2019 Liu et al. 2019). Moreover, it is shown that implementing universal triggers at the arch level requires manually designing "trigger detectors" in the arches and activating such detectors using poisoning data during training (Bober-lrizar et al. 2022). which does not fit our threat model.
we adopt anput-aware triggers (Nguyen & Tran 2020). in which trigger generator g (parameterized by 9)) generates trigger Ta specific to each input Compared with universal triggers, it is more challenging to detect or mitigate input-aware triggers. Interestingly. because of the modeling capacity of the trigger generator, it more feasible implement input-aware triggers at the arch level (details in $ For simplicity, below we use ă = g(x; 3 to denote both generating trigger T. for and applying rz to x to generate the trigger-embedded input ă.
Instead, as illustrated in Figure
3.3 EXPLOITABLE ARCHES
In EVAS, we aim to find arches with backdoors exploitable by the trigger generator, which we define the following optimization problem.
3

---

Published as a conference paper at ICLR 2023
Specifically, let and respectively denote f's arch and model parameters. We define f's training as minimizing the following loss:
Cim (0, A B(z.))=D((fa (x; (2;
(2)
where fa denotes the model with arch fixed as a and the underlying data distribution. As is dependent o, we define:
0a - arg min Lim (0,
Further, we define the backdoor attack objective as:
Lax 1) (( (1; + Xl(aal((;(0)00))))
where the first term specifies that works normally clean data, the second term specifies that classifies Note that we assume the testing data follows the same distribution as the training data.
inputs to target class f, and the parameter À balances the two factors.
Overall, we consider an arch a" having exploitable vulnerability if it is possible to find a trigger generator 0", such that Latk (ox, ,0* is below certain threshold.
3.4 SEARCH WITHOUT TRAINING
Searching for exploitable archs by directly optimizing Eq. 4 challenging: the nested optimization requires recomputing (i.e., re- re-training model f) in Ltm whenever a is updated; further, as a and coupled in Latk, requires re-training generator once is changed.
Motivated by recent work (Mellor et al. 2021 Wu et al. 2021 Abdelfattah et al. 2021 Ning et al. 2021) NAS using masy-to-cmpput metrics proxies (without training), present novel method of searching for exploitable arches based on neural tangent kernel (NTK) (Jacot et al. 2018 without training the target model trigger generator. Intuitively, NTK describes model training dynamics by gradient descent (Jacot et al., 2018 Chizat et al. 2019 Lee et al. 2011)). In the limit of infinite-width DNNs. NTK becomes constant, which allows closed-form statements be made about model training. Recent work (Chen et al. 2021 Mok et al.) 2022) shows that NTK serves as an effective predictor of model "trainability" (i.e., how fast the model converges at early training stages). Formally. considering model f (parameterized by d) mapping input
probability vector fxx;; (over different classes), the NTK defined as the product of the Jacobian matrix:
(x;;;) 0)
[z; 0)
(x, 0) A
(5)
08
00
Let Amin (Amax) be the smallest (largest) eigenvalue of the empirical NTK é() & (E 0). The condition number & Amax Amin serves as a metric to estimate model trainability CChn et al. 2021), with smaller conditional number indicating higher trainability.
Condittional Number ot NTK (logs)
-0.422
0.218
100
20
80
ASR (%)
(%)
Figure 2: The conditional number of NTK versus the model performance (ACC) and vulnerability (ASR).
In our context, consider the trigger generator and the target model end-to-end model and measure the empirical NTK of the trigger generator under randomly initialized 0:
(f(
O (x; 8)
è (1) ())= A ((a.))-DDe-
(6)
00
80
4

---

Published as a conference paper at ICLR 2023
where Po, represents the initialization distribution of 8. Here, emphasize that the measure should be independent of f's initialization.
Intuitively, (()) measures the trigger generator's trainability with respect to a randomly initialized target model. The generator's 's trainability indicates the easiness of effectively generating input-aware triggers, implying the model's vulnerability to input-aware backdoor attacks. To verify the hypothesis, on the CIFAR 10 dataset with the generator configured as in Appendix $ A we measure @ with respect 900 randomly generated arches as well as the model accuracy (ACC) on clean inputs and the attack success rate (ASR) on trigger-embedded inputs. Specifically, for each arch a, we first train the model fa measure ACC and then train the trigger generator with respect to fa on the same dataset to measure ASR, with results shown in Figure Observe that the conditional number of (i)) has strong negative correlation with ASR, with smaller value indicating higher attack vulnerability; meanwhile, it has a limited correlation with ACC, with most of the arches having ACC within the range from 80% to 95%.
Leveraging the insights above, we present a simple yet effective algorithm that searches for exploitable arches without training, which is variant of regularized evolution (Real et al., 2019 Mellor et al. 2021). As sketched in Algorithm| [ it starts from a candidate pool .A of n arches randomly sampled from a pre-defined arch space; at each iteration, it samples subset A'' of arches from A, randomly mutates the best candidate (i.e., with the lowest score), and replaces the oldest arch in A with this newly mutated arch. In our implementation, the score function is defined the condition number of Eq. 6 the arch space is defined to be the NATS Bercc search space (Dong & Yang 2020), which consists of 5 atomic operators {none, skip connect, conv 1 X 1, conv and avg pooling 3 3]; and the mutation function is defined to be randomly substituting one operator with another.
Algorithm 1: EVAS Attack
Input: 72 pool size; m sample size; score score function; sample- subset sampling function; mutate
- arch mutation function;
Output: exploitable arch S, A.S,T_] 1.0.0:
// candidate archa, scores, timestamps
for i 4-+1 to do
L A(i] +- randomly generated arch, Sli] + score(A(()), Tp + 0:
best +- O:
while maximum iterations not reached yet do
i < arg ming essaple(Amm STk]:
// best candidate // //ddest candidate
j + arg max T(k);
Ali < muttaeeAA(e))
// mutata candidate
Sli) score(AAlj))
// update score
10
T + - +
// update timestampa
if S < S[best] then best +- j:
12 return A[best]:
EVALUATION
We conduct an empirical evaluation of EVAS on benchmark datasets under various scenarios. The experiments are designed to answer the following key questions: (i) does it work? we evaluate the performance and vulnerability of the arches identified by EVAS; (ii) how does it work? we explore the dynamics of EVAS search as well as the characteristies of its identified arches; and (ii) how does it differ? we compare EVAS with conventional backdoors in terms of attack evasiveness, transferability, and robustness.
4.1 EXPERIMENTAL SETTING
Datasets. In the evaluation, we primarily use three datasets that have been widely used to benchmark NAS methods (Chen et al. 2019: Li al.) 2020: Liu et al.) 2019 Pham et al.) 2018 Xie al.) 2019): CIFARIÓ (Krizhevsky & Hinton 2009). which consists of 32 x32 color images drawn from 10 classes; CIFARTO0, which is similar CIFAR10 but includes 100 finer-grained classes; and
5

---

Published as a conference paper at ICLR 2023
0
2
3
0
2
3
0
2
3
(i) EVAS
(ii) Random 1
(iii) Random 2
Conv 1x1
Conv 3x3
Skip Connect
Avg Pool 3x3
Figure 3: Sample arch identified by EVAS in comparison of two randomly generated arches.
ImageNet16, 6, which is a subset of the ImageNet dataset (Deng et al. 2009) down-sampled to images of size 16x 16 in 120 classes.
Search space. We consider the search space defined by NATS- BeschB (Dong et al. 202] ), which
consists of operators (none, skip connect, conv conv 3 x 3, and avg pooling x 3) defined among 4 nodes, implying search space of 15,625 candidate arches.
Baselines. We compare the arches found by EVAS with ResNet18 (He et al. 2016), manually designed arch. For completeness, we also include two arches randomly sampled from the NATS- Bench space, which are illustrated in Figure B By default, for each arch a, we assume the adversary trains model fa and then trains the trigger generator with respect to fo on the same dataset. We consider varying settings in which the victim directly uses fa. fine-tunes fa, or only uses a and re-trains it from scratch (details in 4.4).
Metrics. We mainly use two metrics, attack success rate (ASR) and clean data accuracy (ACC). Intuitively, ASR is the target model's accuracy in classifying trigger inputs to the adversary's target class during inference, which measures the attack effectiveness, while ACC is the target model's accuracy in correctly classifying clean inputs, which measures the attack evasiveness.
The default parameter setting and the trigger generator configuration are deferred to Appendix A
4.2
Q1: DOES EVAS WORK?
Figure illustrates one sample arch identified by EVAS on the CIFARIO dataset. We use this arch throughout this set of experiments to show that its vulnerability is at the arch level and universal across datasets. To measure the vulnerability of different arches, we first train each arch using clean data, then train trigger generator specific to this arch, and finally measure its ASR and ACC.
rrepors the results. We have the following observations. First, the ASR of EVAS is significantly higher than ResNet18 and the other two random arches. For instance, on CIFARIO, 10, EVAS is 21 11,8% 28.3%. and 34.5% more effective than ResNet18 and random arches, respectively. Second, EVAS has the highest ASR across all the datasets. Recall that use the same arch throughout different datasets. This indicates that the attack vulnerability probably resides at the arch level and is insensitive to concrete datasets, which corroborates with prior work NAS: one dataset often transfers across different datasets (Liu et al. (019).. This may be explained as follows. An arch a essentially defines function family Fa. while trained model fa 0) is an instance in F., thereby carrying the characteristics of Fa (e.g., effective to extract important features or exploitable by trigger generator). Third, all the arches show higher ASR on simpler datasets such as CIFAR1O. This may be explained by that more complex datasets (e.g., more classes, higher resolution) imply more intricate manifold structures, which may interfere with arch-level backdoors.
performant arch found
Table 1. Model performance clean inputs (ACC) and attack performance on trigger-embecdeed inputs (ASR)
of EVAS, ResNet18, and two random arches.
architecture
dataset
EVAS
ResNet18
Random I
Random II
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
CIFARIO CIFAR 100
94.26%
81.51%
96.10%
59.73%
91.91%
53.21%
92.05%
47.04%
71.54%
60.97%
78.10%
53.53%
67.09%
42.41%
67.15%
47.17%
1
45.92%
55.83%
47.62%
42.28%
39.33%
37.45%
39.48%
32.15%
To understand the attack effectiveness of EVAS on individual inputs, we illustrate sample clean inputs and their trigger-embedded variants in Figure 4 Further, using GradCam (Selvaraju et al., (2177). we show the model's interpretation of clean and trigger inputs with respect their original and target

---

Published as a conference paper at ICLR 2023
classes. Observe that the trigger pattern is specific to each input. Further, though the two trigger inputs are classified into the same target class, the difference in their heatmaps shows that the model pays attention distinct features, highlighting the effects of input-aware triggers. "aimlane"
"bird"
"airplane"
"door"
clean imput
triggeer
Figure Sample clean and trigger-embedded inputs as well as their GradCam interpretation by target model.
4.3 Q2: HOW DOES EVAS WORK?
Next, explore the dynamics of how EVAS searches for exploitable arches. For simplicity, given the arch identified by EVAS in Figure 3 we consider the set of candidate arches with the operators on the 0-3 (skip connect) and 0-1 (conv 3x3) connections replaced by others. We measure the ACC and ASR of all these candidate arches and illustrate the landscape of their scores in Figure 5 Observe that the exploitable arch features the lowest score among the surrounding arches, suggesting the existence of feasible mutation paths from random arches to reach exploitable arches following the direction of score descent.
0-3 connection
none
skip connect
conv
conv
avg pool
none
ASR:
ASR: 62.35
ASR: 41.21 ACC: 91.53
ASR: 36.99
ASR: 53.83 Score: 4.22
-30
Score: 12.54
Score: 4.49
Score: 9.84
Score: 11.82
skip connect
ASR: 52.52
ASR: 52.25
ASR: 50.22
ASR: 38.47
ASR: 44 448
25
ACC: 91.67
ACC: 92.62
ACC: 92.47
ACC: 92.11
ACC: 91.
Score: 19.43
Score: Score: 12.28
Score: 10.87
Score: 5,98
connection
20
ASR: 29.29 64
ASR: 4b.76
ASR: 30.71
ASR: 48 66
ASR: 42 62.76
conv
ACC: 91.19
ACC: 91.97
ACC: 92.54
ACC: 92.86
ACC: 91.58
15
ASR: 22 .01
ASR: 81.51
ASR: 51.19
ASR: 32, 66
ASR: 45 50
aug pool conv
Score: 23.35
ACC: 91.52
ACC: 94.26
Score: 3.48
ACC: 91,22
Score: 9.24
Score: 21.89
ACC: 93
ACC: 92
Score: 6.80
10
39.99
ASR:
43.13
ASR:
ASR: 60.71 ACC: 91 1.30
ACC:: 90.30
ACC: 92.66 62.
ACC: 92.36
ACC: 92.59
Score: 13,88
Store: 8.31
Score: 32.61
Scare: 19.37
Score:: 6.16
Figure 5: Landscape of candidate arches surrounding exploitable arches with their ASR, ACC, and scores.
Further, ask the question: what makes the arches found by EVAS exploitable? Observe that the arch in Figure uses the conv x I and 3x3 operators on number of connections. We thus generate arches by enumerating all the possible combinations of conv 1x1 1 and 3x3 on these connections and measure their performance, with results summarized in Appendix $ B Observe that while all these arches show high ASR, their vulnerability varies greatly from about 50% to 90%. We hypothesize that specific combinations of conv 1x1 and conv 3x3 create arch-level "shortcuts" for recognizing trigger patterns. We consider exploring the causal relationships between concrete arch characteristics and attack vulnerability as ongoing work.
4.4
Q3: How DOES EVAS DIFFER?
To further understand the difference between EVAS and conventional backdoors, we compare the arches found by EVAS and other arches under various training and defense scenarios.
7

---

Published as a conference paper at ICLR 2023
Fine-tuning with clean data. We first consider the scenario in which, with the trigger generator fixed, the target model is fine-tuned using clean data (with concrete setting deferred to Appendix S
Table 2 shows the results evaluated on CIFARIO and CIFARI00. Observe that fine-tuning has marginal impact on the ASR of all the arches Take Random as an example, compared with Table T its ASR on CIFAR1O 10 drops only by 7.40% after fine-tuning. This suggests that the effectiveness of fine-tuning to defend against input-aware backdoor attacks may be limited.
Table Model performance on clean inputs (ACC) and attack performance on trigeer-embedder inputs (ASR)
of EVAS, ResNet18, and two random arches after fine-tuning.
architecture
datasct
EVAS
ResNet18
Random I
Random II
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
CIFARIO
90.33%
74.40%
92.22%
53.87%
85.62%
45.81%
87.02%
45.16%
CIFARIO0
72.52%
53.50%
79.02%
50.42%
58.89%
38.91%
60.18%
25.41%
Re-training from scratch. Another common scenario is that the victim user re-initializes the target model and re-trains it from scratch using clean data. We simulate this scenario follows. After the trigger generator and target model are trained, we fix the generator, randomly initialize (using different seeds) the model, and train it on the given dataset. Table 3 compares different arches under this scenario. It is observed that EVAS sigmificantly outperforms ResNet18 and random arches in terms of ASR (with comparable ACC). For instance, it is 33,4%, 24.9%, and 19.6% more effective than the other arches respectively. This may be explained by two reasons. First, the arch-level backdoors in EVAS are inherently more agnostic to model re-training than the model-level backdoors in other archee. Second, in searching for exploitable arches, EVAS explicitly enforces that such vulnerability should be insensitive to model initialization (ef. Eq. 4 Further, observe that, as expected, re-training has a larger impact than fine-tuning on the ASR of different arches; however, it is still insufficient to mitigate input-aware backdoor attacks.
Table Model performance clean inputs (ACC) and attack performance on trigger-embedded inputs (ASR)
of EVAS, ResNet18, and two random arches after re- re-training from scratch.
architecture
dataset
EVAS
ResNet18
Random I
Random II
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
CIFARIO
94.18%
64.57%
95.62%
31.15%
91.91%
39.72%
92.09%
45.02%
CIFARIO0
71.54%
49.47%
78.53%
44.39%
67.09%
35.80%
67.01% %
39.24%
Fine-tuning with poisoning data. Further, we explore the setting in which the adversary is able to poison a tiny portion of the fine-tuning data, which assumes stronger threat model. To simulate this scenario, we apply the trigger generator to generate trigger-embedded inputs and mix them with the clean fine-tuning data. Figure 6 iillustrates the ASR and ACC of the target model as functions of the fraction of poisoning data in the fine-tuning dataset. Observe that, even with an extremely small poisoning ratio (e.g., 0.01%), it can significantly boost the ASR (e.g.. 100%) while keeping the ACC unaffected. This indicates that arch-level backdoors can be greatly enhanced by combining with other attack vectors (e.g., data poisoning).
100
90
85
00
ASR
MDC
75
0
10
poisoning (10%)
Figure 6: Model performance on clean inputs (ACC) and attack performance on embedded inputs (ASR) of EVAS as a function of poisoning ratio.
8

---

Published as a conference paper at ICLR 2023
Backdoor defenses. Finally, evaluate EvAS against three categories of defenses, model inspection, input filterin,, and model sanitization.
Model inspection determines whether given model f is infected with backdoors. We use Neural- Cleanse (Wang et al. 2019) as a representative defense. Intuitively, searches for potential triggers in each class. If class trigger-embedded, the minimum perturbation required change the predictions of inputs from other classes to this class is abnormally small. It detects anomalies using median absolute deviation (MAD) and all classes with MAD scores larger than infected. As shown in Table 4 the MAD scores of EVAS's target classes on three datasets are all below the threshold. This can be explained by that NeuralCleanse is built upon the universal trigger assumption, which does not hold for EVAS.
regarded as
Table Detection results NeuralCleanse and STRIP for EVAS. NeuralCleanse shows the MAD score and STRIP shows the AUROC score of binary classification.
dataset
NeuralCleanse
STRIP
CIFARIO
0.895
0.49
CIFARIO0
0.618
0.51
ImageNet16
0.674
0.49
Input filtering detects at inference time whether an incoming input is embedded with trigger. We use STRIP (Gao et al., 2019) as representative defense in this category. mixes a given input with a clean input and measures the self-entropy of its prediction. If the input is trigger-embedded, the mixture remains dominated by the trigger and tends to be misclassified, resulting in low self-entropy. However, as shown in T the AUROC scores of STRIP in classifying trigger-embedded inputs by EVAS are all close to random guess (i.e., 0.5). This can also be explained by that EVAS uses input-aware triggers, where each trigger only works for specific input and has a limited impact on others.
Table Model performance on clean inputs (ACC) and attack performance on trigger- embedded inputs (ASR)
of EVAS and ResNet18 18 after Fine-Pruning.
architecture
dataset
EVAS
ResNet18
ACC
ASR
ACC.
ASR
CIFARIO
90.53%
72.56%
94.11%
50.95%
CIFAR100
64.92%
54.55%
73.35%
38.54%
ImageNet16
40.28%
32.57%
42.53%
27.59%
Model sanitization, before using a given model, sanitizes it to mitigate the potential backdoors, yet without explicitly detecting whether the model is tampered. We use Sine-Pruning (Liu et al., (0188) as a representative. It uses the property that the backdoor attack typically exploits spare model capacity. It thus prunes rarely-used neurons and then applies fine-tuning to defend against pruning-aware attacks. We apply Fine-Pruning on the EVAS and ResNet18 models from Table I with results shown in Table 5) Observe that Fine- Pruming has a limited impact on the ASR of EVAS (even less than ResNet18). This may be explained as follows. The activation patterns of input-aware triggers different from that of universal triggers, each trigger may activate different set of neurons. Moreover, the arch- backdoors in EVAS may not concentrate on individual neurons but span over the whole model structures.
CONCLUSION
This work studies the feasibility of exploiting NAS an attack vector launch previously improbable attacks. We present a new backdoor attack that leverages NAS to efficiently find neural network architectures with inherent, exploitable vulnerability. Such architecture-leve backdoors demonstrate many interesting properties including evasiveness, transferability, and robustness, thereby greatly expanding the design spectrum for the adversary. We believe our findings raise concerns about the current practice of NAS in security-sensing domains and point to potential directions to develop effective mitigation.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank anonymous reviewers and shepherd for valuable feedback. This work is partially supported by the National Science Foundation under Grant No. 2212323, 21 2119331, 1951729, and 1953893. Any opinions, findings, and conclusions or recommendations are those of the authors and do not necessarily reflect the views of the National Science Foundation. S. Ji is partly supported by the National Key Research and Development Program of China under No. 2022YFB3102100, and NSFC under No. 62102360 and U1936215.
REFERENCES
Mohamed S Abdelfattah, Abhinav Mehrotra, łukasz Dudziak, and Nicholas Donald Lane. Zero- Cost Proxies for Lightweight NAS. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing Neural Network Archi- tectures using Reinforcement Learning. In Proceedings of International Conference Learning Representations (ICLR), 2017.
J. Bergstra, D. Yamins, and D. D. Cox. Making Science of Model Search: Hyperparameter Optimization Hundreds of Dimensions for Vision Architectures. Proceedings of IEEE Conference Machine Learning (ICML), 2013.
Battista Biggio and Fabio Roli. Wild Patterns: Ten Years after The Rise of Adversarial Machine Learning. Pattern Recognition, 84:317-331, 2018.
Mikel Boher-Irizar, Ilia Shumailov, Yiren Zhao, Robert Mullins, and Nicolas Papernot. Architectural Backdoors in Neural Networks. ArXiv e-prints, 2022.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural Architecture Search ImageNet in Four GPU Hours: Theoretically Inspired Perspective. In Proceedings of International Conference on Learning Representations (ICIR), 2021.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural Architecture Search on ImageNet Four GPU Hours: A Theoretically Inspired Perspective. In Proceedings of International Conference Learning Representations (ICLR), 2021.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation. Proceedings of IEEE International Conference on Computer Vision (ICCV), 2019.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted Backdoor Attacks Deep Learning Systems Using Data Poisoning. ArXiv e-prints, 2017.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On Lazy Training in Differentiable Programming. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2019.
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-scale Hierarchical Image Database. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2009.
Chaitanya Devaguptapu, Devansh Agarwal, Gaurav Mittal, Pulkit Gopalani, and Vineeth Balasub- ramanian. On Adversarial Robustness: A Neural Architecture Search perspective. In RobustML Workshop of International Conference on Learning Representationss 2021.
Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the Scope of Reproducible Neural Archi- tecture Search. In Proceedings International Conference Learning Representations (ICLR), 2020.
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size. In Proceeddings of IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
10

---

Published as a conference paper at ICLR 2023
Yansong Gao, Chang Xu, Derui Wang, Shiping Chen, Damith Ranasinghe, and Surya Nepal. STRIP:
A Defence Against Trojan Attacks on Deep Neural Networks. In ArXiv e-prints, 2019.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. ArXiv e-prints, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image
Recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernl:: Convergence and Generalization in Neural Networks. In Proceedings of IEEE Conference on Machine Learning (ICML), 2018.
Yujie Ji, Xinyang Zhang, Shouling Ji, Xiapu Luo, and Ting Wang. Model-Reuse Attacks on Deep Learning Systems. Proceedings of ACM SAC Conference on Computer and Communications (CCS), 2018.
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. An Empirical Exploration of Recurrent Network Architectures. In Proceedings of IEEE Conference on Machine Learning (ICML), 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto, 2009.
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide Neural Networks of Any Depth Evolve as Linear Models under Gradient Descent. In Proceedings of Advances in Neurul Information Processing Systems (NeurIPS), 2019.
Guohao Li, Guocheng Qian, Itzel C. Delgadillo, Matthias Müller, Ali Thabet, and Bernard Ghanem. SGAS: Sequential Greedy Architecture Search. In Proceedings of International Conference on Learning Representations (ICLR), 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. In Proceedings of International Conference on Learning Representations (ICLR), 2019.
K. Liu, B. Dovin-Gavitt, and S. Garg. Fine-Pruning Pruningrn Defending Against Backdooring Attacks on Deep Neural Networks. ArXiv e-prints, 2018.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai Weihang Wang, and Xiangyu Zhang. Trojaning Attack on Neural Networks. In Proceedings of Network and Distributed System Security Symposium (NDSS), 2018
Yingqi Liu, Wen- Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang, ABS: Scanning Neural Networks for Back-Doors by Artificial Brain Stimulation. In Proceedings of ACM SAC Conference on Computer and Communications (CCS), 2019.
Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley. Neural Architecture Search without Training. In Proceedings of IEEE Conference on Machine Learning (ICML), 2021.
Jisoo Mok, Byunggook Na. Ji-Hoon Kim, Dongyoon Han, and Sungroh Yoon. Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training? In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Tuan Anh Nguyen and Tuan Anh Tran. Input-Aware Dynamic Backdoor Attack. In Proceedings of Advances in Neural Information Processing Systems (NIPS), 2020.
Xuefei Ning, Changcheng Tang. Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Yu Wang. Evaluating Efficient Performance Estimators of Neural Architectures. In Proceedings of Advances in Neural Information Processing Systems (NIPS), 2021.
Ren Pang, Hua Shen, Xinyang Zhang, Shouling Ji, Yevgeniy Vorobeychik, Xiapu Luo, Alex Liu, and Ting Wang. A Tale of Evil Twins: Adversarial Inputs versus Poisoned Models. In Proceedings of ACM SAC Conference Computer and Communications (CCS), 2020.

---

Published as a conference paper at ICLR 2023
Ren Pang, Zhaohan Xi, Shouling Ji, Xiapu Luo, and Ting Wang. On the Security Risks of AutoML. In Proceedings of USENIX Security Symposium (SEC), 2022.
Ren Pang, Zheng Zhang, Xiangshan Gao, Zhaohan Xi, Shouling Ji, Peng Cheng, and Ting Wang. Tro- janZoo: Towards Unified, Holistic, and Practical Evaluation of Neural Backdoors, In Proceedings of IEEE European Symposium on Security and Privacy (Euro S&P), 2022.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient Neural Architecture Search via Parameter Sharing. In Proceedings of IEEE Conference Machine Learning (ICML), 2018.
Xiangyu Qi, Tinghao Xic, Ruizhe Pan, Jifeng Zhu, Yong Yang, and Kai Bu. Towards Practical Deployment-Stage Backdoor Attack Deep Neural Networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized Evolution for Image Classifier Architecture Search. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), 2019.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In Proceedings of IEEE International Conference Computer Vision (ICCV), 2017.
Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2018.
Di Tang, XiaoFeng Wang. Haixu Tang, and Kehuan Zhang. Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection. In Proceedings of USENIX Security Symposium (SEC), 2020.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral Signatures in Backdoor Attacks. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2018
B. Wang, Y. Yao, Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao. Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks. In Proceedings of IEEE Symposium Security and Privacy (S&P), 2019.
Meng- Ting-Tingggggn Wu, Hung-I Lin, and Chun-Wei Tsai. A Training-Free Genetic Neural Architecture Search. In Proceedings of ACM International Conference on Intelligent Computing and its Emerging (ICEA), 2021.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: Stochastic Neural Architecture Search.
In Proceedings of International Conference on Learning Representations (ICLR), 2019.
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Latent Backdoor Attacks on Deep Neural Networks. In Proceedings of ACM SAC Conference on Computer and Communications (CCS), 2019.
Barret Zoph, Vijay Vasudevan, Jonathon Shiens, and Quoe V. Le. Learning Transferable Architectures for Scalable Image Recognition. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
A EXPERIMENTAL SETTING
A.1 PARAMETER SETTING
Table 6 summarizes the default parameter setting.
A.2 GENERATOR ARCHITECTURE
Table 7 lists the architecture of the trigger generator.
12

---

Published as a conference paper at ICLR 2023
Table 6. Default parameter setting.
Type
Parameter
Setting
Mask generator
training epochs
25
Mark generator training epochs
Backdoor probability Ps
0.1
Backdoor attack
Cross-trigger probability Pe
0.1
Optimizer
Adam
Initial learning rate
0.01
Batch size
96
Training epochs
50
Optimizer
SGD
Fine-tuning
Initial learning rate
0.01
LRR scheduler
Cosine annealing
Batch size
96
Pool size
50
Sample size m
10
Arch search
Mutation function
random substitution
Iterations
4,000
Table Generator network architecture.
Block
Layer
Setting
ConvBNReLL 3x3
Cin = 3, Cout 32
block
ConvBNReLU 3x3
Cin Cout 32
Max Pooling
Gerne__size =
ConvBNReLL 3x3
Cin 32, Cous 64
Encoder
block
ConvBNReLU 3x3
Cn = Cout = 64
Max Pooling
kernel_sizee =2
ConvBNReLJU 3x3
Cin = 64, Cout = 128
block
ConvBN 3x3
Cin Cout 128
Max Pooling
kernel_size 2
Middle
ConvBNReLU 3x3
Cin Cout 128
Upsample
scale factor
block
ComvBNReLJU 3x3
Cin = Cous 128
ConvBNRel.U 3x3
Cin 128, Cout 64
Upsample
scale_ facto_facoor_ =
Decoder
block
ConvBNReLU 3x3
C.n Cout 64
ConvBNReLU 3x3
Cin 64, Cout = 32
Upsample
scale factor =2
block 2
ConvBNReLU 3x3
Cin Cout 32
ConvBN 3x3
Cin = 32, Cout = mask 3
B ADDITIONAL RESULTS
B. NTK TARGET MODEL
Here, we measure the NTK conditional number of the target model f under random initialization using the implementation of (Chen al. (02T) and its corresponding ASR and ACC. Figure shows their correlation. Observed that the NTK conditional number is negatively correlated with ACC (with Kendall's coefficient = -0.385) and has a very weak correlation with ASR (with T = 0.100),
which is consistent with (Chen et al. 2021).
The difference between Figure 2 and Figure/can be explained as follows. Figure 2 measures the NTK conditional number Kg of the trigger generator (with respect to the randomly initialized target model f), which indicates g's trainability (or f's vulnerability). As backdoor attacks embed two functions (one classifying clean inputs and the other classifying trigger inputs) into the same model, there tends to exist a natural trade-off between ASR and ACC. Therefore, Kg shows negative correlation with ASR and weak positive correlation with ACC. Meanwhile, Figure 7 measures the
13

---

Published as a conference paper at ICLR 2023
Conditional Number ot N (log
13-
0.100
-0.385
40
60
80
100
ASK (%)
ACC (%)
Figure 7: The NTK conditional number of target model versus the model performance (ACC) and vulnerability (ASR).
NTK conditional number ny of the target model f. which indicates f's trainability. Therefore, Ky
shows a negative correlation with ACC but a very weak correlation with ASR.
B.2 ASR-ACC TRADE-OFF
Figure 8 shows the correlation between the ASR and ACC of sampled arches (with Kendall's coefficient -0.390). Intuitively, as backdoor attacks embed two functions (one classifying clean inputs and the other classifying trigger inputs) into the same model, there tends to exist natural trade-off between ASR and ACC. This trade-off also implies that it is feasible to properly optimize ASR only to find performant but vulnerable arches.
0.390
80
ASR (%d
60
40
100
Figure 8: Trade-off between model performance (ACC) and vulnerability (ASR).
B.3 INTERPRETABILITY VERSUS VULNERABILITY
To understand the possible correlation between the attack vulnerability of arch a and its in- terpretability, we compare the interpretation each model fa regarding 100 clean inputs using GradCam (Selvaraju et al. 2017). Figure 9 illustrates sample inputs and their interpretation by different models.
Further, quantitatively measure the similarity of interpretation, we use the intersection-over-uiioo (IoU) score, which is widely used in object detection to compare model predictions with ground tround-truth bounding boxes. Formally, the IoU score of binary-valued heatmap m with respect another map m'' is defined as their Jaccard similarity:
ToU(m) 10(m) ((mmm)
(0(m) O(mm))
where ((m) denotes the set of non-zero elements in m. In our case, as the values of heatmaps are floating numbers, first apply thresholding to binarize the values. Figure I0 shows the average IoU score of each arch with respect to another. Observe that (i) the arches generated by NAS (EVAS, Random and Random II) have more similar interpretability among themselves than manually
14

---

Published as a conference paper at ICLR 2023
EVAS
Random
Random II
"bird"
"deer"
Figure 9: Sample clean inputs as well as their GradCam interpretation by different models.
designed arches (ResNet-18); (ii) the arch with high vulnerability (EvAS) significantly different from the arch with low vulnerability (Random I, II) in terms of interpretability.
Evas
0.518
0.8
Random I-
0.537
0.610
Random
0.531
0.597
0.638
0.2
ResNet-18
Evas
Random
Figure 10: ToU scores of heatmaps of different arches.
B.4 ABLATION OF ATTACK EVASIVENESS
The evasiveness of EVAS may accounted by (i) input-dependen triggers (Nguyen Tran 2020)
and (ii) arch-level vulnerability. Here, explore the contribution of input-dependent triggers the attack evasiveness. We train the trigger generator with respect to different arches (EVAS, ResNer18 88, random arches) and run NeuralCleanse and STRIP detect the attacks, with results summarized in Table Observed that while the concrete measures vary, all the attacks have MAD scores below the threshold and AUROC scores close to random guess, indicating that the input-dependent triggers mainly account for the attack evasiveness with respect to NeuralCleanse and STRIP.
Table 8. Detection results of NeuralCleanse
STRIP over EVAS, ResNet18, and two random arches on CIFAR-10. NeuralCleanse shows the MAD score and STRIP shows the AUROC score of hinary classification.
architecture
NeuralCleanse
STRIP
EVAS
0.895
0.49
ResNet18
1.110
0.49
Random
1.133
0.52
Random II
1.505
0.51
B.5 IMPORTANCE OF CONV 1 IX 1 AND CONV 3x3
We generate neighboring arches by enumerating all possible combinations of conv 1x1 and com 3x3 on the connections of the arch identified by EVAS (*[(0} ~ 0|+1(1} ~ 0/{2) ~ /|+//kipp sesss__connneet
connect ~ 0 (3) ~ 1|(4] ")" The ASR and ACC these arches are summarized Table
15

---

Published as a conference paper at ICLR 2023
Table 9. ASR and ACC of arches perturbed from "1[0} +|+ ~ 0/12]
01{3}
1|14] 2|"
(0)
(1)
(2)
(3)
(4)
ASR
ACC
Conv 1x1
63.04
91.66
Conv Ix1
Conv 3x3
52.56
92.77
Conv Ixl
Conv Ix1
57.41
92.56
Conv 3x3
Conv 3x3
57.27
93.32
Conv 1x1
Conv Ix1
63.35
93.27
Conv 1x1
Conv 3x3
62.01
94.10
Conv 3x3
Conv 1x1
62.19
93.64
Conv 3x3
Cony 3x3
58.48
93.93
Conv 1x1
Conv 1x1
56.05
93.37
Conv
Conv 3x3
63.23
93.83
Conv Ix1
Conv 1x1
51.02
93.99
Conv
Conv 3x3
69.86
93.99
Conv 3x3
Conv 1xl
55.87
94.22
Conv 1x1
Conv 3x3
64.19
93.89
Conv 3x3
Conv 1x1
55.82
93.84
Conv 3x3
Conv 3x3
55.32
94.44
Conv 1x1
53.97
93.45
Conv Ix1
Conv 3x3
87.05
94.26
Conv
Conv 1x1
58.70
93.74
Conv 3x3
Conv 3x3
48.46
94.13
Conv 1x1
Conv 1xl
63.85
94.44
Conv 1x
Conv 3x3
57.20
94.36
Conv 3x3
Conv
62.11
94.41
Conv 3x3
Conv 3x3
57.49
94.21
Conv 3x3
Conv Ix1
54.94
93.78
Conv Ix1
Conv 3x3
59.88
94.16
Conv Ix1
Conv Ixl
55.29
94.11
Conv 3x3
Conv 3x3
59.29
94.39
Conv 3x3
Conv Ixl
66.35
94.45
Conv Ixl
Conv 3x3
79.74
94.20
Conv 3x3
Conv Ixl
54.47
94.67
Conv
Conv 3x3
52.92
94.47
16


Reviews:
[{"review_id": "vjAUE0epvL", "reviewer": "Reviewer_8ZqE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes one new way to conduct back door attacks: through neural architecture search to find architectures that are vulnerable to certain triggers. The proposed EVAS is unaware of training data and model parameters so naturally evades defenses that rely on training data filtering or model parameters inspection.", "summary_of_the_review": "The paper is interesting, a new attack approach: backdoor in the model architecture is proposed. I am not an expert in attacking and defending. It seems that the paper is a natural extension of backdoors attacks from data, model parameters to model architectures.", "strength_and_weaknesses": "Strength:\n1. I think this paper may be valuable to the community.  Utilizing NAS as a way to launch attacks is quite interesting and the authors properly explain this by architecture-level “shortcuts” that recognize trigger patterns.\n2. The paper is well-motivated and clearly written.  Experiments and ablation studies are well performed. \n\nWeakness:\n1. The right figure of Figure 2 is mysterious to me. It seems that the right figure indicates that the model accuracy arises as the conditional number of NTK arises, which contradicts the result in [1]. I understand this is the NTK w.r.t trigger generator parameters instead of the model parameters but still, I would like to see some more discussion on this phenomenon. BTW., I recommend reporting Pearson correlation between ASR., Acc. and the conditional number of the NTK. \n2. It's also unclear why the architectures found on cifar-10 work well on multiple datasets. This is an interesting observation. More discussion on why there exists universal vulnerable architecture would further strengthen the paper. \n3. The ingredients of EVAS are not that novel. The input-aware triggers, NTK-based zero-cost proxy as well as the searching algorithms are all proposed in previous works. The fact that certain architectures are more vulnerable is also elaborated in [2]. Nevertheless, this paper made good efforts to combine them all. \n4. It is not clear to me if the parameters in g(.) will be trained.  Will a trained g(.) be used to launch attack?\n\nReferences:\n[1] W. Chen et.al. NEURAL ARCHITECTURE SEARCH ON IMAGENET IN FOUR GPU HOURS: A THEORETICALLY INSPIRED PERSPECTIVE. ICLR’21.\n[2] M. Bober-Irizar et.al. Architectural Backdoors in Neural Networks. arXiv preprint: arXiv.2206.07840\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cWwyijtWiKU", "reviewer": "Reviewer_EGbv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an attack exploitable and vulnerable arch search (EVAS). EVAS search for an architecture that is more vulnerable to backdoor triggers. By using NTK, EVAS can perform a training-free search. Also, by using a dynamic trigger generator, the proposed method can generate sample-specific triggers. Empirically, the proposed method demonstrated it is effectiveness. The overall attack framework is new and demonstrates that the backdoor vulnerability could also exist in the architecture. ", "summary_of_the_review": "It is a very interesting paper that exposes the backdoor vulnerability in the architecture. Despite some unclear issues mentioned in the weakness part, the merit and contribution out-weight the weakness. I would recommend accepting. ", "strength_and_weaknesses": "---\n\nStrength \n- A new backdoor attack that exploits DNN architecture results are surprisingly effective. This could open up another direction for the backdoor attacks. \n- Using NTK that search without training makes this method practical. The relationship between the conditional number and ASR clearly demonstrated the vulnerability in architecture. Empirical results comparing randomly generated arches and manually designed ones confirms the discovered vulnerability.\n- EVAS shows that attackers could operate in a black box setting without access to the model parameters. This further confirms that the vulnerability is from the architecture. \n- Most of the different existing backdoor defense types have been examined, showing that this attack can evade certain backdoor defenses.  \n- Vulnerability from the architecture perspective could offer insights for future works on attacks and defenses. Certain architectures are easy/hard to attack or defend. The overall presentation is straightforward. Good work.\n\n---\nWeaknesses\n- The practical setting is that the victim does not have the computational resource to perform NAS and ask the adversary to provide NAS as a service. In such cases, performance could be the main interest. Based on Figures 2 and 5, it seems like a low-performance arch could also have a low conditional number. Does that have a higher ASR? An analysis of the relationship between the ASR, clean accuracy, and the conditional number would make this paper more comprehensive. I think this part is currently unclear. \n- How does the $\\lambda$ affect the performance? \n- Based on the experiments, I think the attacker needs access to the training data (or data with a similar distribution) of the victim models to train the generator. However, the threat model claims that victims could train on arbitrary data. This should be further clarified, either in the threat model or in the experiments. If the victim model can use arbitrary data, the model generator for one dataset should also be able to attack the victim model trained on a different dataset. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LNFQuYkRJKp", "reviewer": "Reviewer_JhVa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deals with using NAS to search for attackable (image) classification architectures/models and attack generators. The generated attacks have to be image-dependent to make it difficult to fix them. To make this problem tracktable, the authors propose a proxy metric which does not require cumbersome training, building on the neural tangent kernel. The approach builds on the NATS-Bench search space and is evaluated for CIFAR10, CIFAR100, Imagenet. For the latter, the authors use an exemplary architecture which was found during their search process. The results focus on attack success rate (ASR) and clean data accuracy (ACC) for the found models, where a superior ASR with acceptable ACC is reported. The authors also report feature attributions and hypotheses for the properties of the attackable models. Lastly, the authors also show that the found model is sufficiently robust against available established defense mechanisms.", "summary_of_the_review": "The paper proposes an interesting and well-motivated research direction with sensible approach. The used components (regularized evolution, proxy metrics, fitness function) are clearly described, but the paper could improve the distinct motivations and design decisions for the particular search algorithm and proxy metric. A weakness in the current version of the paper is the lack of high-level results for search process experiments to prove the claim that proxy metric, search process, baseline architectures and learning objective generally work well together. In addition, I am wondering if multi-objective experimental could be fruitful to analyse and show what sacrifices we still need to make to get high attackable models (e.g., a potential lack of clean data accuracy or less interpretability). Lastly, a minor improvement would be to reference recent applications of the used search algorithm, which should be quite close to regularized evolution (at least it is referred to as such in recent ML applications).\n\n## Post author response\nI want to thank the authors for their answers to the questions and for their added material. After reading the responses as well as the other reviews, I am updating more score above the acceptancce threshold. The new material adds more details about the search method and the used parameters, and shows the dynamics of the search process.", "strength_and_weaknesses": "Strengths:\n* Novel and interesting direction of research using NAS to search for attackable models and respective generators\n* Elegant and sensible approach to make the search problem tractable via proxy metrics\n* Insightful discussion of the properties of the derived, representative model\n\nWeaknesses:\n* Only single result is evaluated and discussed. Do other models with close fitness reach similar quality? It would also be helpful to show a high-level result of the search process with selected (actual fitted/evaluated) results for ACC/ASR to further signal that the proxy metric and overall hypothesis of the approach are working\n* The parameters of the search process are missing + how long was it evaluated for?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1nuFJGwlVY9", "reviewer": "Reviewer_QG61", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates using NAS to discover architectures that are easier to backdoor without requiring any poisoned training data (i.e., the setting is to just release the architecture and let the victim train it on clean data). A metric derived from NTK serves as the score, enabling efficient exploration of the search space. Experiments demonstrate that the approach can find architectures that are easier to backdoor across many datasets and that several existing defenses do not work (in part because the considered attacks are input-dependent).", "summary_of_the_review": "The weaknesses are fairly minor. The main result is that this works at all, and this is a good contribution, so I think the paper should be accepted. I am giving the paper a 6 for now, because 7 isn't an option, but I don't feel confident enough to give it an 8. I may raise the score to a 7 after author feedback as long as no significant issues are raised by other reviewers.\n\n-----------\nUpdate after rebuttal:\n\nThe authors have addressed my concerns. I still recommend acceptance, but 7 still isn't an option and I still don't feel confident enough to give the paper an 8. I hope the AC can take this into account.", "strength_and_weaknesses": "Strengths:\n- The idea of the paper is interesting. It does seem plausible that NAS models will become more common in model sharing libraries, so preparing for this sort of attack vector seems prudent.\n- The experiment in Figure 2 nicely motivates Algorithm 1. NTK is used in an interesting way.\n- The results are promising and demonstrate that this is an interesting direction for future work.\n\nWeaknesses:\n- The ASR is a bit low, especially in the retraining from scratch experiments (the most realistic setting). However, the discovered architecture does still improve over the other architectures, so this isn't a big weakness.\n- To what extent do the input-dependent triggers lead to the results on NC and STRIP? Other architectures are not shown, but they should be.\n\nQuestions:\n- I'm confused about the experimental setup in figure 2. Is it the case that the trigger generators are trained on the datasets, and then the networks are fine-tuned on the same datasets?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster