SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency
Authors: Junfeng Guo
Yiming Li
Xun Chen
Hanqing Guo
Lichao Sun
Cong Liu
Venue: ICLR 2023 poster (2023)
Keywords: Backdoor Detection
Backdoor Defense
Backdoor Learning
AI Security
Deep Learning

Abstract: Deep neural networks (DNNs) are vulnerable to backdoor attacks, where adversaries embed a hidden backdoor trigger during the training process for malicious prediction manipulation. These attacks pose great threats to the applications of DNNs under the real-world machine learning as a service (MLaaS) setting, where the deployed model is fully black-box while the users can only query and obtain its predictions. Currently, there are many existing defenses to reduce backdoor threats. However, almost all of them cannot be adopted in MLaaS scenarios since they require getting access to or even modifying the suspicious models. In this paper, we propose a simple yet effective black-box input-level backdoor detection, called SCALE-UP, which requires only the predicted labels to alleviate this problem. Specifically, we identify and filter malicious testing samples by analyzing their prediction consistency during the pixel-wise amplification process. Our defense is motivated by an intriguing observation (dubbed \emph{scaled prediction consistency}) that the predictions of poisoned samples are significantly more consistent compared to those of benign ones when amplifying all pixel values. Besides, we also provide theoretical foundations to explain this phenomenon. Extensive experiments are conducted on benchmark datasets, verifying the effectiveness and efficiency of our defense and its resistance to potential adaptive attacks. Our codes are available at \url{https://github.com/JunfengGo/SCALE-UP}.

Published as a conference paper at ICLR 2023
SCALE-UP: AN EFFICIENT BLACK-BOX INPUT- LEVEL BACKDOOR DETECTION VIA ANALYZING SCALED PREDICTION CONSISTENCY
Junfeng Guol* Yiming Li Xun Chen Hanqing Guo Lichao Sun', Cong Liu
1 *Deparmment of Computer Science, UT Dallas
TTinnguua Shenzhen International Graduate School, Tsinghua University
3 *Samsung Research America, Mountain View
'Department of Computer Science, Michigan State University
DDeparmeett of Computer Science, Lehigh University
6 "Department of Electricity and Computer Engincering Department, UC Riverside
ABSTRACT
Deep neural networks (DNNs) are vulnerable to backdoor attacks, where adver- saries embed hidden backdoor trigger during the training process for malicious prediction manipulation. These attacks pose great threats to the applications of DNNs under the real-world machine learning as a service (MLaaS) setting, where the deployed model is fully black-box while the users can only query and obtain its predictions. Currently, there are many existing defenses to reduce backdoor threats. However, almost all of them cannot be adopted in MLaaS scenarios since they require getting access to or even modifying the suspicious models. In this paper, we propose simple yet effective black-box input-level backdoor detec- tion, called SCALE-UP, which requires only the predicted labels to alleviate this problem. Specifically, we identify and filter malicious testing samples by analyz- ing their prediction consistency during the pixel-wise amplification process. Our defense is motivated by an intriguing observation (dubbed scaled prediction con- sistency) that the predictions of poisoned samples are significantly more consistent compared to those of benign ones when amplifying all pixel values. Besides, we also provide theoretical foundations to explain this phenomenon. Extensive ex- periments are conducted on benchmark datasets, verifying the effectiveness and efficiency of our defense and its resistance to potential adaptive attacks. Our codes are available at https: : / //gi thub. con/ funfengGo/////oooo SCALE- UP.
INTRODUCTION
Deep neural networks (DNNs) have been deployed in wide range of mission-critical applications, such as autonomous driving (Kong et al., 2020; Grigorescu et al., 2020; Wen & Jo, 2022). face recognition (Tang & Li, 2004; Li al., 2015; Yang al., 2021), and object detection (Zhao al., 2019; Zou et al., 2019; Wang et al.,, 2021). In general, training state-of-the-ar DNNs usually requires extensive computational resources and training samples. Accordingly, in real-world applications, developers and users may directly exploit third-party pre-trained DNNs instead of training their new models. This is what we called machine learning service (MLaaS).
However, recent studies (Gu et al., 2019; Goldblum et al., 2022: Li et al., 2022a) revealed that DNNs can be compromised by embedding adversary-specified hidden backdoors during the training pro- cess, posing threatening security risks to MLaaS. The adversaries can activate embedded backdoors in the attacked models maliciously manipulate their predictions whenever the pre-defined trigger pattern appears. Users are hard to identify these attacks under the MlaaS setting since attacked DNNs behave normally on benign samples.
'The first two authors contributed equally this paper. "This work was done when Junfeng Guo and Hanqing Guo interned in Samsung Research America. *Corresponding Author: Xun Chen (e-mail: xun.c xwn.chen www.chen@ssamsunn samsung.com).

---

Published as a conference paper at ICLR 2023
Hackdoor-infeete DNN
Whirddparty Device (otaakkkott Setting)
Backdoor Detection
Backdloo Tnput
P.4
EXE
Ellnnry File
Clusa API
Edge Drvia
Prediction
Malicions
Deploy
Figure 1: An illustration of the black-box input-level backdoor detection.
To reduce backdoor threats, there are many different types of defenses, such as model repairing (Li et al., 2021b; Wu & Wang, 2021; Zeng et al., 2022), poison suppression (Du et al., 2020; Li al., 2021a; Huang et al., 2022), and backdoor detection (Xiang al., 2022; Liu et al., 2022; Guo et al., 2022c:d). However, most of these defenses were designed under the white-box setting, requiring accessing or even modifying model weights. Accordingly, users cannot adopt them in MLaaS scenarios. Currently, there are also few model-level (Huang et al., 2020; Dong al., 2021; Guo et al., 2022c) and input-level (Li et al., 2021c; Qiu et al., 2021; Gao et al., 2021) black-bor backdoor defenses where users can only access final predictions. However, these defenses have some implicit assumptions of backdoor triggers (e.g., a small static patch), leading to being easily bypassed by advanced backdoor attacks (Nguyen & Tran, 2021; Li et al.. 2021d). Their failures raise an intriguing question: what are the fundamental differences between poisoned and benign samples that can be exploited design universal black-box backdoor detection?
In this paper, we focus on the black-box input-level backdoor detection, where we intend to iden- tify whether a given suspicious input is malicious based on predictions of the deployed model (as shown in Fig. 1). This detection is practical in many real-world applications since it can serve as the 'firewall' helping to block and trace back malicious samples MI LasS scenarios. However, this problem is challenging since defenders have limited model information and no prior knowledge of the attack. Specifically, we first explore the wixel-wise amplification effects on benign and poisoned samples, motivated by the understanding that increasing trigger values does not hinder or even im- prove the attack success rate of attacked models (as preliminarily suggested in (Li et al., 2021c)). We demonstrate that the predictions of attacked images generated by both classical and advanced attacks are significantly more consistent compared to those of benign ones when amplifying all pixel values. We refer to this intriguing phenomenon also provide theoretical insights to explain this phenomenon. After that, based on these findings, we propose a simple yet effective method, dubbed scaled prediction consistency analysis (SCALE-UP), under both data-free and data-limited settings. Specifically, under the data-free setting, the SCALE- UP examines each suspicious sample by measuring its scaled prediction consistency (SPC) value, which is the proportion of labels of scaled images that are consistent with that of the input image. The larger the SPC value, the more likely this input is malicious. Under the data-limited setting, we assume that defenders have a few benign samples from each class, based on which we can reduce the side effects of class differences to further improve our SCALE-UP.
scaled prediction consistency. In particular,
In conclusion, our main contributions are four-fold. 1) We reveal an intriguing phenomenon (i.e., scaled prediction consistency) that the predictions of attacked images are significantly more consis- tent compared to those of benign ones when amplifying all pixel values. 2) We provide theoretical insights trying to explain the phenomenon of scaled prediction consistency. 3) Based on our findings, propose simple yet effective black-box input-level backdoor detection (dubbed 'SCALE-UP") under both data-free and data-limited settings. 4) We conduct extensive experiments on benchmark datasets, verifying the effectiveness of method and its resistance to potential adaptive attacks.
RELATED WORK
2.1 BACKDOOR ATTACK
Backdoor attacks (Gu et al., 2019; Li et al., 2022a; Hayase & Oh, 2023) compromise DNNs by contaminating the training process through injecting poisoned samples. These samples are crafted trigger patterns into the selected benign samples. Backdoor attacks are stealthy since the attacked models behave normally on benign samples and the adversaries only need to craft few poisoned samples. Accordingly, they introduce serious risks DNN-based applications. In general, existing attacks can be roughly divided into two main categories based on the trigger property, including 1) patch-based attacks and 2) non-patch-basee attacks, as follows:
by adding
2

---

Published as a conference paper at ICLR 2023
Patch-based Backdoor Attacks. (Gu et al., 2019) proposed the first backdoor attack, which was called BadNets. Specifically, BadNets randomly selected and modified a few benign training sam- ples by stamping the trigger patch and changing their label with a pre-defined target label. The generated poisoned samples associated with the remaining benign samples will be released to users to train their models. After that, (Chen al., 2017) first discussed the attack stealthiness and intro- duced trigger transparency, where they suggested that poisoned images should be indistinguishable compared with their benign version evade human inspection. (Turner et al., 2019) argued that making trigger patches invisible is not stealthy enough since the ground-truth labels of poisoned samples are different from the target label. They designed the first clean-label backdoor attack where adversaries can only poison samples from the target class. Recently, (Li et al., 2021c) pro- posed the first physical backdoor attack, where the location and appearance of the trigger contained in the digitized test samples may be different from that of the one used for training.
Non-patch-based Backdoor Attacks. Different from classical attacks whose trigger pattern is a small patch, recent advanced methods exploited non-patch-based triggers trying to bypass backdoor defenses. For example, (Zhao et al., 2020a) exploited full-image size targeted universal adversar- ial perturbation (Moosavi-Dezfool) et al., 2017) as the trigger pattern to design more effective clean-label backdoor attack. (Nguyen & Tran, 2021) adopted image warping as the backdoor trig- ger, which deforms the whole image while preserving image content. Recently, (Li et al., 2021d) proposed the first poison-only sample-specific trigger patterns, inspired by the DNN-based image steganography (Tancik et al., 2020). This attack broke the fundamental assumption (i.e., the trigger is samplc-agnostic) of most existing defenses and therefore could easily bypass them.
2.2 BACKDOOR DEFENSE
Currently, there are many backdoor defenses to alleviate backdoor threats. In general, existing meth- ods can be roughly divided into two main categories based on the defender's capacities, including 1) woite-box defenses and 2) black-box defenses, as follows:
White-box Backdoor Defenses. In these approaches, defenders need to obtain the source files of suspicious models. The most typical white-box defenses are model repairing, aiming at removing hidden backdoors in the attacked DNNs. For example, (Liu et al., 2018a; Wu & Wang, 2021) proposed to remove backdoors based on model pruning; (Li et al., 2021b; Xia et al., 2022) adopted model distillation to eliminate hidden backdoors. There are also other types of white-box defenses, such as poison suppression (Du et al., 2020; Li et al., 202 Huang et al., 2022) and trigger reversion (Wang et al., 2019; Hu et al., 2022: Tao al., 2022). However, users cannot use them under the machine learning as a service (MLaaS) setting, where they can only obtain model predictions.
Black-box Backdoor Defenses. In these methods, defenders can only query the (deployed) model and obtain its predictions. Currently, there are two main types of black-box defenses, including 1) model-level defenses (Huang al., 2020; Dong et al., 2021: Guo et al., 2022c) and 2) input-level defenses (Li et al., 2021c; Qiu et al., 2021; Gao et al., 2021). Specifically, the former ones intend to identify whether the (deployed) suspicious model is attacked while the latter ones detect whether given suspicious input is malicious. In this paper, since it can serve as the 'firewall' helping to block and trace back malicious samples in MLaaS scenarios. However, be easily bypassed by advanced backdoor attacks since they have some strong implicit assumptions of backdoor triggers. How to design effective black-box input-level backdoor detectors is still an important open question and worth further exploration.
focus the input-level black-box defense
we will demonstrate in our experiments, existing input-level defenses can
3 THE PHENOMENON OF SCALED PREDICTION CONSISTENCY
In this section, we explore the prediction behaviors of benign and poisoned samples generated by attacked DNNs since it is the comerstone for designing black-box input-level backdoor defense. Before illustrating our key observations, we first review the general process of backdoor attacks.
The Main Pipeline of Backdoor Attacks. Let D {(x,, v. represent unmodified be- nign training set and C' X + y is the deployed DNN, where x; E X' [0. xxx is the image, y E y = (1...., ....,KK K} is its label, and K is the number of different labels. The back- door adversaries will select some benign samples (i.e., D.) to generate their modified version
3

---

Published as a conference paper at ICLR 2023
Average Contidence
1.0
Berign Sample
Average Confidence
1.0
Average Confidence
0.8
0.8
0.6
0.6
Pesoned Sumple Bonign Strplc
Paisoned
0.4
0.4
0.4
Berign Sanple
0.2
0.2
0.2
0.0
3
7
0.0
3
9
11 13
0.0
3
5
7
9
11
13
Multiplication Times
Multiplication Times
Multiplication Times
(a) Benign Model
(b) BadNets
(c) ISSBA
Figure 2: The average confidence (i.e., average probabilities on the originally predicted label) of
benign and poisoned samples w.r.t. pixel-wise multiplications under benign and attacked models.
by Dm = {(2',,))<'' = x g()). (a: y) E D,}. where Yr is an adversary-specified target label and g(-) is pre-defined poison generator. For example, g(x) m @ x) in BadNets (Gu et al., 2019) and blended attack (Chen et al., 2017), where @ represents the element-wise product, m E [0, x|CxWxH is transparency mask, and & X' is the trigger pattern. Given N, benign sam- ples and N poisoned samples, the backdoor adversaries will train the attacked DNN ((; based on the following optimization process (with loss L):
N
min
(lf(x=0).vl + ()(z)
(1)
As preliminarily demonstrated in (Li et al., 2021c), increasing the pixel value backdoor triggers improve the attack success rate. However, defenders not accurately ma-
does not hinder nipulate these pixel values since they have no prior knowledge about trigger location. Accordingly, explore what will happen if we scale up all pixel values of benign and poisoned images.
Settings. In this section, we adopt BadNets (Gu et al., 2019)) and ISSBA (Li et al., 2021d) as the example for our discussion. They are representative of patch-based and non-patch-based backdoor attacks, respectively. We conduct experiments on the CIFAR-I0 dataset (Krizhevsky, 2009) with ResNet (He al., 2016). For both attacks, we inject a large number of poisoned samples to ensure high attack success rate (2 99%). For each benign and poisoned image, we gradually enlarge its pixel values with multiplication. We calculate the average confidence defined as the average probabilities of samples on the originally predicted label. In particular, we select the label predicted upon the original sample as the originally predicted label for each varied sample and constrain all pixel values within [0, 1] during the multiplication process. More details are in our appendix.
Results. As shown in Figure the average confidence scores of both benign and poisoned samples decrease during the multiplication process under the benign model. In other words, the predictions of modified benign and poisoned samples are changed during this process. In contrast, poisoned and benign samples have significantly distinctive behaviors under attacked models. Specifically, the average confidence of benign samples decreases whereas that of poisoned samples is relatively stable with the increase of multiplication times. We call this phenomenon as scaled prediction consistency.
To further explain this intriguing phenomenon (i.e., scaled prediction consistency), we exploit recent studies neural tangent kernel (NTK) (Jacot al., 2018) for analyzing the backdoor-infected models inspired by (Guo et al., 2022c), as follows:
Theorem 1. Suppose the poisoned training dataset consists of N₂ benign sumples and Np poisoned
samples, i.i.d sampled from uniform distribution and belonging to K classes. Assume that deep neural network f(:0) be multivariate kernel regression (RBF kernel) with the objective in Eq. (1), For a given attacked sample x' = (1 m)) 3 x m 1t, we have: limN,-Ns C(n-x') = Yt n 2 1.
In general, Theorem reveals that when the amount of poisoned samples closes to the benign sam- ples or the attacked DNN over-fit fits the poisoned samples, will still constantly predict the scaled attacked samples (i.e., n x²) the target label yt- Its proof can be found in Appendix A.
4

---

Published as conference paper at ICLR 2023
Cls
Casa
Cam
0.12
0.12
0.12
Density
Cls
Densty
Density
Oua
Cia
Gan5
Cua
0.08
Caa
0.08
0.08
Cls
Cla
0.04
Gem
0.04
Clas
Clubs
0
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
SPC Score
SPC Score
SPC Score
(a) BadNets
(b) Label-Consistent
(c) ISSBA
Figure 3: The SPC scores benign samples from different classes under attacked models.
4 SCALED PREDICTION CONSISTENCY ANALYSIS (SCALE-UP)
Motivated by the phenomenon of scaled prediction consistency demonstrated in the previous sec- tion, we propose simple yet effective black-box input-level backdoor detection, dubbed scaled prediction consistency analysis (SCALE-UP), in this section.
4.1 PRELIMINARIES
Defender's Goals. In general, defenders have two main goals, including effectiveness and efficiency. Effectiveness requires that the detection method can accurately identify whether given image is malicious or not. Efficiency ensures that detection time is limited and therefore the deployed model can provide final results on time after the detection and prediction process.
Threat Model. We consider backdoor detection under the black-box setting in machine learning as service (MIaaaS) applications. Specifically, defenders can only query the third-party deployed model and obtain its predictions. They do not have any prior information about the backdoor attack or the model. In particular, we consider two data settings, including 1) data-free detection and 2) data-limited detection. The former assumes that defenders have holding benign samples, while the latter one allows defenders to have few benign samples from each class. Note that we only assume have the predicted label instead of the predicted probability vector in our method.
4.2 DRTE-FREE SCALED PREDICTION CONSISTENCY ANALYSIS
As demonstrated in Section 3, use the average probability the originally predicted label across its scaled images to determine whether a given suspicious image is malicious. In general, the larger the probability, the more likely the sample is poisoned. However, predicted labels while predicted probability vectors are inaccessible under our settings. Accordingly, propose examine whether the predictions of scaled samples consistent, as follows:
only obtain
Let S denotes a defenderr-seeiffied scaling set (e.g., S = (3, 5, 7,9, 11}). For given input image and the deployed classifier C, we define its scaled prediction consistency (SPC) as the proportion of labels of scaled images that are consistent with that of the input image, i.e.,
SPC(x) Enes 11C(n x) C(xx))
[S]
where is the indicator function and |S] denotes the size of scaling set S. In particular, we constrain
E [0, 1] during the multiplication process.
Once we obtain the SPC value of suspicious input æ. we can determine it is malicious based on defender-specifee threshold T. If SPC(x) T, deem it as a backdoor sample.
4.3 DATA-LIMITED SCALED PREDICTION CONSISTENCY ANALYSIS
In data-free scaled prediction consistency analysis, we treat all labels equally. However, notice that the SPC values benign samples under attacked models are different across classes (as shown in Figure 3). In other words, some classes pared to the remaining ones. These benign samples with have high SPC values may be mistakenly treated malicious samples, leading relatively low precision of our method.
consistent against image scaling com-
5

---

Published as a conference paper at ICLR 2023
Suspidios Data
Ampilified Darn
Infeeted DNN
[(n=a)
Amplls
Prodint
Label Dog
SPU SPU-eerre
Pelsoaed Imsgs
Label Dog
(9.)
(2.69)
Sa
Eated Deer
Summary
Normallee
Benign Samples (Avallable)
Figure 4: The main pipeline of our (data-limited) SCALE-UP. For each suspicious input, it first generates a set of its amplified images. After that, takes amplified images to query the deployed DNN and obtain their predicted labels. Thirdly, compute and normalize the SPC value based on the results and that of some local benign samples. SCALE-UP treats the input as a malicious image if the normalized SPC value is greater than defender--peccfied threshold T.
BadNets
Label-Consistent
PhysicalBA
TUAP
WaNet
ISSBA
Poisoned
Image
Trigger
Pattern
Figure 5: The demonstration of various trigger patterns of attacks used in experiments.
data-limited scaled prediction consistency analysis, we assume that defenders have few benign samples from each class. This setting has been widely used in existing backdoor defenses (Li et al., 2021b; Guo et al., 2022c: Zeng et al., 2022). In this paper, we propose to exploit a set of statics sum- mary (i.e., mean and standard deviation σ)) of these local benign samples to alleviate this problem, inspired by (loffe & Szegedy, 2015). We first estimate the statics summary for SPC values on sam- ples from different classes, based on which normalize the SPC value of suspicious input images according to their predicted labels. Specifically, for each class i, we calculate its corresponding mean μr and standard deviation σ; based samples X; belonging class as follows:
#4 = Exex [SPC())),
o, = = (x)-)
(3)
In the detection process, given a suspicious image æ and the deployed model C, we normalize the SPC value generated by data-free SCALE-UP based on its predicted label ý A C(z), as follows:
NSPC(#) SPC(x) Pg og
(4)
Besides, for more stable and effective performance, automatically balance two terms in Eq. (4) to make their values at the same level. The main pipeline of our method is summarized in Figure 4.
EXPERIMENTS
5.1
MAIN SETTINGS
Dataset and DNN Selection. Following the settings in existing backdoor defenses, conduct experiments on CIFAR-10 (Krizhevsky, 2009) and (Tiny) ImageNet (Russakovsky et al., 2015) datasets with ResNet (He et al., 2016). Please find more detailed information in our appendix.
Attack Baselines. In this paper. we evaluate our methods under six representative attacks, includ- ing 1) BadNets (Gu al., 2019), 2) label consistent backdoor attack (dubbed "Label-Consiisentt)

---

Published as a conference paper at ICLR 2023
Table 1: The performance (AUROC) the CIFAR-10 dataset. Among all different methods, the best result is marked in boldface while the value with underline denotes the second-best result. The failed cases (i.e., AUROC 0.55) are marked in red. Note that STRIP require obtaining predicted probability vectors while other methods only need the predicted labels.
Atlack-
Defensel
BadNets
Label-Consistent
PhysicalBA
TUAP
WaNet
ISSBA
Average
STRIP
0,989
0.941
0.971
0,671
0.475
0.498
0.758
ShrinkPad
0.951
0.957
0.631
0.869
0.531
0,513
0.742
DeepSweep
0.967
0.921
0.946
0.743
0.506
0.729
0.802
Frequency
0.891
0.889
0.881
0.851
0.461
0.497
0.745
Ours (data-free)
0,971
0,947
0,969
0.816
0.918
0.945
0.928
Ours (data-limine (imi-li
0.971
0.954
0.970
0.830
0.925
0.945
0.933
Table 2: The performance (AUROC) on the Tiny ImageNet dataset. Among all different methods, the best result marked in boldface while the value with underline denotes the second-best result. The failed cases (i.e., AUROC < 0.55) are marked red. Note that STRIP require obtaining predicted probability vectors while other methods only need the predicted labels.
Attack+
BadNets
Sabel-Consistent
PhysicalBA
TUAP
WaNet
ISSBA
Average
Defensel
STRIP
0.959
0,939
0.959
0.638
0.501
0.471
0.745
ShrinkPad
0.871
0.938
0.672
0.866
0.498
0.492
0.737
DeepSweep
0.951
0,930
0.939
0.759
0.503
0.714
0.799
Frequency
0.864
0,859
0,864
0.837
0.428
0,540
0.732
Ours (data-free)
0.936
0,904
0,939
0.763
0.943
0.948
0.905
Ours (data-limited) 0.947
0.911
0.939
0.763
0.946
0.949
0.909
(Turner et al., 2019), 3) physical backdoor attack (dubbed 'PhysicalBA') (Li et al., 2021c), 4) clean- label backdoor attack with targeted universal adversarial perturbation (dubbed 'TUAP') (Zhao et al., 2020a), 5) WaNet (Nguyen & Tran, 2021), and 6) ISSBA (Li al., 2021d). They are the rep- resentative of patch-based and non-patch-baser backdoor attacks under different settings. For each attack, we randomly select the target label and inject sufficient poisoned samples to ensure the attack success rate 2 98% while preserving the overall model performance. We implement these attacks based on the open-sourced backdoor toolbox (Li al., 2023). We demonstrate the trigger patterns of adopted attacks for Tiny ImageNet in Figure 5. More detailed settings are in the appendix.
Defense Baselines. this paper, we focus on the backdoor detection under the black-box setting where defenders only query the deployed model and obtain its predicted label. Accordingly, compare our methods to ShrinkPad (Li et al., 2021c), DeepSweep (Qiu et al., 2021), and artifacts detection in the frequency domain (dubbed 'Frequency') (Zeng al., 2021). We also compare methods to STRIP (Gao et al., 2021) that requires additional requirement (i.c., obtaining predict probability vectors). We assume that defenders have 100 benign samples per class under the data- limited setting. Please find more defense details in our appendix.
Settings for Evaluation Datasets. Following the previous work (Lee al., 2018), tive (i.e., attacked) and a negative (i.e., benign) dataset to evaluate each defense. Specifically, the positive dataset contains the attacked testset and its augmented version, while the negative dataset contains a benign testset and its augmented version. The augmented datasets are created by adding small random noise their original version. The noise magnitude is set 0.05. In particular, adding these random noises will not significantly affect the attack success rate and the benign accuracy of deployed models. The introduction of the augmented datasets is to prevent evaluated defenses from over-fitting the benign or the poisoned testsets.
posi-
Evaluation Metries. Following existing detection- based backdoor defenses (Gao et al., 2021; Guo et al., 2022c), we adopt the area under receiver operating curve (AUROC) (Fawcett, 2006) to evaluate defense effectiveness, while use the inference time for evaluating efficiency. In general, the higher the AUROC and the lower the inference time, the better the backdoor detection.
7

---

Published as a conference paper at ICLR 2023
5.2 MAIN RESULTS
As shown in Table 1-2, all baseline detection methods fail de- fending against some evaluated attacks. Specifically, they have rel- atively low AUROC in detecting advanced non-patch-bar (i.e., WaNet and ISSBA). This failure is mostly because these de- fenses have some implicit assumptions (e.g., the trigger pattern is sample-agnostic
attacks
Inference Timee
2.045
3.500
static) about the attack, which are not necessar- ily true in practice. In contrast, our methods reach promising per- formance all cases both datasets. For example, the AUROC of our data-limited SCALE-UP is 0.5 greater than all baseline defenses in detecting WaNet on the Tiny ImageNet dataset. Even under the classical patch-based attacks (i.e., BadNets, aabel-Consistent, and PhysicalBA), the effectivvness of our methods on par with bet- ter than all baseline defenses. Our methods are even better than STRIP, which requires obtaining predicted probability vectors instead of predicted labels. We also provide the ROC curves of de- fenses against all attacks in Appendix N. These results verify the effectiveness of our defenses.
2.035
0.000
5TMP
Piceu Shine Dug
Our
Saferce
Figure 6: The inference time the CIFAR-IO dataset.
Besides, we also calculate the inference time of all defenses under the same computational facili- ties. In particular, we calculate the inference time of methods requiring to obtain the predictions of multiple images by feeding them simultaneously (in a data batch) into the deployed model instead of predicting them by one. Besides, only report the inference time of our SCALE-UP under the data-limited setting, since both of them have very similar running times. As shown in Figure 6, our method requires fewer inference times compared almost all baseline defenses. The only exception is ShrinkPad, whereas its effectiveness is significantly lower than that of our method. Our detection is approximately 5% slower compared with the standard inference process without any defense. These results show the efficiency of our SCALE-UP detection.
5.3 DISCUSSION
In this section, we discuss whether our method is still effective under different (adversarial) settings.
5.3.1
DEFENDING AGAINST ATTACKS WITH LARGER TRIGGER SIZES
Recent studies (Qiu et al., 2021) revealed that some defenses (e.g-, ShrinkPad) may fail in detecting samples with a relatively large trigger size. In this part, we use two patch-based attacks (e.g., BadNets and PhysicalBA) on the Tiny ImageNet dataset for discussion. As shown in Figure 7(a), our methods have high AUROC values (> 0.93) across different trigger sizes under both data- free and data-limited settings, although there are some mild fluctuations. These results verify the resistance of our SCALE-UP detection to adaptive attacks with large trigger patterns.
5.3.2 THE RESISTANCE TO POTENTIAL ADAPTIVE ATTACKS
Most recently, (Qi al., 2023) demonstrated that reducing the poisoning rate is simple yet effective method to design adaptive attacks for detection-based defenses, since it can reduce the differences between benign and poisoned samples. Motivated by this finding, SCALE-UP methods are still effective in defending against attacks with low poisoning rates. We adopt BadNets on the CIFAR-10 dataset as an example for our discussions. port the results of all poisoned testing samples and those that can be predicted as the target label, respectively. We design this setting since attacked models may still correctly predict many poisoned samples even if they contain trigger patterns when the poisoning rate is relatively low.
first explore whether our
particular, we re-
As shown Figure 7(b), the attack success rate (ASR) increases with the increase the poison- ing rate. Our method can still correctly detect poisoned samples that can successfully attack the deployed model even when the poisoning rate is set to 0.4% where the ASR is lower than 70%. In these cases, the AUROC 0.95. Besides, method can still reach promising performance (AU- ROC > 0.8) in detecting all poisoned samples. These results verify the resistance of our defense to adaptive attacks with low poisoning rates, where attacked models don't over-fit backdoor triggers.
8

---

Published as a conference paper at ICLR 2023
1.00
0.95
ASF: AUROCC
AUROC
Acaptive Atack
0.90
BadNets limled]
Varilla Atack
PlyskcuBA (Data-Iniaed)
Attack Sucpees Rate
6-1
0.85
RadRwss (Data-See)
SCALE- (Attacked)
PhysicaBBA free)
SCAL E-UP (Al))
0.80
8xB
16x16 24x24 32x32 40x40 Trigger Size
00
Polson Rate(S)
Magnituee
(a)
(c)
Figure 7: The results of additional experiments in our discussion. (a) The performance of our methods under attacks with different trigger sizes. (b) The attack performance and the defense effectiveness on all poisoned testing samples and those that can successfully attack the deployed model. (c) The effectiveness of adaptive and vanilla backdoor attacks poisoned samples with random noise under different magnitudes.
To further evaluate the resistance of our SCALE-UP potential adaptive methods, we evaluate under the worst scenario, where the backdoor adversaries are fully aware of our mechanism. Specifi- cally, we design a strong adaptive attack by introducing additional defense-resistant regularization term to the vanilla attack illustrated in Eq. (1). This regularization term is used to prevent scaled poisoned samples n x'j being predicted the target label ye. as follows:
No
N
Ny
min
((f(z,))).)) 0). + ((f(a):)),8)
+ Σ C(f(n- - x(;0),,)
(5)
Similar to previous experiments, we adopt BadNets to design the adaptive attack on the CIFAR-10 dataset. As we expected, this method can bypass our detection resulting in low AUROC (i.e., 0.467). However, the adaptive alluck would make the poisoned sumples significantly more vul- nerable to small random Gaussian noises. As shown in Figure 7(c), random noises with small magnitude (< 0.3) will significantly reduce the attack success rate of the adaptive attack, while hav- ing minor adverse effects on the vamilla attack. In other words, defenders can easily adopt random noises to defend against this adaptive attack. We speculate that its vulnerability is mostly because the regularization term significantly constrains the generalization of attacked DNNs on the poisoned samples. We will further explore its intrinsic mechanism in our future work.
5.3.3 THE EFFECTIVENESS OF SCALING PROCESS
Technically, the scaling process in our SCALE-UP detection can be regarded data augmentation method generating different modified versions of the suspicious input image. It naturally raises an intriguing question: other augmentation methods are adopted, is SCALE-UP detection still effective? Since flip operations and frequency domain analysis have been adopted in (Li et al., 2021c; Zeng al., 2021) for defense and proved have minor benefits to detecting advanced backdoor attacks (Li et al., 2021d; Nguyen & Tran, 2021), we investigate the effectiveness of adding increasing magnitudes of random noise. Due the limitations of space, we include the detailed experimental design and evaluation in Appendix O.
6 CONCLUSION
In this paper, we proposed simple yet effective black-box input-level backdoor detection (dubbed SCALE-UP) that can be used in real-world applications under the machine learning as service (MLaaS) setting. Our method was motivated by an intriguing new phenomenon (dubbed scaled pre- diction consistency) that the predictions of poisoned samples are significantly more consistent com- pared to those of benign ones when amplifying all pixel values. We also provided theoretical foun- dations to explain this phenomenon. In particular, we designed our SCALE-UP detection method under both data-free and data-limited settings. Extensive experiments benchmark datasets veri- fied the effectiveness and efficiency of our method and its resistance to potential adaptive attacks.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
DNNs have been widely and successfully adopted in many mission-critical applications. Accord- ingly, their security is of great significance. The existence of backdoor threats raises serious con- cerns about using third-party models under the machine learning as service (MLaaS) setting. In this paper, we propose simple yet effective black-box input-level backdoor detection. Accord- ingly, this work has no ethical issues since it does not reveal any new security risks and is purely defensive. However, we need to notice that our methods can only be used to filter poisoned testing samples whereas they do not reduce the intrinsic backdoor vulnerability of deployed models. Our defense also couldn't 't recover trigger patterns. People should not be too optimistic about eliminating backdoor threats. We will further improve method by exploring how to recover triggers.
ACKNOWLEDGMENT
This work is mainly supported by Samsung Research America, Mountain View, CA and partially supported by NSF CNS 2135625, CPS 2038727, CNS Career 1750263, and Darpa Shell grant.
REPRODUCIBILITY STATEMENT
We have provided detailed information on datasets, models, training settings, and computational facilities in our main manuscript and appendix. The codes for reproducing our main experiments are also soen-soureed at https: / /gthhub. com/ Jun SCALEEU
REFERENCES
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv: 1712.055/2 2017.
Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated class-incremental learning. In CVPR, 2022.
Yinpeng Dong, Xiao Yang, Zhijie Deng. Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black- box detection of backdoor attacks with limited information and data. ICCV, 2021.
Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. In ICLR, 2020.
Tom Fawcett. An introduction roc analysis. Patlern recognition letters, 27(8):861-874 2006.
Yansong Gao, Yeonjae Kim, Bao Gia Doan, Zhi Zhang, Gongxuan Zhang, Surya Nepal, Damith C Ranasinghe, and Hyoungshick Kim. Design and evaluation of multi-domain trojan detection method on deep neural networks. IEEE Transactions on Dependable and Secure Computing, 2021.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362-386, 2020.
Tianyu Gu, Kang Liu, Brendan Dolan -aviit,,, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 2019.
Hanqing Guo, Yuanda Wang, Nikolay Ivanov, Li Xiao, and Qiben Yan. Specpatch: Human-in-the loop adversarial audio spectrogram patch attack on speech recognition. CCS, 2022a.
Hanqing Guo, Qiben Yan, Nikolay Ivanov, Ying Zhu, Li Xiao, and Eric J Hunter. Supervoice: Text-independent speaker verification using ultrasound energy in human speech. In CCS, 2022b.
10

---

Published as a conference paper at ICLR 2023
Junfeng Guo, Ang Li, and Cong Liu. AEVA: Black-box backdoor detection using adversarial ex- treme value analysis. In ICLR, 2022c.
Junfeng Guo, Ang Li, and Cong Liu. Backdoor detection in reinforcement learning, arXiv preprint arXiv:2202.033000 03600, 2022d.
Jonathan Hayase and Sewoong Oh. Few-shot backdoor attacks via neural tangent kernels. In ICLR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Xiaoling Hu, Xiao Lin, Michael Cogswell, Yi Yao. Susmit Jha, and Chao Chen. Trigger hunting with a topological prior for trojan detection. In ICLR, 2022.
Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In ICLR, 2022.
Shanjiaoyang Huang, Weiqi Peng, Zhiwei Jia, and Zhuowen Tu. One-pixel signature: Characterizing cnn models for backdoor detection. In ECCV, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and gen- cralization in neural networks. In NeurlPS, 2018.
Zelun Kong, Junfeng Guo, Ang Li, and Cong Liu. Physgan: Generating sessica--world-rssiie adversarial examples for autonomous driving. In CVPR, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009,
Kimin Lee, Kihok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distrbbution samples and adversarial attacks. In NeurlPS, 2018.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learn- ing: Training clean models on poisoned data. In NeurlPS, 2021a.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR, 2021b.
Yiming Li, Tongqing Zhai, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor attack in the physical world. In ICLR Workshop, 2021c
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor leamning: A survey. IEEE Transac- tions on Neural Networks and Learning Systems, 2022a.
Yiming Li, Haoxiang Zhong, Xingjun Ma. Yong Jiang, and Shu-Tar Xia. Few-shot backdoor attacks on visual object tracking. In ICI.R, 2022b.
Yiming Li, Mengxi Ya, Yang Bai, Yong Jiang, and Shu-Tao Xia. BackdoorBox: A python toolbox for backdoor learning. arXiv preprint 2022:2200..0762,, 2023.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV, 2021d.
Zhifeng Li, Dihong Gong, Xuelong Li, and Dacheng Tao. Learning compact feature descriptor and adaptive matching framework for face recognition. IEEE Transactions on Image Processing, 2015.
Kang Liu, Brendan Dolan-Gasd Davin, and Siddharth Garg. Fine Fine-pruning: Defending against backdoor- ing attacks on deep neural networks. In RAID, 2018a.
Yingqi Liu, Shiqing Ma. Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. In NDSS, 2018b.
11

---

Published as a conference paper at ICLR 2023
Yingqi Liu, Guangyu Shen, Guanhong Tao, Zhenting Wang. Shiqing Ma, and Xiangyu Zhang. Com- plex backdoor detection by symmetric feature differencing. CVPR, 2022.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In CVPR, 2017.
Tuan Anh Nguyen and Anh Tuan Tran. Wanet imperceptible warping- based backdoor attack. In ICLR, 2021.
Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the as- sumption of latent separability for backdoor defenses. ICLR, 2023.
Han Qiu, Yi Zeng, Shangwei Guo, Tianwei Zhang, Meikang Qiu, and Bhavani Thuraisingham. Deepsweep: An evaluation framework for mitigating dnn backdoor attacks using data augmenta- tion. In AsiaCCS, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11:2211-252, 2015.
Matthew Tancik, Ben Mildenhall, and Ren Ng. Stegastamp: Invisible hyperlinks physical pho- tographs. In CVPR, 2020.
Xiaoou Tang and Zhifeng Li. Frame synchronization and multi-level subspace analysis for video based face recognition. In CVPR, 2004.
Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An, Qiuling Xu, Shiqing Ma. Pan Li, and Xiangyu Zhang, Better trigger inversion optimization in backdoor scanning. In CVPR, 2022.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao
Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao, Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In IEEE S&P, 2019.
Lixu Wang, Shichao Xu, Ruiqi Xu, Xiao Wang, and Qi Zhu. Non-transferable learning: A new approach for model ownership verification and applicability authorization. In ICLR, 2022a.
Wenguan Wang, Qiuxia Lai, Huazhu Fu, Jianbing Shen, Haibin Ling, and Ruigang Yang. Salient object detection in the deep learning era: An in-depth survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(())3233-33) 3259, 2021.
Zhuoyi Wang, Dingcheng Li, and Ping Li. Latent coresct sampling based data-free continual learn- ing. In CIKM, 2022b.
Li-Hua Wen and Kang-Hyun Jo. Deep learning-based perception systems for autonomous driving: A comprehensive survey, Neurocomputing, 2022.
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurlPS, 2021.
Jun Xia, Ting Wang, Jieping Ding, Xian Wei, and Mingsong Chen. Eliminating backdoor triggers for deep neural networks using attention relation graph distillation. In IJCAI, 2022.
Zhen Xiang, David J Miller, and George Kesidis. Post-training detection of backdoor attacks for two-class and multi-attack scenarios. In ICLR, 2022.
Xiaolong Yang, Xiaohong Jia, Dihong Gong, Dong-Ming Yan. Zhifeng Li, and Wei Liu. Larne: Lie algebra residual network for face recognition. In ICMI, 2021.
12

---

Published as a conference paper at ICLR 2023
Yi Zeng, Won Park. Zhuoqing Morley Mao, and R. Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV, 2021.
Yi Zeng. Si Chen, Won Park, Z Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In ICLR, 2022.
Tongqing Zhai, Yiming Li, Ziqi Zhang, Baoyuan Wu, Yong Jiang, and Shu-Tao Xia. Backdoor attack against speaker verification. In ICASSP, 2021.
Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu- Jiang. Clean- label backdoor attacks on video recognition models. In CVPR, 2020a.
Xujjang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. In NeurlPS, 2020b.
Zhong-Qiin Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu. Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems, 2019.
Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey. arXiv preprint arXiv11905.05055, 2019.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
A THE OMITTED PROOF OF THEOREM
Theorem 1, Suppose the poisoned training dataset consists No benign samples and N, poisoned samples, i..d. sampled from uniform distribution and belonging to K classes. Assume that deep neural network f adopt RBF kernel and cross- entropy loss with the optimization objective Eq. For given attacked sample x' = (1-m) 3 m tt, we have: N,,, C(n-x') yt, 2 1.
Proof of Theorem 1: Following (Guo et al., 2022c), we have the regression solution for NTK is:
E No K(, x4t) D Σ K(, EN K(, x4)
de(-)
(6)
K(,
where (c(+) E I is the predictive probability output of f(: () for the target class t and y is the corresponding one-hot label. ((x, m;;) e 2=112=0111 (7 > 0). Since the training samples are evenly distributed, there are F assume the target label y/+ while others are Then the regression solution can be converted to:
No benign samples belonging to Without loss of generality, we
Σ
Nofk ((,,,;)) + E 1-1
1= K (1))))
(.)-)
No ΣM K(., x4) + Σ ((+,x())
For given backdoored sample x' = (1 m) & x + m @ t, we can simplify Eq. (7) as:
Σ K( xx)) No Li=1 K(w,, 2.) Σn K(
(,(2') >
(8)
we here remove the term Nookk K(x', vz). This because x' typically don't 't belong to the target << Σi K('', æz)), otherwise the attacker has no incentive to craft
Ye and INo/k K( vx;)
poisoned sample.
When Np close to No. which implies that the poisoning rate close to 50% the attacker achieve the optimal attack efficacy (Liu et al., 2018b; Gu et al., 2019; Li et al., 2021d), Given K(x, xi) = (r 0), if N₂ = No. we have:
MN Nr K(n x², xx{) de(n x²) MNo, (n- x', xx.) Di',, K(n x x())
If 1, easily obtain that:
N
2 [|| m000-mm)[= 21)1 -e 27/1(11 (000-mm)@==--- ( -m)) m3)1
(10)
N
2 (((=-2)))))) (1 - -22111m >
(11)
Since the internal term (1 - e 2t can be always larger than 0, thus it is clear that f(=') Vt. which is also consistent with the practice.
14

---

Published as a conference paper at ICLR 2023
However, when 1, to compare Σ K(n a', x²() and E No K(n xz)), have:
No
No
((n x', ,/,x)))
>
Kn x xx4)
(12)
No
(13)
N.
(14)
(15)
Regarding the internal term I| @ (n . 2---- xz)ll Im o (n 1 we have:
I|m @ (n. t wi)ll - ||m @ (n 1)
(16)
Σ
II(n 1) ti,k + (fi, Ti) 11(n - 1). x
(17)
j.kE trigger
Σ
j. + 2(n 1) dej.
(18)
J.KE trigger
where &ijj is the pixel-level residue between the trigger and benign samples. We assume that the bij.k close to a zero mean for inputs, thus we can rewrite Eq.(12) as follows:
N
No.
K(n x,, ax{)
Σ
K (n x,, 22:)
(19)
(1 22 Siheeeennn L....)
(20)
0.
(21)
Put Eq. (19) and Eq. (9) together, we know that é x²) 2 0.5, as No + No, we have:
C(n xx)- ye, n 1. No
lim
B THE DETAILED CONFIGURATIONS OF THE EMPIRICAL STUDY
We adopt BadNets (Gu et al., 2019)) and ISSBA (Li al., 2021d) the example for discussion. They are the representative of patch-based and non-patch-based -based backdoor attacks, respectively. We conduct experiments on the CIFAR- 10 dataset (Krizhevsky, 2009) with ResNet-34 (He al., 2016). For both attacks, we inject a large number of poisoned samples to ensure high attack success rate (z 99%). For each benign and poisoned image, we gradually enlarge its pixel values with multipli- cation. We calculate the averaged confidence defined as the averaged probabilities of samples on the originally predicted label. In particular, we select the label predicted upon the original sample as the originally predicted label for each varied sample and constrain all pixel values within [0, 1] during the multiplication process. In particular, we follow previous works (Gu et al., 2019; Li et al., 202 to implement the backdoor attacks. Specifically, the trigger for BadNets is a 4 x square consisting of random pixel values; the trigger of ISSBA is generated via DNN-basee image steganography (Tancik et al., 2020). Both attacks are implemented via Backdoo BBox (Li et al., 2023).
15

---

Published as a conference paper at ICLR 2023
Table 3: Detailed information about the adopted datasets.
Dataset
# Classes
Image Size # Training Images
CIFAR-10
10
3 x 32 32
50,000 1,000,000
Tiny ImageNet
200
3 x 64 x 64
C THE DETAILS FOR TRAINING ATTACKED MODELS
We train backdoor-infected models using BackdoorBox (Li et al., 2023), We set the training epoch as 200 and the poisoning rate as 5 10% for each attack to ensure high attack success rate. In par- ticular, except for PhysicalBA, don't involve additional data augmentation in training infected models as we want to better reveal the properties of various backdoor approaches. For each in- fected model, randomly select infected labels to ensure their predictions on benign inputs similar to the benign models, which ensures the stealthiness of backdoor attacks. Regarding the data-limited scenario and ablation study, we intentionally affect multiple labels to ensure the in- fected models similar to benign models except for the Trojan behaviors. Specifically, we inject less amount(S 5%) of poisoned samples to affect labels other than the target label. This is because previ- ous work (Guo et al., 2022c) found that certain dense backdoor attacks (e.g., ISSBA, WaNet) would make the infected DNNs sensitive to noisy or out-of-distribution samples on CIFAR-10 dataset. Ac- cordingly.they are less stealthy and are easy to be detected during the sampling process of the data- limited setting and settings of our ablation study. As such, in these settings, to evaluate SCALE-UP in rather practical scenario, train infected DNNs to have similar behaviors noisy samples as the benign DNNs. The details for each dataset are included in Table 3.
C.I THE ACCURACY AND ATTACK SUCCESS RATE (ASR) FOR EVALUATED MODELS
The accuracy and ASR for the evaluated models for each task in included in Table 4.
Table 4: The BA and ASR for the evaluated models on each dataset.
Infected Model
Task Model=>
Normal Model Accuracy
BA
ASR
CIFAR-10
90.04%
> 97.7%
2 92.31%
Tiny ImageNet 2 36.98%
2 97.22%
2 40.11%
Table 5: The performance six defense baselines against partial backdoor attacks
Task| Attack
STRIP
ShrinkPad
Frequency
DeepSweep
Ours (data-frec)
Ours (data-limited)
CIFAR-10
0.617
0.949
0.891
0.967
0.971
0.971
Tiny ImageNet 0.601
0.868
0.861
0.951
0.936
0.971
D
THE DETAILED CONFIGURATIONS FOR BASELINE DEFENSES
STRIP: We implement STRIP following their official open-sourced codes".
ShrinkPad: We implement ShrinkPad following their official open-sourced codes".
Frequeny: We implement Frequency approach following their official codes".
We implement DeepSweep using Scipy package to remove the high- frequency noise and use torchvis sisn transforms and keras. keras.preprow preprocess packages to conduct transformation to inputs. Notably, we don't apply finetune pro- cess within DeepSweep since we only focus on the black-box detection scenarios.
DeepSweep:
http://twwbb.oooo////og/gggaaaaa git
thub.
16

---

Published as a conference paper at ICLR 2023
25-
21
30
26-
26
49
26
20
40
Trigger
I
(b) Trigger II
(c) Trigger III
Figure 8: The demonstration of dynamic triggers.
E
THE DESCRIPTIONS FOR MAIN EVALUATION METRIC
The receiver operating curve (ROC) shows the trade-off between detection the success rate for poisoned samples and detection error rate for benign samples across different decision thresholds T under infected-DNNe DNNNs
Inference Time: we implement each approach under the platform with one NVIDIA GPU 1080 Ti and a Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz with batch size = 1. We test the inference time each approach with an average of 1,000 runs.
F
SETTINGS FOR MEASURING THE INFERENCE TIME
Since focus defending against backdoor attacks in the inference phase, here measure the inference time by:
Identifying whether the input sample poisoned not.
* If the input is benign sample, we next should use the target model to predict it.
For STRIP, ShrinkPad, DeepSweep, and SCALE-UP, we leverage the target model's prediction on the (augmented) inputs for defense purpose, which means the input can be identified and predicted the same time. for Frequency, which leverages secondary neural network to predict the frequency domain of each given input. However, if the input is identified as benign, the target DNNs should also deliver prediction on We here assume the benign and poisoned samples have equal possibilities. Therefore, we calculate the inference time for Frequency as follows:
time TIME (IMEEFrreuency((nnu))) + 0.5 TIME TIME(DNN(ipuut))
(22)
While for other approaches, we measure their inference time via:
time (uuuueNN(() INPUT))).
(23)
In particular, we calculate inference time of methods required to obtain the predictions multi- ple images by feeding them simultaneously (in a batch) into the deployed model instead of predicting them one by one. This approach is feasible since defenders can easily and efficiently obtain all of them before feeding them into the deployed model.
G PERFORMANCE UNDER MULTIPLE-BACKDOOR TRIGGERS WITHIN A SINGLE INFECTED LABEL
Consistent with (Guo al., 2022c: Wang et al., 2019), we also eval- uate the efficacy of SCALE- UP under a more challenging scenario where multiple backdoors are embedded within single target la- bel. We randomly select a label as the infected label and inject various types of poisoned samples in the training phase. We inject arbitrary amounts of poison samples for each backdoor trigger ensure the attack efficacy ASR 2 99%. The demonstrations for used backdoor triggers shown in Figure 8. Under such consid- ered scenario, we evaluate our SCALE-UP on CIFAR-10 10 and Tiny ImageNet datasets using ResNet-34.
077
Date-ree
0.0 05
0.6
Prpprrion
Figure 10: The performance SCALE-UP for multiple infected labels.
17

---

Published as a conference paper at ICLR 2023
Datection Perforannce (AUROO))
Data-free
Datection IefoomanceeAUUROO
0.9725
Data-limited
0.9475
0.9700
0.9450
0.9675
0.8425
Data-free
0.9825
Data-limited
0.9800
3
2
Number injected triggers
Number of injected triggers
(a) CIFAR-10
(h) TinyImageNet
Figure 9: The average results for multiple triggers within single label.
As shown in Figure 9, SCALE-UP performs resilient to the increas- ing number of injected backdoor triggers. This may be caused by infected models already generalized for backdoor triggers.
H THE PERFORMANCE AGAINST MULTIPLE INFECTED LABELS
We also test SCALE-UP under the scenario where the suspicious model has multiple infected labels. Under this scenario, we test SCALE-UP on CIFAR-10. This because models on Tiny ImageNet would be prone to multiple infected labels, reported by (Guo et al., 2022c), affecting more than 14% labels can make the accuracy significantly drop. 3%. We implement BadNets as backdoor attacks, the trigger size is 4 x 4. The results shown in Figure 10. These results show that affecting multiple infected labels could slightly reduce the performance of SCALE-UL Besides, the data-limited scenario performs better than the data-free scenario. However, even with 100% labels are infected, SCALE-UP can still perform effectively with AUROC 2 0.883.
I IMPACTS FOR THE NUMBER OF COEFFICIENTS
We test SCALE-UP on six attacks with varying We here use ResNet-34 on TinylmageNet for evaluation. As shown in Figure 11, we find that the performance of SCALE-UP increases along with increasing. Moreover, we find that SCALE-UP performs more sensitive compared with other attack techniques. Moreover, we find that SCALE-UP performs similarly sensitive on in both data-limited and data-free settings. In most settings, with 11 SCALE-UP can achieve optimal performance on six different attacks.
for the TUAP attack
J IMPACT FOR THE SIZE OF LOCAL SAMPLES
We also test the sensitivity of SCALE-UP on the size of local samples per label under the data- limited setting. We test SCALE-UP Tiny ImageNet using ResNet-34 against six attacks. The results are shown in Figure 12. We can see that with the size of local samples increases, the perfor- mance of SCALE-UP improves and achieves optimal performance when the size 100.
K PERFORMANCE UNDER SOURCE-LABEL-SPEECIII BACKDOOR SCENARIOS
The Source-label-specffic (Partial) backdoor scenarios is that backdoor attacks can perform ef- fectively when it applies to images of certain specific class. Such scenario makes backdoor attacks very hard to detect (Wang et al., 2019; Gao et al., 2021), thus the attacker may have a great incentive to implement such backdoor attack in the real world. Therefore. evaluate SCALE-UP under such a practical scenario and compare SCALE-UP with previous work. We test SCALE-UP using ResNet-34 CIFAR-10 and Tiny ImageNet. As shown Table find that most defense
18

---

Published as a conference paper at ICLR 2023
Delection Performance (orormnnne(AAAOO (AUROC)
1.0
Detection Performance (AUROC)
0.85
0.80
0.9
0.75
0.8
0.70
0.7
0.65
0.60
Data-free
Data-free
Data-limited
0.55
Data- limitediied
0.5
8
10
12
14
0
8
10
12
14
(a) BadNets
(b) TUAP
Detection Performance (UROC))
Detection PerooraanceAUURO
0.9
0.8
0.8
0.7
0.7
0.6
Data-free
Data-free Data-limited
Data-limited
6
8
10
12
14
2
10
12
14
(c) Lahel-Consistent
(d) PhysicalBA
Detection Performance (AUROC)
1.0
Detection Performance(AU (AURC))
0.8
0.7
0.7
0.6
Data-free
0.6
Data-free
Data-limited
Data-limited
0.5
10
12
14
14
(e) ISSBA
(f) WaNet
Figure 11: The impact for the coefficients n.
approaches perform resilient against the partial backdoor attack except STRIP (Gao et al., 2021). This is because STRIP assumes the trigger can perform effectively across various images. Under this scenario, SCALE-UP can outperform all baseline defenses.
L THE ROBUSTNESS OF SCALE-UP
Since SCALE-UP is an inference- phase backdoor defense approach, it is necessary to investigate the robustness of SCALE-UP on benign and poisoned samples. Following previous work (Du al., 2020), we evaluate the robustness of our approach by testing different magnitudes of noisy inputs. Notably, we test SCALE-UP on benign and poisoned samples, respectively, which is because they exhibit different robustness under random noise nitudes of added random noise ensure the classification accuracy and attack success rate for benign and poisoned samples. The results are shown in Figure 13. We test approach using ResNet-34
show in Section 5.3.3. Moreover, the mag-
19

---

Published as a conference paper at ICLR 2023
Deleclion Performance AUUOC
0.84
0.9475
Data-lime limited
Delecton Performsnce/AARRG
0.82
0.9425
0.80
0.9375
0.78
Data-lime limited
0.9325
0.78
25
50
75
100
125
150
175
200
50
75
100
125
150
175
200
S22 of required inputs
size required imputs
(a) BadNets
(b) TUAP
0.915
Delection Deftrmaacce/AROC)
1.00
Delectonn BerformanasAARROO
Data-limitd
0.98
0.910
0.98
0.906
0.94
0.900
0.82
0.886
0.90
Data-limite
0.800
0.88
25
150
50
75
100
125
150
175
200
size of required inputs
size of required imputs
(c) Label-Consistent
(d) PhysicalBA
0.9490
Delection P
0.9510
Detacton Porboraanca/Auea
Date-limited
Data-limited
0.9500
0.5489
0.9996
0.9484
0.9490
0.9485
0.9480
0.9480
150
size required Inputs
size of required inputs
ISSBA
WaNet
Figure 12: The impact for the size of required inputs.
Detection Performance/AUROO (RC)
Detection Performanca/ARRO
0.9
0.9
0.8
0.6
Data-free Data-limited
0.6
Data-free
Data-limited
0.5
0.5
0.0
0.1
0.2
0.3
0.4
0.0
0.2
0.3
0.4
0.5
Magnitudes of random noise
Magnitudes random noise
(a) BadNets
(b) PhysicalBA
Figure 13: The robustness of SCALE-UP.
on the TinyImageNet task. The noise randomly sampled from Gaussian distribution and we inten- tionally filter the failed poisoned samples. We only test BadNets and PhysicalBA since only these
20

---

Published as a conference paper at ICLR 2023
Table 6: The performance (AUROC) the Tiny ImageNet dataset under VGG-19. Among all different methods, the best result is marked in boldface while the value with underline denotes the second-best result. The failed cases (i.e., AUROC 0.55) are marked in red. Note that STRIP requires obtaining predicted probability vectors while other methods only need the predicted labels.
Attack->
BadNets
Label-Consistent
PhysicalBA
TUAP
WaNet
ISSBA
Average
Defensel
STRIP
0.941
0.908
0.941
0.576
0.521
0,489
0.729
ShrinkPad
0.857
0,919
0.631
0.831
0,499
0.490
0.705
DeepSweep
0.939
0.907
0.921
0,744
0.511
0.711
0.788
Frequency
0.864
0.859
0.864
0.827
0,428
0.540
0.730
Ours (data-free)
0.936
0.846
0.907
0.858
0.893
0.767
0.868
Ours (data-limited)
0.936
0.851
0.907
0.888
0.904
0.836
0.887
Table The performance (AUROC) of SCALE-UP variants with random noises on CIFAR-10 and Ima-emtteNett datasets. The failed cases (i.e., AUROC < 0.55) are marked in red.
Attack
Dataset
SettingJ
BadNets
Label Consistent
PhysicalBA
TUAP
WaNet
ISSBA
Average
data-free
0.939
0.816
0.976
0.698
0.497
0.421
0.724
CIFAR-I0
data data-limitd
0.939
0.873
0.981
0.706
0.432
0.444
0.729
Tiny ImageNet
data-Iree
0.551
0.711
0.899
0.632
0.531
0.467
0,706
data-limited
0.761
0.899
0.644
0.534
0.501
0.706
two attacks perform robustness against random noise, as illustrated in Section 5.3.3. We find that our approach is robust against noisy poisoned samples.
M ADDITIONAL RESULTS UNDER VGG ARCHITECTURE
In our main manuscript, evaluate our method under the ResNet architecture. In this section, we conduct additional experiments under VGG-19 (BN) on Tiny ImageNet, to verify that the phe- nomenon of scaled prediction consistency is valid across different model architectures.
As shown in Figure 14, the scaled prediction consistency still holds in all cases. Specifically, the average confidence of benign samples decreases significantly faster than that of poisoned ones with the increase in multiplication time. Besides, as shown in Table 6, our methods are still better than all baseline defenses. These results verify the effectiveness of our methods again.
N THE ROC CURVES OF DEFENSES
To better compare our method with baseline defenses, we also visualize the ROC curves of defenses (as shown in Figure 15-16) under each attack on both CIFAR-10 and Tiny ImageNet in this section.
O DETAILS FOR THE EFFECTIVENES OF SCALING PROCESS
Specifically, we design the SCALE-UP variant by replacing the scaling process with adding the same varied magnitudes random noise the given inputs. As shown in Table using random noises is far less effective compared to the standard SCALE-UP methods, especially in detecting advanced attacks (i.e., WaNet and ISSBA). We speculate that mostly because they adopted invisible full- image size trigger patterns and therefore the trigger-related features are less robust. Although we currently fail to provide thcoretical analysis for the aforementioned phenomena, at least they verify the effectiveness of our scaling process. We will further discuss it in our future work.
P POTENTIAL LIMITATIONS AND FUTURE WORK
Our work is the first black-box label-only input-level backdoor detection and early-stage defenses under the block-box setting. Accordingly, have admit that work still has some limitations.
21

---

Published as a conference paper at ICLR 2023
Average Confidence
1.0
Poisoned Sample
Average Confidence
1.0
Poisoned Sample
0.8
Benign Sample
0.8
Benign Sample
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
3
5
7
9
11
7
11
Multiplication Times
Multiplication Times
(a) BadNets
(b) TUAP
Average Confidence
1.0
Average Confidence
1.0
Paisoned Sample
0.8
0.8
Benign Sample
0.6
Polsoned Sample
0.6
Benign Sample
0.4
0.4
0.2
0.2
0.0
0.0
3
5
7
9
11
5
7
11
Multiplication Times
Multiplication Times
(c) Label- -oonss-Cnnitt
(d) PhysicalBA
1.0
1.0
Average Confidence
Poisoned Sample
Average Confidence
Poisoned Sample
0.8
Benign Sample
0.8
Benign Sample
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
3
5
7
9
3
5
7
11
Multiplication Times
Multiplication Times
(c) ISSBA
(f) WaNet
The average confidence (i.e., average probabilities on the originally predicted label) of benign and poisoned samples w.r.l. pixel-wise multiplications under benign and attacked models on the Tiny ImageNet dataset with VGG-19 (with batch normalization).
Figure 14:
Firstly, our defense requires that the attacked DNNs overfit their poisoned samples. This assumption or potential limitation is also revealed by our theoretical analysis in Section 3. In other words, if the attack success rate a malicious model relatively low, the detection performance of our SCALE- UP defense may degrade sharply. Secondly, we found that our SCALE-UP detection may fail when defending against attacks some cases simple tasks (e.g.. MNIST and GTSRB). We speculate that it is mostly because attacked DNNs also overfit to benign samples due to the lack of diversity and simplicity of the dataset, making them indistinguishable from some poisoned samples when analyzing the scaled prediction consistency. We will further explore the latent mechanisms of these limitations and alleviate them in our future work.
Besides, regarding another future direction of our methods, intend generalize and adopt them to more settings and applications, such as continual learning (Wang et al., 2022b), non-transferable learning (Wang al., 2022a), federate learning (Dong al., 2022), audio signal processing (Zhai et al., 2021: Guo et al., 2022a;b), and visual object tracking (Li et al., 2022b). We will also evaluate our methods under other DNN structures (e.g., ViT (Tu al., 2022) and GCN (Zhao et al., 2020b)).
22

---

Published as a conference paper at ICLR 2023
1.0
1.0
True Positive Rate
0.8
True Positive Rate
0.8
0.6
STRIP (soft-label)
0.6
STRIP (soft-label)
ShrinkPad
ShrinkPad
0.4
0.4
Frequency
Frequency
DeepSweep
DecpSweep
0.2
0.2
SCALE-UP (data-free)
SCALE-UP (data-free)
SCALE-UP (data-limited)
SCALE-UP (data-limited)
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
(a) BadNets
(b) TUAP
1.0
1.0
True Positive Rate
True Positive Rate
0.8
0.8
0.6
STRIP (soft-label)
0.6
STRIP (soft-label)
ShrinkPad
ShrinkPad
0.4
0.4
Frequency
Frequency
DeepSweep
DeepSweep
0.2
0.2
SCALE-UP (data-free)
SCALE-UP (data-free)
SCALE-UP (data-limited)
SCALE-UP (data-limited)
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
(c) Label-Consistent
(d) PhysicalBA
1.0
1.0
STRIP (soft-label)
True Positive Rate
True Positive Rate
ShrinkPad
0.8
0.8
Frequency
DeepSweep
0.6
STRIP (soft-label)
0.6
SCALE-UP (data-free)
ShrinkPad
SCALE UP
0.4
0.4
Frequency
DeepSweep
0.2
SCALE-UP (data-free)
SCALE-UP (data-limited)
0.0
0.0
0.0
0.4
0.6
0.8
1.0
0.0
0.4
0.6
1.0
False Positive Rate
False Positive Rate
(e) ISSBA
(I) WaNet
Figure 15: The ROC curves defenses under each attack on CIFAR-10.
23

---

Published as a conference paper at ICLR 2023
1.0
1.0
True Positive Rate
0.8
True Positive Rate
0.8
0.6
STRIP (soft-label)
0.6
STRIP (soft-label)
ShrinkPad
ShrinkPad
0.4
0.4
Frequency
Frequency
DeepSweep
DecpSweep
0.2
0.2
SCALE-UP (data-free)
SCALE-UP (data-free)
SCALE-UP (data-limited)
SCALE-UP (data-limited)
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
(a) BadNets
(b) TUAP
1.0
1.0
True Positive Rate
True Positive Rate
0.8
0.8
0.6
STRIP (soft-label)
0.6
STRIP (soft-label)
ShrinkPad
ShrinkPad
0.4
0.4
Frequency
Frequency
DeepSweep
DeepSweep
0.2
0.2
SCALE-UP (data-free)
SCALE-UP (data-free)
SCALE-UP (data-limited)
SCALE-UP (data-limited)
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
(c) Label-Consistent
(d) PhysicalBA
1.0
1.0
True Positive Rate
True Positive Rate
0.8
0.8
0.6
STRIP (soft-label)
0.6
STRIP (soft-label)
ShrinkPad
ShrinkPad
0.4
Frequency
0.4
Frequency
DeepSweep
DeepSweep
SCALE-UP (data-free)
SCALE-UP (data-free)
SCALE-UP (data-limited)
SCALE-UP (data-limited)
0.0
0.0
0.0
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
(e) ISSBA
(I) WaNet
Figure 16: The ROC curves of defenses under each attack on Tiny ImageNet.
24


Reviews:
[{"review_id": "uf1FziHcPE2", "reviewer": "Reviewer_GMnc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A. Paper summary\n\n- This paper proposes an easy-to-understand blackbox trojan detection method. By leveraging the phenomenon called \"scaled prediction consistency\", the author suggests scaling up the input images and checking the confidence score. If the confidence drops, the input should be a benign sample; otherwise, it is poisoned. The evaluation result shows that the method is very effective. The paper also proposes a new adaptive attacking trojan to fully understand the limitation of the current detection method introduced in the paper. ", "summary_of_the_review": "Questions of the paper are given in the \"weakness\". I don't have too many questions because the appendix answered most of them.\n\nOverall, the paper is good but still has room for improvement. \n", "strength_and_weaknesses": "B. Strength: \n- The like the idea and the writing which are very clear and easy to understand.\n- The method is simple and effective.\n- The analysis and discussions have a lot of insights.\n- The appendix covers a lot of missing details in the paper.\n\nC. Weaknesses:\n- I think the key limitation of the method is that SCALE-UP does not recover the trojan pattern. In other words, it cannot identify if the model is trojaned or not offline (e,g, https://www.ijcai.org/proceedings/2019/647). So even though your method achieves a very high AUROC score, it is not applicable in real-time applications such as a self-driving car; you cannot afford to lose ~2% of the real-time frames due to the false positive samples. \n\n- I don't understand why your method is only ~5% slower compared to the \"no defense\" method. You need to infer the sample images multiple times (up to 14 times as given in Figure 12).  Also, there is no computation reuse between different inferences (due to input changes). Can you explain this issue?\n\nMinor: AUROC is the only metric you applied throughout the paper (and appendix). Is it possible to provide the evaluation score of other metrics as well? Or maybe you can simply plot out some ROC curves.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "uNMrdj0YaR5", "reviewer": "Reviewer_4qGQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers backdoor threats under the real-world machine learning as a service (MLaaS) setting where users can only query and obtain predictions of the deployed model. In this setting, the existing defenses fail to work because they assume that the suspicious models are transparent to users and can be modified. To reduce backdoor threats in the MLaaS setting, the paper proposes a simple yet effective black-box input-level backdoor detection called SCALE-UP, which requires only the predicted labels. Motivated by an intriguing observation, the proposal identify and filter malicious testing samples by analyzing their prediction consistency. The experiments verify the effectiveness and efficiency of the proposed defense method.", "summary_of_the_review": "The paper proposes a simple method to alleviate backdoor attacks when users cannot access or modify suspicious models. The motivation is clear, and the method is effective.  ", "strength_and_weaknesses": "The authors consider backdoor detection under the black-box setting in machine learning as a service (MLaaS) applications. And the proposed method alleviates backdoor threats in this case. \n\nStrength:\n+ The MLaaS setting makes sense. In real-world applications, developers and users may directly exploit third-party pre-trained DNNs instead of training their new models. The practical application of this work is instructive.\n+ The phenomenon, dubbed scaled prediction consistency, is very interesting. It is novel to utilize this phenomenon to defend against the backdoor attack.\n+ The method is easy and effective. \n\nThough the proposal shows an excellent performance in the experiments presented in this paper, I still have some concerns about the adequacy of the experiment. \n\nWeaknesses / Questions\n+ In Table 1-2, the defense methods(STRIP, ShrinkPad, DeepSweep, Frequency) are all designed for patch-based attacks. However, the no-patch-based attacks are used for detection, which may be more favorable for the method in the article, and the comparison may be unfair. I suggest the authors add experiments about no-patch-based defenses for comparison.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aniIE5d_6vo", "reviewer": "Reviewer_JRLs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a backdoor/poisoned data detection for deep neural networks (DNN) in the black-box setting (i.e., the defender only has access to the model's final output). The authors first argue that DNN users might rely on third-party providers to get their models. However, since those models might be proprietary, the users can only query the model and get their final prediction. As such, they investigate how one can detect a backdoor input without having access to the model weights. To this end, the authors empirically show that DNNs exhibit *scaled prediction consistency* for poisoned data: an input image $\\boldsymbol{x}$ and its scaled version $n\\cdot\\boldsymbol{x}$ consistently result in similar predictions if $\\boldsymbol{x}$ is poisoned. A similar result is theoretically proved for neural tangent kernels (NTK). Based on these findings, a backdoor data detection called *SCALE-UP* is proposed. This method can be used with or without having access to benign samples. Experimental results over two datasets (CIFAR-10 and TinyImageNet) plus six different backdoor attacks show the proposed method's success in detecting backdoor-poisoned data.", "summary_of_the_review": "Based on my understanding presented above, the paper introduces an interesting phenomenon about backdoor attacks and uses this as a motivation for a defense mechanism. The experimental results are interesting, though they lack diversity in DNN architecture. Overall, I think the paper can provide good insights for the researchers in this area. Thus, I recommend acceptance at this stage. Still, I'd be keen to see my peers' take on the paper.", "strength_and_weaknesses": "### Strengths:\n- The paper is very well-written. The motivation behind each section is clear, and the paper navigates the reader smoothly.\n- The observations around *scaled prediction consistency* shown in the paper are pretty interesting. The authors also take a step further and provide theoretical justification for the observed phenomenon using NTKs (though this reviewer hasn't checked the proof rigorously.)\n- The experimental analysis of the proposed method seems thorough. More importantly, the paper presents various counter-arguments around the proposed method (such as adaptive attacks against the proposed defense and analysis of noisy input images) and provides a fair empirical analysis for each case. \n\n### Weaknesses:\nThe most critical weakness of this work is the lack of diversity in terms of neural network architectures. To the best of this reviewer's attention, the method was only evaluated over ResNet. This choice would impose the question of whether the same observations are valid for different DNN architectures, such as various ConvNets and vision transformers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster