Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A. Paper summary

- This paper proposes an easy-to-understand blackbox trojan detection method. By leveraging the phenomenon called "scaled prediction consistency", the author suggests scaling up the input images and checking the confidence score. If the confidence drops, the input should be a benign sample; otherwise, it is poisoned. The evaluation result shows that the method is very effective. The paper also proposes a new adaptive attacking trojan to fully understand the limitation of the current detection method introduced in the paper. 
strength_and_weaknesses: B. Strength: 
- The like the idea and the writing which are very clear and easy to understand.
- The method is simple and effective.
- The analysis and discussions have a lot of insights.
- The appendix covers a lot of missing details in the paper.

C. Weaknesses:
- I think the key limitation of the method is that SCALE-UP does not recover the trojan pattern. In other words, it cannot identify if the model is trojaned or not offline (e,g, https://www.ijcai.org/proceedings/2019/647). So even though your method achieves a very high AUROC score, it is not applicable in real-time applications such as a self-driving car; you cannot afford to lose ~2% of the real-time frames due to the false positive samples. 

- I don't understand why your method is only ~5% slower compared to the "no defense" method. You need to infer the sample images multiple times (up to 14 times as given in Figure 12).  Also, there is no computation reuse between different inferences (due to input changes). Can you explain this issue?

Minor: AUROC is the only metric you applied throughout the paper (and appendix). Is it possible to provide the evaluation score of other metrics as well? Or maybe you can simply plot out some ROC curves.

clarity,_quality,_novelty_and_reproducibility: Mentioned in the strength of the paper. 

summary_of_the_review: Questions of the paper are given in the "weakness". I don't have too many questions because the appendix answered most of them.

Overall, the paper is good but still has room for improvement.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers backdoor threats under the real-world machine learning as a service (MLaaS) setting where users can only query and obtain predictions of the deployed model. In this setting, the existing defenses fail to work because they assume that the suspicious models are transparent to users and can be modified. To reduce backdoor threats in the MLaaS setting, the paper proposes a simple yet effective black-box input-level backdoor detection called SCALE-UP, which requires only the predicted labels. Motivated by an intriguing observation, the proposal identify and filter malicious testing samples by analyzing their prediction consistency. The experiments verify the effectiveness and efficiency of the proposed defense method.
strength_and_weaknesses: The authors consider backdoor detection under the black-box setting in machine learning as a service (MLaaS) applications. And the proposed method alleviates backdoor threats in this case. 

Strength:
+ The MLaaS setting makes sense. In real-world applications, developers and users may directly exploit third-party pre-trained DNNs instead of training their new models. The practical application of this work is instructive.
+ The phenomenon, dubbed scaled prediction consistency, is very interesting. It is novel to utilize this phenomenon to defend against the backdoor attack.
+ The method is easy and effective. 

Though the proposal shows an excellent performance in the experiments presented in this paper, I still have some concerns about the adequacy of the experiment. 

Weaknesses / Questions
+ In Table 1-2, the defense methods(STRIP, ShrinkPad, DeepSweep, Frequency) are all designed for patch-based attacks. However, the no-patch-based attacks are used for detection, which may be more favorable for the method in the article, and the comparison may be unfair. I suggest the authors add experiments about no-patch-based defenses for comparison.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to understand. Please provide the code at the beginning of the rebuttal for reproducibility.
summary_of_the_review: The paper proposes a simple method to alleviate backdoor attacks when users cannot access or modify suspicious models. The motivation is clear, and the method is effective.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a backdoor/poisoned data detection for deep neural networks (DNN) in the black-box setting (i.e., the defender only has access to the model's final output). The authors first argue that DNN users might rely on third-party providers to get their models. However, since those models might be proprietary, the users can only query the model and get their final prediction. As such, they investigate how one can detect a backdoor input without having access to the model weights. To this end, the authors empirically show that DNNs exhibit *scaled prediction consistency* for poisoned data: an input image $\boldsymbol{x}$ and its scaled version $n\cdot\boldsymbol{x}$ consistently result in similar predictions if $\boldsymbol{x}$ is poisoned. A similar result is theoretically proved for neural tangent kernels (NTK). Based on these findings, a backdoor data detection called *SCALE-UP* is proposed. This method can be used with or without having access to benign samples. Experimental results over two datasets (CIFAR-10 and TinyImageNet) plus six different backdoor attacks show the proposed method's success in detecting backdoor-poisoned data.
strength_and_weaknesses: ### Strengths:
- The paper is very well-written. The motivation behind each section is clear, and the paper navigates the reader smoothly.
- The observations around *scaled prediction consistency* shown in the paper are pretty interesting. The authors also take a step further and provide theoretical justification for the observed phenomenon using NTKs (though this reviewer hasn't checked the proof rigorously.)
- The experimental analysis of the proposed method seems thorough. More importantly, the paper presents various counter-arguments around the proposed method (such as adaptive attacks against the proposed defense and analysis of noisy input images) and provides a fair empirical analysis for each case. 

### Weaknesses:
The most critical weakness of this work is the lack of diversity in terms of neural network architectures. To the best of this reviewer's attention, the method was only evaluated over ResNet. This choice would impose the question of whether the same observations are valid for different DNN architectures, such as various ConvNets and vision transformers.
clarity,_quality,_novelty_and_reproducibility: #### **Clarity**: The paper is put very well together. The authors provide ample justifications for each design choice and try to convey that clearly to the readers.

#### **Quality**: The paper is of high quality. The observations made in the paper would indeed be interesting to many researchers in this area.

#### **Novelty And Reproducibility**: The proposed method is novel and interesting. The authors have also provided the code for their approach.
summary_of_the_review: Based on my understanding presented above, the paper introduces an interesting phenomenon about backdoor attacks and uses this as a motivation for a defense mechanism. The experimental results are interesting, though they lack diversity in DNN architecture. Overall, I think the paper can provide good insights for the researchers in this area. Thus, I recommend acceptance at this stage. Still, I'd be keen to see my peers' take on the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper focuses on an easy-to-understand blackbox trojan detection method called SCALE-UP. By exploiting the phenomenon called scaled prediction consistency, the author suggests scaling up the input images and checking the confidence score. The philosophy behind sounds quite interesting to me, namely, if the confidence decreases, the input should be a benign sample; otherwise, it is poisoned. In theoretical part, the authors take a step further and provide theoretical justification for the observed phenomenon using NTKs. In experimental part, the evaluation result shows that the method is effective. Moreover, the paper proposes a new adaptive attacking trojan to fully understand the limitation of the current detection method introduced in the paper.

The clarity and novelty are clearly above the bar of ICLR. While the reviewers had some concerns on the real-time limitation of SCALE-UP, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to accept this paper for publication! Please include the additional experimental results and further explanation in the next version.
justification_for_why_not_higher_score: According to my expertise and reviewing process, this paper should belong to an Accept with poster.
justification_for_why_not_lower_score: According to my expertise and reviewing process, this paper should belong to an Accept with poster.
Decision: Accept: poster