Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes the backdoor defense even without a clean data set. Specifically, after randomly shuffling the filters and comparing the changes in DNN output, the authors find there is an obvious difference between backdoored DNNs and benign DNNs. Futher They propose some metrics to detect whether a given model is backdoored or not. The experimental results demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: **Strength**
1. When backdoor attacks bring practical threats to deployed DNNs, it becomes urgent and important to explore the backdoor defense. This paper has a good try.
2. To the best of my knowledge, the proposed method is novel and intriguing. I have never seen such a method, i.e., shuffling the filters, in backdoor defense.

**Weakness**
1. The method seems only to support the all-to-one attack, that is, all samples are misclassified into the same class in the presence of the trigger pattern.
2. This paper is a bit difficult to read and follow.
questions: For all-to-all backdoor attacks (samples from different classes are misclassified into different classes in the presence of the trigger pattern), does the proposed method still work?
limitations: Yes. This paper tries to provide a novel backdoor defense, which may mitigate the potential risk of backdoor attacks.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary:  In this paper, the authors introduce and investigate a new backdoor detection setup without access to clean datasets. An effective detection algorithm is proposed by contrasting representations shifts before and after randomized channel shuffling of clean and trigger features. Moreover, by leveraging the proposed shuffly trained backdoor models, the performance of trigger recovery is substantially boosted. Extensive experiments validate the effectiveness of our proposal.
strengths_and_weaknesses: Pros:

+ This paper is novel and introduces an interesting finding for backdoor attack detection. Specifically, it is the first time to try to detect such attacks without any clean data. Since we do not need clean data here, the proposed method can be used for many scenarios compared to previous methods.

+ The whole paper is easy to follow, and the main contributions are introduced clearly. The trigger feature hypothesis is very interesting and worthy to study more in the future.

+ The authors raise that the unreliability of training data source has become a serious security threat. They find that clean image features and trigger features are different in terms of their activation distribution in channels, as well as their sensitivity to channel re-ordering.

+ Utilizing trigger features’ unique characteristics, this paper proposes an effective, generalizable approach with minimal overhead based on channel shuffling to help data users examine datasets. The authors randomly shuffle weights of convolutional layers at the channel level, revealing the secretly hidden trigger features. This paper also uses the proposed shuffling method to retrain the models, helping generate more effective triggers.

+ Extensive experiments are used to verify the proposed methods.

Cons:

- Can we illustrate the trigger feature? It would be helpful to understand this key concept.

-  The trigger feature lacks a good mathematical definition, making further theoretical analysis hard.

- What is the false alarm rate of the proposed method? Namely the type I error in the field of hypothesis testing.

- Can the proposed method be used for adversarial data detection? Is that possible to distinguish between two attacks?
questions: Cons:

- Can we illustrate the trigger feature? It would be helpful to understand this key concept.

-  The trigger feature lacks a good mathematical definition, making further theoretical analysis hard.

- What is the false alarm rate of the proposed method? Namely the type I error in the field of hypothesis testing.

- Can the proposed method be used for adversarial data detection? Is that possible to distinguish between two attacks?
limitations: no potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper casts a new "self-supervised" backdoor detection setting that was not explored by prior arts: there is no prior knowledge nor control of the training data quality; and no access to a comparable clean sample set. The authors presented several interesting observations and motivated an efficient backdoor detection approach for this new setting.
strengths_and_weaknesses: Strengths:

+ This paper seems to identify a practically meaningful and novel backdoor setting, as ubiquitous and non-certified data sources are used for training big models nowadays (yet, potentially suffering over-rigor, see weakness). 
+ The work has an empirical nature and has no theoretical foundation. However, the authors managed to dig some reasonable foundation to support their hypothesis, both from prior literature and from proof-of-concept experiments, so this is appreciated. The storytelling in Sections 3.1 and 3.2 help readers understand a lot.
+ Fundamentally, their observation is rooted in the “shortcut” property of backdoor features (noticed by prior work), which could be why the backdoor features are more sparsely encoded in channels. However, leveraging this property via shuffling is an original idea, and Figure 2 gives out why. Just a thought: would this shuffling possibly be made into a more general purpose to probe a trained neural network’s stability, with little overhead?
+ Empirical results appear to be extensive. Two datasets, two architectures, and three attacks are evaluated, including the advanced WaNet. The performance gains of 0.291 ∼ 0.640 AUROC over SOTA are impressive, considering its little overhead. Both backdoor detection and trigger recovery are considered. 

Weaknesses:

My main concern is whether the authors’ proposed new setting is over-restricted or unrealistic at all. Specifically, the authors verbally claim that online public datasets are not helpful because in domain-specific applications they can be out-of-distribution. They might be true, but never manifested in practical experiments, so I’m not convinced. For example, what if one just uses a clean set from ImageNet as clean data surrogate to detect backdoors? Please justify with more concrete evidence or experiments. 

Even admitting this setting, I feel more methods are available. For example, what if first performing self-supervised pre-training, then noisy student training? Many semi-supervised and active learning approaches could have been discussed, and some might potentially be more accurate than the proposed one. However, I do acknowledge one possible merit of the authors’ work, in generating a very lightweight baseline that requires no repeated training. However, this new method in a new problem setting needs clearer contextualization for sure.
questions: Please check and answer the questions in "Weaknesses" section and the following questions:

- How sensitive the algorithm is to the threshold hyperparameter?
- Why use the last layers, and what if applying the shuffling to the first few or a random subset of layers?
- Why WaNet is missing from Table 3 and Table 4?
- Writing quality is overall not high, except for the abstract which appears to be well polished. The same polishing must be done towards the remaining paper.
- “Feature Disentanglement” is used in this paper to refer to a completely different meaning from its common context (in GANs etc). Using another term such as differentiation is perhaps better.
limitations: Social impacts have been properly discussed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a backdoor detection method based on shuffling convolutional filters and assessing the resulting shift in deep representations. Experiments are presented on two datasets, two architectures, and three backdoor attacks.
strengths_and_weaknesses: Strengths:
- To the best of my knowledge, this is a novel approach wherein filter shuffling changes the representation.
- There is an interesting method, and the results support the hypothesis in Section 3.

Weaknesses:
- Novelty: The abstract makes big claims (in bold) that may not be true. There are existing defense techniques that do not require clean data. Some papers that should be cited and methods to compare with the proposed method (others may exist):
    + Steinhardt, Jacob, Pang Wei W. Koh, and Percy S. Liang. "Certified defenses for data poisoning attacks." Advances in neural information processing systems 30 (2017). (https://arxiv.org/pdf/1706.03691.pdf)
    + Paudice, Andrea, et al. "Detection of adversarial training examples in poisoning attacks through anomaly detection." arXiv preprint arXiv:1802.03041 (2018). (https://arxiv.org/pdf/1802.03041.pdf)
    + Peri, Neehar, et al. "Deep k-nn defense against clean-label data poisoning attacks." European Conference on Computer Vision. Springer, Cham, 2020. (https://arxiv.org/pdf/1909.13374.pdf)
- This work considers relatively outdated architectures and weak attacks. Some stronger poisoning and backdoor attacks (more exist):
    + Geiping, Jonas, et al. "Witches' brew: Industrial scale data poisoning via gradient matching." arXiv preprint arXiv:2009.02276 (2020).(https://arxiv.org/pdf/2009.02276.pdf)
    + Zhu, Chen, et al. "Transferable clean-label poisoning attacks on deep neural nets." International Conference on Machine Learning. PMLR, 2019. (http://proceedings.mlr.press/v97/zhu19a/zhu19a.pdf)
    + Aghakhani, Hojjat, et al. "Bullseye polytope: A scalable clean-label poisoning attack with improved transferability." 2021 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2021.
    + Saha, Aniruddha, Akshayvarun Subramanya, and Hamed Pirsiavash. "Hidden trigger backdoor attacks." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 07. 2020.
- Missing related work. Some of the papers above have not even been cited. Some other papers probably should be cited. I'll point to a couple survey papers here:
    + Ahmed, Ibrahim M., and Manar Younis Kashmoola. "Threats on Machine Learning Technique by Data Poisoning Attack: A Survey." International Conference on Advances in Cyber Security. Springer, Singapore, 2021.
    + Goldblum, Micah, et al. "Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses." IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).
- The writing is often unclear with many grammatical errors. Some examples:
    + Line 67 says that the paper will focus on backdoor attacks. Presumably, this is as opposed to poisoning attacks without a trigger. However, one of the three attacks is CLA, which has no trigger in the original method.
    + Line 30 says "poinson."
    + Line 97: "Specifically, we retrain the model using the channel shuffling operation, and then recover trigger based on the shuffle model..." Perhaps this should be "... recover the trigger..." or "... recover triggers ...".
    + Line 52 in the appendix says "We will future extend to this case." 

Overall, I think there are interesting experiments, but there are problematic claims and missing relevant works/benchmarks. In my opinion, this paper is well below the threshold of acceptance in the state it is in. Short of showing that the proposed method does, in fact, beat the state of the art in poisoning detection/defense and better contextualized this method in the existing work, I recommend rejection.


----

After reading the authors' response and the updated submission (August 7):

My major concerned have been largely addressed. Thanks to much improved clarity around the setting, better contextualization with existing work, and additional experimental results, I can say this paper has much improved. I have increased my score accordingly and followed up with the comments below. 

As a minor note, the discussion on limitations is still lacking.

questions: - How does this method work on recent neural network architectures?
- How does this method fair on datasets that are larger, i.e. ImageNet?
- How does this defense perform against stronger poisoning attacks?
limitations: The limitations are described in Section A4. There is very little description of ways in which this work on data security and model vulnerability is applicable (or not applicable) to modern architectures and real world data. In other words, a paper that purports to defend against any kind of attack should probably explain when and where this defense will work in practice. In my opinion, discussing limitations is critical for work on security and privacy.
review_title: null

Meta-review: metareview: This work proposes a channel shuffling as a way to distinguish between backdoor and clean examples, based on the hypothesis that trigger features are sparsely encoded and activated in only a few channels. Reviewers all agreed that is a pretty intuitive, yet effective method and that it had solid evaluations after the rebuttal.

Reviewer oFCu had the concern that this paper is entirely empirical and has no supporting theory. I think this is ok given the precedence of papers in this field and also the framing of security and privacy is a more practically oriented one anyway.

The most critical reviewer (PjHg) pointed out that the benchmarks and related work contextualization were severely lacking. After the rebuttal period, these concerns were mostly alleviated.

Given the strength of the evaluations and the novelty of the idea, I believe this paper should be accepted.

That said, please address the following for the camera ready:
Please improve the writing as this was brought up by several reviewers. There are a lot of grammatical errors.
Please improve the discussion of the limitations section of this detection method as suggested by reviewer PjHg.
confidence: Certain
Decision: Accept