Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The proposed approach presents scalable certified defense against adversarial patch attack.  The method relies on pruning unimportant neurons and identifying regions in the input image which contain the adversarial patch. Once the patch location is identified, the label is predicted is using prediction consistency. A certification algorithm is provided which shows significant improvements over previous methods on ImageNet. 
main_review: Strengths:

— The method shows significant improvement compared to previous approaches on ImageNet, addressing an important problem of scalability for defenses against adversarial patch attacks.

 — The paper is well written and easy to follow.

— Detailed ablation study shows effect of individual component of the algorithm.

Weaknesses:

— Results are shown on only adversarial patch attack, which is one form of physical world attack. Other attack formulations such as Adversarial framing and Adversarial camouflage are not considered.

— Other large scale datasets such as Places can be considered for large scale datasets. ImageNet being an object centric dataset might be helpful in identifying the 

patched region.

Additional Comments:

— Adversarial framing [1] and Adversarial camouflage[2] are more recent forms of physical world attack. Does the proposed approach defend well against such attacks?

— Although the proposed approach works well on ImageNet, more explanation is required for explaining the drop in performance w.r.t both clean and certified accuracy on Cifar10. This gap increases as the size of patch increases.

— The choice of superficial layer is an important experiment which does not lead to clear understanding of the method. Why does the performance drop for Layer 2 and increase again for Layer3 in Table 7 of the appendix?  A more thorough analysis seems necessary.

[1] -  arXiv:1812.04599

[2] - arXiv:2003.08757 

-----POST REBUTTAL-----
The authors have addressed most of my concerns. Having gone through other reviews and the rebuttal, I agree that this is a good paper and I raise my score to 6.  I encourage the authors to include the discussion on the drop in performance for low resolution datasets and the guidelines for superficial layer selection.   
limitations_and_societal_impact: The authors have not addressed limitations and societal impact of their work. A discussion on this seems to be missing from the appendix.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose a scalable certified defense framework to detect patch attacks. The authors empirically show that patch attacks rely on localized Superficial Important Neurons (SINs) to cause misclassification. By occluding image patches corresponding to these localized neurons, the predictions on patched images become unstable, while those of benign images rarely change. The proposed approach ScaleCert uses these SINs to check if a patch attack is present by occluding certain candidate regions and observing if the prediction changes. The proposed method leads to significant improvements in certified detection accuracy on ImageNet, while having lower computational cost, thus promoting its usage in the real world.
main_review: Strengths:
 -  While many existing certified detection methods (MRD, PatchGuard++) are based on the idea of occluding patches in the input image, and observing the stability of the output prediction, the key insight in this work is in the SIN-based pruning step, where top-k neurons are retained and the rest are pruned. This improves the scalability of the approach and also leads to less noise in prediction labels, improving the certified accuracy significantly at a low computational cost
 -  The results show huge improvements in certified detection accuracy on ImageNet, achieving 60.6% against 1%-pixel patches, significantly surpassing the highest record of 36.3% in current state-of-the-art methods. The work also doesn’t incur a huge computational cost compared to other defenses, which is beneficial for real-world adoption.
 -  The paper is clearly written and well organized. It provides enough information for the reader to reproduce its results. 

Questions for rebuttal/ suggestions for improvement:
 -  Could the authors share the execution latency on Imagenet for PatchGuard++ as well, which can be compared with the results in Table-4? 
 -  Could the authors clarify how the top-k neurons are found for a given layer? Is it an average/ max across all activations at that layer? 
 -  The process of certification and inference seem to be quite similar. For a clean input image, if occlusion prediction gives consistent outputs and if the prediction is same as the true label, it is said to be certified. The same process is also used during inference of a clean image for prediction. So does this mean that the difference between clean and certified accuracy is from images whose predictions are consistent and incorrect? It would be worth visualizing such images to see if they are confusing visually as well. 
 -  Adding more detailed captions to tables and figures would improve the readability. The x-axis label for Fig.1(e) is missing. 
 -  It could be made more clear throughout the paper that this is a certified detection method rather than a certified prediction/recovery method, perhaps in the title as well. It seems unfair to directly compare certified accuracy of recovery methods to detection methods as it’s easier to detect whether a patch is present than it is to predict the correct class in any patched image. A note on why this method cannot be used for certified prediction would also help the reader understand the same better. 

Limitations/ Possible concerns: 
 -  As seen in Table-5 of the Supplementary, clean accuracy drops significantly for 5% and 8% pixel patch attacks. 
 -  The process of selecting only top-k neurons for inference could potentially make the network more vulnerable to minor distribution shifts such as addition of gaussian noise/ other corruptions when compared to a normally trained model. 


**Update Post-Rebuttal**

I thank the authors for their detailed replies to clarify all concerns. Although the work is novel and shows a large improvement over prior work, there are many unclear parts and missing details, specifically about the certification, due to which I lower my score to 6. 

limitations_and_societal_impact: No

Limitations: The proposed method detects the presence of a patch, and does not recover the true label.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to leverage superficial important neurons (SIN) to prune DNN to reduce the sensitivity of DNN to adversarial patches, thus resulting in certified defense of adversarial patches. The approach outperforms existing approaches largely on large ImageNet dataset.
main_review: Pros:
- The usage of SIN, or general interpretation/compression technique, for certified defense is quite novel.

- The empirical evidence that motivates the method seems compelling.

- Strong empirical results.

Cons:
- Method: 
1. The proposed method, ScaleCert, is an attack detection method that does not aim to recover from the attack. The certification counts the input benign example as "certified" as long as the method outputs the correct label or raises an alert. Thus, in contrast to attack recovery methods, the proposed method is less useful in practice than what the certification result indicates since it may be too conservative for slightly patched inputs.

2. If I understand correctly, the SIN mask should be static given a model? Does it mean that the model only uses sparse, scattered, and tiny regions to sense the whole image space? If this is the case, how does the masked model preserve high benign accuracy given diverse key regions across different benign input images? Is the SIN mask generated only based on training dataset statistics?

3. The approach is limited to CNNs.

- Empirical result: 
1. As shown in Table 2, the method is not as good as existing methods on low-resolution image datasets like CIFAR-10.

2. It would enhance the paper if there are more studies about the empirical patch detection ability, given that Line 14 in Algorithm 1 provides the ability to locate the patch.

- Writing: The details on SIN is lacking in this paper. For example, what is the formal definition of SIN? How does SIN determine what neurons to mask? How does the static SIN mask be generated in experiments?

Minor: Line 348, "sparese" => "sparse"

In summary, given the strong empirical results, interesting findings, and novel techniques, I weekly support the acceptance of the paper. If my concerns can be adequately addressed, especially those on SIN, I would consider raising my score.
limitations_and_societal_impact: The authors adequately addressed the limitations.
The authors may need to include the discussion about the potential negative societal impact of their work if the manuscript is accepted.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed an algorithm called ScaleCert that attempted to improve the accuracy of certified detection against large-resolution adversarial samples. The experimental analysis about the motivation in the paper is great, the logic is clear and easy to understand, and the experiments prove the efficiency (time consuming) of the method.
main_review: 1.	Although the related work mentions attack recovery and robust detection, the title as well as the abstract make this work appear not to be doing adversarial detection and may mislead researchers in the field of adversarial machine learning.
2.	Although the authors' analysis is better, I still have doubts about whether ScaleCert is provable. On the one hand, some adversarial patches can make the activation values of the network concentrated in the wrong places, such as [1], and it is possible to circumvent detection; on the other hand, there is no guarantee that the pruned network can always classify the images correctly after masking different locations, that is, the adversary can carefully design patches to lower the activation values to circumvent detection, but make the non-patched regions wrongly classified after being masked. And according to the algorithm, the patch may be detected, but causes a false alarm, and such a patch is harmful to the subsequent processing. Summing up the above two points I think ScalCert is a detection with high security for general attacks, but not provable.
3.	Why is the clear ACC of 2% size patch lower than the 3% case in the ImageNet experiment in Table 2? Detection accuracy should not be compared to the accuracy of robust defense, which are two completely different tasks. Experiments should give AUC values and not just detection accuracy. These experiments make me unconvinced of the validity of the method.


[1] Subramanya, Akshayvarun, Vipin Pillai, and Hamed Pirsiavash. 2019. "Fooling network interpretation in image classification." In Proceedings of the IEEE International Conference on Computer Vision, 2020-29.

limitations_and_societal_impact: They have mentioned the limitations in the Appendix; the certified accuracy of ScaleCert maybe a little bit worse than the original accuracy. I don't see that is a limitiation to me.
review_title: null

Meta-review: The authors develop a novel scalable robustness certification technique to certify robustness of deep networks to adversarial patches. The paper provides a theoretically and empirically interesting contribution to the literature on robustness and verification of deep networks. 

In the initial reviews, several concerns were raised around the correctness of the proposed approach. These were adequately addressed in the rebuttal. The paper would be acceptable for publication provided that the authors make the revisions suggested by the reviewers during the rebuttal phase in the camera ready version.

Specific detailed feedback from the reviewers on how to improve the paper:

1. L254-255 in the main paper needs to be rephrased to indicate that only sliding windows that overlap with  (corresponding to the benign image) do not affect prediction results for the image . Justification of why this is sufficient for certification needs to be included. This could include the figure shared by the authors in the rebuttal to illustrate that the top-k region of the attacked image cannot differ by more than a fixed amount from the top-k region of the original image. In addition, for the cases when the patch is occluded completely, the top-k region of the attacked image would be a subset of the top-k region of the benign image. This can be formally stated and proved.
2. It could be made more clear throughout the paper that this is a certified detection method rather than a certified prediction/ recovery method, perhaps in the title as well. It seems unfair to directly compare certified accuracy of recovery methods to detection methods as it’s easier to detect whether a patch is present than it is to predict the correct class in any patched image. A note on why this method cannot be used for certified prediction would also help the reader understand the same better.
3. Results on execution latency on ImageNet for PatchGuard++ (shared in the rebuttal) could be included
4. Details on how the top-k neurons are computed could be included
5. Adding detailed captions to tables and figures would improve the readability. The x-axis label for Fig.1(e) is missing
6. A discussion on false alarms and what is done to reduce this could be included.
7. Clearly define SINs and states the important properties of SINs, e.g., what is the SIN of the masked region?
8. Clearly and formally define the inference protocol. It seems to be first computing SINs then dynamically pruning the network to only pass the signals for top-k SIN neurons. The final inference protocol that takes windows into account runs multiple  inferences independently --- the top-k SINs are independently computed for each masked input.
9. A detailed proof of Theorem 1 with more illustrations.
10. Training details for reproducibility. The current submission lacks many training details nor contains code implementation. Please make sure to include them especially the description of training details.
11. Compress Section 3.1 to gain space.
12. Authors should include the discussion on the drop in performance for low-resolution datasets and the guidelines for superficial layer selection.
13. Comparison of provable detection and provable recovery should not be placed in the main text. The comparison is meaningless due to the difference in settings.
14. The statement related to the algorithm is not very clear. I suggest that the authors clearly explain the algorithm in future revisions. In addition, Line 246-257 is copied from PatchGuard++, which is not allowed.
15. Reporting error bars of results (shared in the rebuttal)
16. A discussion on limitations could be included: sub-optimal performance on low-resolution datasets, a large drop in clean accuracy with large patch attacks, and limited generalization (only works for image CNNs due to exploiting localized low-level features).
Decision: Accept (Poster)