Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

DetectorGuard: Provably Securing Object Detectors
against Localized Patch Hiding Attacks
Chong Xiang

Prateek Mittal

Princeton University
Princeton, NJ, USA
cxiang@princeton.edu

Princeton University
Princeton, NJ, USA
pmittal@princeton.edu

KEYWORDS

ABSTRACTState-of-the-art object detectors are vulnerable to localized patch
hiding attacks, where an adversary introduces a small adversarial
patch to make detectors miss the detection of salient objects. The
patch attacker can carry out a physical-world attack by printing
and attaching an adversarial patch to the victim object; thus, it imposes a challenge for the safe deployment of object detectors. In thispaper, we propose DetectorGuard as the first general framework forbuilding provably robust object detectors against localized patch
hiding attacks. DetectorGuard is inspired by recent advancements
in robust image classification research; we ask: can we adapt robust image classifiers for robust object detection? Unfortunately, due
to their task difference, an object detector naively adapted from arobust image classifier 1) may not necessarily be robust in the adversarial setting or 2) even maintain decent performance in the cleansetting. To address these two issues and build a high-performancerobust object detector, we propose an objectness explaining strategy:
we adapt a robust image classifier to predict objectness (i.e., theprobability of an object being present) for every image location andthen explain each objectness using the bounding boxes predictedby a conventional object detector. If all objectness is well explained,
we output the predictions made by the conventional object detector; otherwise, we issue an attack alert. Notably, our objectness
explaining strategy enables provable robustness for “free": 1) in theadversarial setting, we formally prove the end-to-end robustness ofDetectorGuard on certified objects, i.e., it either detects the object
or triggers an alert, against any patch hiding attacker within our
threat model; 2) in the clean setting, we have almost the same performance as state-of-the-art object detectors. Our evaluation on the
PASCAL VOC, MS COCO, and KITTI datasets further demonstrates
that DetectorGuard achieves the first provable robustness against
localized patch hiding attacks at a negligible cost (<1%) of clean
performance.

Provable Robustness; Adversarial Patch Attack; Object Detection
ACM Reference Format:
Chong Xiang and Prateek Mittal. 2021. DetectorGuard: Provably Securing
Object Detectors against Localized Patch Hiding Attacks. In Proceedings ofthe 2021 ACM SIGSAC Conference on Computer and Communications Security(CCS ’21), November 15–19, 2021, Virtual Event, Republic of Korea. ACM, New
York, NY, USA, 20 pages. https://doi.org/10.1145/3460120.3484757

1

INTRODUCTION

Localized adversarial patch attacks can induce mispredictions in Machine Learning (ML) systems and have gained significant attention
over the past few years [4, 21, 50, 57, 61]. A patch attacker constrains
all adversarial perturbations within a small region so that they can
carry out a physical world attack by printing and attaching the adversarial patch to the victim object.1 To counter the threat of patch
attacks on real-world ML systems, the security community has
been actively seeking defense mechanisms [9, 18, 24, 25, 33, 58, 65].
However, most existing defenses are restricted to the image classification domain. In this paper, we aim to secure object detectors,
which are used in critical applications like autonomous driving,
video surveillance, and identity verification [52].
We focus on the threat of localized patch hiding attacks against
object detectors: an attacker uses a localized patch for physicalworld attacks that cause the object detector to fail to detect victim
objects. Lee et al. [23] show that a physical patch far away fromthe objects can successfully “hide" victim objects. Wu et al. [57]and Xu et al. [61] have succeeded in evading object detection via
wearing a T-shirt printed with adversarial perturbations. The patch
hiding attack can cause serious consequences in scenarios like anautonomous vehicle missing a pedestrian (or an upcoming car).
Unfortunately, securing object detectors is extremely challenging
due to the complexity of the detection task. A single image can contain multiple objects, and thus an object detector needs to output alist of object bounding box coordinates and class labels. To the bestof our knowledge, there is only one prior work [46] discussing defenses for YOLOv2 [43] detectors against patch attacks, in contrast
to numerous new patch attacks being proposed [7, 14, 50, 57, 61, 66].
Furthermore, this only defense [46] is restricted to the setting of a
non-adaptive adversarial patch at the image corner and does nothave any security guarantee (i.e., only heuristics-based). To overcome these weaknesses, we propose a defense framework named
DetectorGuard that can achieve provable robustness against any
patch hiding attack within our threat model.

CCS CONCEPTS• Security and privacy → Domain-specific security and privacy architectures; • Computing methodologies → Object detection; Neural networks.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea© 2021 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-8454-4/21/11.
https://doi.org/10.1145/3460120.3484757

1 The patch attack significantly differs from classic 𝐿 -norm-bounded adversarial𝑝examples [6, 16, 48] that require global perturbations and thus are difficult to realizein the physical world.

3177

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Adversarial
Setting

Clean
Setting
dog

dog

dog

dog

dog

dog

Detection Output

Input Image(clean)Objectness
Predictor

ALERT!

Base Detector

Base Detector

Detection Output

Input Image(adversarial)
Objectness
Predictor

Objectness Explainer

Objectness Explainer

Figure 1: DetectorGuard Overview. Base Detector is a conventional detector that typically predicts precise bounding boxes on clean images.
Objectness Predictor aims to robustly predict an objectness map. Objectness Explainer uses the predicted bounding boxes to explain/match thepredicted objectness and determines the final output. In the clean setting (left figure), the dog on the left is detected by both Base Detector andObjectness Predictor. The objectness is well explained/matched by the green bounding box, and DetectorGuard outputs the bounding box
predicted by Base Detector. In the meantime, due to the imperfection of the robust classifier, the dog on the right is missed by Objectness
Predictor. Objectness Explainer will consider this as a benign mismatch, and DetectorGuard will also output the predicted bounding box fromBase Detector. In the adversarial setting (right figure), a patch makes Base Detector fail to detect any object while Objectness Predictor stillrobustly outputs high objectness. Objectness Explainer detects a cluster of unexplained objectness and triggers an attack alert. Our defense
can achieve provable robustness for certified objects while maintaining a clean performance comparable to state-of-the-art object detectors.
Insight & Challenge: a bridge between robust image classification and robust object detection. To start with, we observethat the robust image classification research is making significant
advancement [9, 24, 33, 36, 58, 65] while object detectors remain
vulnerable to attacks. This sharp contrast motivates us to ask: canwe take advantage of robust image classification research and adapt
robust image classifiers for robust object detection? Unfortunately,
there is a huge gap between these two tasks: an image classifier
outputs a single-label prediction while an object detector predicts alist of bounding boxes and class labels. This gap brings two major
challenges for such an adaptation. Challenge 1: Lack of End-to-end
Provable Robustness. A robust image classifier only provides robustness for a single-label prediction while an object detector requires
robustness for multiple object labels and bounding boxes in each
image. Therefore, using a robust image classifier does not guarantee
the end-to-end robustness of the object detector, and we need a
new defense design and proof of robustness. Challenge 2: Amplified
Cost of Clean Performance. All existing provably robust image classifiers [9, 24, 36, 58, 65] attain robustness at a non-negligible cost ofclean performance (e.g., state-of-the-art defense [58] incurs a >20%
clean accuracy drop on ImageNet [11]). The imperfection of robust
classifiers can be severely amplified during the adaptation towards
the much more demanding object detection task. Therefore, weneed to prevent our object detectors from being broken in the cleansetting (even in the absence of an adversary). In DetectorGuard, weovercome these two challenges as discussed below.
Defense Design: an objectness explaining strategy. We provide our defense overview in Figure 1. DetectorGuard has threemodules: Base Detector, Objectness Predictor, and Objectness Explainer. Base Detector can be any state-of-the-art object detector
that can make accurate predictions on clean images but is vulnerable
to patch hiding attacks. Objectness Predictor aims to predict a robust
objectness map, which indicates the probability of an object being

present at different locations. We build Objectness Predictor using
adapted provably robust image classifiers together with carefully
designed feature-space operation and error filtering mechanisms(Section 3.3). Finally, Objectness Explainer uses each predictedbounding box from Base Detector to explain/match high objectness
predicted by Objectness Predictor (Section 3.4). If all objectness iswell explained/matched, we output the prediction of Base Detector;
otherwise, we issue an attack alert. In the clean setting, we optimize
the configuration of Objectness Predictor towards the case where
all objectness can be explained and then use Base Detector for accurate final predictions (Figure 1 left). When a hiding attack occurs,Base Detector could miss the object while Objectness Predictor canstill robustly output high objectness. This will lead to unexplained
objectness and trigger an attack alert (Figure 1 right). Notably, we
can show that our defense design successfully addresses Challenge
1 and 2, as discussed next.
End-to-end provable robustness for “free". First, our objectness explaining strategy enables us to rigorously prove the endto-end robustness of DetectorGuard (Theorem 1 in Section 4). We
will show that DetectorGuard will always perform robust detection or issue an alert on objects certified by our provable analysis(Algorithm 2 in Section 4). We note that this robustness property
is agnostic to attack strategies and holds for any patch hiding attacker within our threat model, including adaptive attackers who havefull access to our defense setup. This strong theoretical guarantee
addresses Challenge 1. Next, in contrast to most security-criticalsystems whose robustness comes at the cost of clean performance,
DetectorGuard achieves provable robustness for “free" (at a negligible cost of clean performance). In Objectness Predictor, we design
error mitigation mechanisms to handle the imperfection of theadapted robust classifier. In Objectness Explainer, our explaining
strategy ensures that even when our Objectness Predictor fails to3178

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

predict high objectness (missing objects; false negatives), DetectorGuard still performs as well as state-of-the-art Base Detector.2
These designs solve Challenge 2.
Evaluating the first provable robustness against patch hiding attacks. We extensively evaluate DetectorGuard performanceon the PASCAL VOC [13], MS COCO [26], and KITTI [15] datasets.
In our evaluation, we instantiate the Base Detector with YOLOv4 [2,53], Faster R-CNN [45], and a hypothetical object detector thatis perfect in the clean setting. We build Objectness Predictor byadapting multiple variants of robust image classifiers [58, 65]. Ourevaluation shows that our defense has a minimal impact (<1%)
on the clean performance and achieves the first provable robustness against patch hiding attacks. Our code is available at https://github.com/inspire-group/DetectorGuard.
Our contributions can be summarized as follows.

detection directly on the input image without any explicit region
proposal step. SSD [29], YOLO [2, 42–44, 53], RetinaNet [25], andEfficientDet [49] are representative one-stage detectors.
Conventionally, a detection is considered correct when 1) the
predicted label matches the ground truth and 2) the overlap between
the predicted bounding box and the ground-truth box, measured
by Intersection over Union (IoU), exceeds a certain threshold 𝜏. We
term a correct detection a true positive (TP). On the other hand, any
predicted bounding box that fails to satisfy both two TP criteria isconsidered as a false positive (FP). Finally, if a ground-truth object
is not detected by any TP bounding box, it is a false negative (FN).

2.2• We solve two major challenges in adapting robust image
classifiers for robust object detection via a careful design of
Objectness Predictor and Objectness Explainer.
• We formally prove the robustness guarantee of DetectorGuardon certified objects against any adaptive attacker within our
threat model.
• We extensively evaluate our defense on the PASCAL VOC [13],MS COCO [26], and KITTI [15] datasets and demonstrate thefirst provable robustness against patch hiding attacks and asimilar clean performance as conventional object detectors.

2

BACKGROUND AND PROBLEM
FORMULATIONIn this section, we introduce the object detection task, the patch
hiding attack, the defense formulation, and the key principles forbuilding provably robust image classifiers that we will adapt forrobust object detection in DetectorGuard.

2.1

Object Detection

Detection objective. An object detector aims to predict a listof bounding boxes (and class labels) for all objects in the imagex ∈ [0, 1]𝑊 ×𝐻 ×𝐶 , where pixel values are rescaled into [0, 1], and𝑊 , 𝐻, 𝐶 is the image width, height, and channel, respectively. Eachbounding box b is represented as a tuple (𝑥 min, 𝑦min, 𝑥 max, 𝑦max, 𝑙),where 𝑥 min, 𝑦min, 𝑥 max, 𝑦max together illustrate the coordinates ofthe bounding box, and 𝑙 ∈ L = {0, 1, · · · , 𝑁 − 1} denotes the predicted object label (𝑁 is the number of object classes).3
Conventional object detector. Object detection models canbe categorized into two-stage and one-stage detectors dependingon their pipelines. A two-stage object detector first generates proposals for regions that might contain objects and then uses the
proposed regions for object classification and bounding-box regression. Representative examples include Faster R-CNN [45] and Mask
R-CNN [19]. On the other hand, a one-stage object detector does

Attack Formulation

Attack objective. The hiding attack [30, 50, 57, 61, 66], also referred to as the false-negative (FN) attack, aims to make object
detectors miss the detection of certain objects (which increases FN)
at the test time. The hiding attack can cause serious consequences inscenarios like an autonomous vehicle missing a pedestrian. Therefore, defending against patch hiding attacks is of great importance.
Attacker capability. We allow the localized adversary to arbitrarily manipulate pixels within one restricted region.4 Formally,
we can use a binary pixel mask pm ∈ {0, 1}𝑊 ×𝐻 to represent thisrestricted region, where the pixels within the region are set to 1.
The adversarial image then can be represented as x ′ = (1 − pm) ⊙x + pm ⊙ x ′′ where ⊙ denotes the element-wise product operator, and x ′′ ∈ [0, 1]𝑊 ×𝐻 ×𝐶 is the content of the adversarial patch,which the adversary can arbitrarily modify. pm is a function ofpatch size and patch location. The patch size should be limited suchthat the object is recognizable by a human. For patch locations,
we consider three different threat models: over-patch, close-patch,
far-patch, where the patch is over, close to, or far away from the victim object, respectively. The adversary can pick any valid location
within the threat model for an optimal attack.
Previous works [23, 30, 46] have shown that attacks against
object detectors can succeed even when the patch is far away fromthe victim object. Therefore, defending against all three threat
models is of interest.

2.3

Defense Formulation

Defense objective. We focus on defending against patch hiding
attacks. We consider our defense to be robust on an object if we can1) detect the object on the clean image is correct and 2) detect partof the object or send out an attack alert on the adversarial image.5
Crucially, we design our defense to be provably robust: for an
object certified by our provable analysis, our defense can either
detect the certified object or issue an alert regardless of what theadversary does (including any adaptive attack at any patch location
within the threat model). This robustness property is agnostic to
the attack algorithm and holds against an adversary that has fullaccess to our defense setup.

2 Objectness Predictor can also have other types of errors. However, we can optimize
4 Provably robust defenses against one single patch are currently an open/unsolved

its configuration to ensure most errors are false-negatives, which our objectness
explaining strategy can tolerate. More discussions are in Section 3. We also note that
we manage to build a system with high clean performance (i.e., DetectorGuard) despite
the use of a module with poor clean performance (i.e., provably robust image classifier).
We provide additional discussion on this intriguing property in Appendix F.
3 Conventional object detectors usually output objectness score and prediction confidence as well—we discard them in notation for simplicity.

problem, and hence the focus of this paper. In Appendix C, we will justify our one-patch
threat model and quantitatively discuss the implication of multiple patches.
5 We note that in the adversarial setting, we only require the predicted bounding box
to cover part of the object because it is likely that only a small part of the object isrecognizable due to the adversarial patch (e.g., the left dog in the right part of Figure 1).
We provide additional justification for our defense objective in Appendix E.

3179

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Remark: primary focus on hiding attacks. In this paper, we
focus on the hiding attack because it is the most fundamental andnotorious attack against object detectors. We can visualize dividing
the object detection task into two steps: 1) detecting the object
bounding box and then 2) classifying the detected object. If thefirst step is compromised by the hiding attack, there is no hope forrobust object detection. On the other hand, securing the first stepagainst the patch hiding attack lays a foundation for the robust object
detection; we can design effective remediation for the second step
if needed (Section 6).
Take the application domain of autonomous vehicles (AV) as an
example: an AV missing the detection of an upcoming car could leadto a serious car accident. However, if the AV detects the upcoming
object but predicts an incorrect class label (e.g., mistaking a carfor a pedestrian), it can still make the correct decision of stopping
and avoiding the collision. Moreover, in challenging application
domains where the predicted class label is of great importance (e.g.,traffic sign recognition), we can feed the detected bound box toan auxiliary image classifier to re-determine the class label. The
defense problem is then reduced to the robust image classificationand has been studied by several previous works [24, 36, 58, 65].
Therefore, we make the hiding attack the primary focus of thispaper and will also discuss the extension of DetectorGuard against
other attacks in Section 6.

3.1

2.4

• Challenge 1: Lack of End-to-end Provable Robustness. A robust
image classifier only provides robustness for single-label predictions while a robust object detector requires robustness
for multiple labels and bounding boxes. Therefore, an object
detector adapted from a robust image classifier can still bevulnerable without any security guarantee, and we aim tocarefully design our defense pipeline to enable the proof ofend-to-end robustness for object detection.
• Challenge 2: Amplified Cost of Clean Performance. All existingprovably robust image classifiers [9, 24, 36, 58, 65] attainrobustness at a non-negligible cost of clean performance(e.g., >20% clean accuracy drop on ImageNet [11]), and thiscost can be severely amplified when adapting towards themore demanding object detection task. An object detector
with poor clean performance (even in the absence of an
adversary) prohibits its real-world deployment; therefore, we
aim to minimize the clean performance cost in our defense.

Provably Robust Image Classification

In this subsection, we introduce two key principles that are widelyadopted in recent research on provably robust image classification against adversarial patches [24, 36, 58, 65]. In Section 3.2, wewill discuss how to adapt these two principles to build a robust
image classifier, which will be alter used in Objectness Predictor(Section 3.3).
Feature extractor – use small receptive fields. The receptive
field of a Deep Neural Network (DNN) is the input pixel region
where each extracted feature is looking at. If the receptive field of a
DNN is too large, then a small adversarial patch can corrupt most
extracted features and easily manipulate the model behavior [30, 46,58]. On the other hand, the small receptive field bounds the numberof corrupted features by ⌈(p+r−1)/s⌉, where p is the patch size, r is
the receptive field size, and s is the stride of receptive field (the pixel
distance between two adjacent receptive field centers) [58], andmakes robust classification possible [24, 36, 58, 65]. Popular design
choices including the BagNet architecture [3, 36, 58, 65] and an
ensemble architecture using small pixel patches as inputs [24, 58].
Classification head – do secure feature aggregation. Givena feature map, DNN uses a classification head, which consists of afeature aggregation layer and a fully-connected (classification) layer,
to make final predictions. Since the small receptive field bounds
the number of corrupted features, we can use secure aggregation
techniques to build a robust classification head; design choices
include clipping [58, 65], masking [58], and majority voting [24, 36].

3

Defense Overview

Bridging robust image classification and robust object detection. There has been a significant advancement in (provably) robust
image classification research [9, 24, 36, 58, 65] while object detectors remain vulnerable. This sharp contrast motivates us to ask: canwe adapt robust image classifiers for robust object detection? Unfortunately, there is a huge gap between these two tasks: an image
classifier only robustly predicts one single label for each image
while an object detector has to robustly output a list of class labelsand object bounding boxes. This gap leads to two major challenges.

DetectorGuard: an objectness explaining strategy. In DetectorGuard, we propose an objectness explaining strategy to addresses the above two challenges. Recall that Figure 1 provides an
overview of DetectorGuard, which will either output a list bounding box predictions (left figure; clean setting) or an attack alert(right figure; adversarial setting). There are three major modules in
DetectorGuard: Base Detector, Objectness Predictor, and Objectness
Explainer. Base Detector is responsible for making accurate detections in the clean setting and can be any popular high-performance
object detector such as YOLOv4 [2, 53] and Faster R-CNN [45]. Objectness Predictor is adapted from the core principles for building
robust image classifiers as introduced in Section 2.4 and aims to output a robust objectness map in the adversarial environment. We also
carefully design Objectness Predictor to mitigate the errors made by
the robust image classifier in the clean setting. Finally, Objectness
Explainer leverages predicted bounding boxes from Base Detector
to explain/match the objectness predicted by Objectness Predictor
and aims to catch a malicious attack. When no attack is detected,
DetectorGuard will output the detection results of Base Detector(i.e., a conventional object detector), so that our clean performanceis close to state-of-the-art object detectors. When a patch hiding
attack occurs, Base Detector can miss the object while Objectness
Predictor can robustly predict high objectness. This mismatch willlead to unexplained objectness and trigger an attack alert. Notably,
our objectness explaining strategy can achieve end-to-end provablerobustness for “free" (at a negligible cost of clean performance) andsolve two major challenges. We will introduce the module details
and theoretically analyze the free provable robustness property.

DETECTORGUARDIn this section, we first introduce the key insight and overview ofDetectorGuard, and then detail the design of our defense modules(Objectness Predictor and Objectness Explainer).

3180

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Table 1: Summary of important notation
Notation

Description

Notationx
fmv(𝑤 𝑥 , 𝑤 𝑦 )𝑇u, lInput imagebbounding box
Feature mapomObjectness map
classification logits𝑁number of object classes
window size
(𝑝𝑥 , 𝑝 𝑦 )patch size
binarizing threshold D
detection results
upper/lower bound of classification logits values of each class

Robust feature-space window classification

Robustfeatureextraction

Objectness map generation (with error filtering)𝟎𝟎𝟎

Max element(ignoring background)

Objectness
map

Algorithm Pseudocode. We provide the pseudocode of DetectorGuard in Algorithm 1 and a summary of important notationin Table 1. The main procedure DG(·) has three sub-procedures:
BaseDetector(·), ObjPredictor(·), and DetMatcher(·). The subprocedure BaseDetector(·) can be any off-the-shelf object detector as discussed in Section 2.1. All tensors/arrays are represented
with bold symbols and scalars are in italic. All tensor/array indicesstart from zeros; the tensor/array slicing is in Python style (e.g.,[𝑖 : 𝑗] means all indices 𝑘 satisfying 𝑖 ≤ 𝑘 < 𝑗). We assume that the“background" class corresponds to the largest class index.
In the remainder of this section, we first introduce how we instantiate robust image classifiers and then discuss the design of
Objectness Predictor and Objectness Explainer.

Dog

𝟏
𝟏 Cat
𝟎
𝟎
𝟐
𝟎 Back𝟏
𝟖𝟓
𝟖
𝟓
𝟖 ground
𝟏
𝟕
𝟎
𝟏
𝟔
𝟓
Robust𝟐 𝟖 𝟔 𝟎𝟎 𝟏
classification 𝟓
𝟎𝟏

Description

𝟎
𝟏

𝟏

𝟏
𝟏

𝟐
𝟏
𝟎 Binarization 𝟏
𝟏
𝟖
𝟓
𝟐
𝟔
𝟔

Figure 2: Visualization of Objectness Predictor operations

Overview. Objectness Predictor aims to output a robust objectness map that indicates the probability of objects being present at
different locations. Its high-level idea is to perform robust image

classification on different regions to predict an object class label or“background". We provide a simplified visual overview in Figure 2.
Objectness Predictor involves two major operations: robust featurespace window classification and objectness map generation. The
first step aims to perform efficient robust classification at different
image regions and the second step aims to filter out clean errors
made by the robust classifier and generate the final objectness map.
Robust feature-space window classification. To perform robust image classification at different image locations, we first extract
a feature map via FE(·), and then apply the robust image classification head RCH(·) to a sliding window over the feature map.
Pseudocode. The pseudocode of Objectness Predictor in Line 1223 of Algorithm 1. We first extract the feature map fm with FE(·)(Line 13). Next, for every valid window location, represented as(𝑖, 𝑗),7 we feed the feature window fm[𝑖 : 𝑖 + 𝑤 𝑥 , 𝑗 : 𝑗 + 𝑤 𝑦 ] to arobust classification head RCH(·) to get the classification label 𝑙and the classification logits v ∈ R𝑁 +1 for 𝑁 object classes and the“background" class (Line 17). The use of RCH(·) ensures that theclassification is robust when window features are corrupted.
Remark: defense efficiency. We note that our window classification operates in the feature space, and this allows us to reuse theexpensive feature map generation (i.e., FE(·)); each classification
only needs a cheap computation of the classification head (i.e.,
RCH(·)). Therefore, our defense only incurs a small overhead (willbe evaluated in Section 5.5).
Objectness map generation: handling clean errors of robust classifiers. Next, we aim to filter out incorrect window classifications and generate the final objectness map. We reduce eachprediction vector at each location to its maximum non-backgroundelement to discard label information (eliminating Clean Error 1)
and perform binarization (with a threshold) to remove objectness
predicted with low confidence (mitigating Clean Error 2).
Pseudocode. First, we initialize an all-zero R𝑁 +1 vector (𝑁 object
classes plus the “background" class) at every feature location and¯ to represent all vectors (Line 15). Second, we
use one tensor om
aim to gather all window classification results: for each window,
we add the classification logits v to every vector located within thewindow (Line 18). Third, we take the maximum non-backgroundelement in each vector as the objectness score at the corresponding

6 The local logits vector [58] is the classification logits based on each local feature that

7 We will use feature-space coordinates for the remainder of the paper. The mapping

is extracted from a particular region (i.e., the receptive field) of the input image.

between pixel-space and feature-space coordinates is discussed in Appendix G.

3.2

Instantiating Robust Image Classifiers

To start with, we discuss how we build robust image classifiers thatwill be used in Objectness Predictor.
As discussed in Section 2.4, we can build a robust image classifierRC(·) using a feature extractor FE(·) with small receptive fields, anda robust classification head RCH(·) with secure feature aggregation.
In our design, we choose BagNet [3] backbone as the feature extractor FE(·), and we clip elements of local logits vectors6 into [0, ∞]for secure aggregation in RCH(·). This implementation is similar
to the robust image classifier Clipped BagNet (CBN) [65], but wenote that we use a different clipping function that is tailored to ourmore challenging task of object detection. In Appendix B, we provide
additional details of FE(·) and RCH(·) and also discuss alternative
design choices of robust image classifier RC(·) (e.g., robust maskingfrom PatchGuard [58]).
Remark: Limitations of robust classifiers. We note that theadapted robust image classifier RC(·) achieves robustness at the costof a non-negligible clean performance drop [24, 36, 58]. Therefore,
in the clean setting, classification at different image locations canbe imprecise with three typical errors that lead to Challenge 2:• Clean Error 1: Confusion between two object class labels• Clean Error 2: Predicts background pixels as objects• Clean Error 3: Predicts objects as “background"
In the next two subsections, we will discuss how DetectorGuard
design can eliminate/mitigate these three clean errors.

3.3

Objectness Predictor

3181

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Base Detector Objectness Predictor

Algorithm 1 DetectorGuard
Input: input image x, window size (𝑤 𝑥 , 𝑤 𝑦 ), binarizing threshold 𝑇 , Base Detector BaseDetector(·), robust classifier RC(·)(consisting of a feature extractor FE(·) and a robust classification head RCH(·)), cluster detection procedure DetCluster(·)
Output: robust detection D ∗ or ALERT
1: procedure DG(x, 𝑤 𝑥 , 𝑤 𝑦 ,𝑇 )2:
D ← BaseDetector(x)⊲ Conventional detection
3:om ← ObjPredictor(x, 𝑤 𝑥 , 𝑤 𝑦 ,𝑇 )⊲ Objectness
4:𝑎 ← ObjExplainer(D, om)⊲ Detect hiding attacks
5:
if 𝑎 == True then
⊲ Malicious mismatch
6:
D ∗ ← ALERT⊲ Trigger an alert
7:
else
8:
D∗ ← D⊲ Return Base Detector’s predictions
9:
end if10:
return D ∗11: end procedure

DetectorGuard Output

Match

Malicious
mismatch

1

ALERT!

Benignmismatch

Figure 3: Visualization of explaining/matching rules

classification confidence. We discuss the strategy for Clean Error 3
in the next subsection.

12: procedure ObjPredictor(x, 𝑤 𝑥 , 𝑤 𝑦 ,𝑇 )fm ← FE(x)⊲ Extract feature map
𝑋, 𝑌 , _ ← Shape(fm)⊲ Get the shape of fm¯ ← ZeroArray[𝑋, 𝑌 , 𝑁 + 1]
15:om
⊲ Initialization
16:
for each valid (𝑖, 𝑗) do⊲ Every window location
17:
𝑙, v ← RCH(fm[𝑖 : 𝑖 + 𝑤 𝑥 , 𝑗 : 𝑗 + 𝑤 𝑦 ])⊲ Classify⊲ Add classification logits¯ : 𝑖 +𝑤 𝑥 , 𝑗 : 𝑗 +𝑤 𝑦 ] ← om[𝑖¯ : 𝑖 +𝑤 𝑥 , 𝑗 : 𝑗 +𝑤 𝑦 ] + v
18:
om[𝑖
19:
end for¯ ← MaxObj( om,¯ axis = −1) ⊲ Max objectness score
20:
om
¯
21:
om ← Binarize( om,𝑇· 𝑤𝑥 · 𝑤 𝑦 )⊲ Binarization
22:
return om
23: end procedure
13:

3.4

14:

Objectness Explainer

Objectness Explainer takes as inputs the predicted bounding boxesof Base Detector and the generated objectness map of Objectness Predictor, and tries to use each predicted bounding box to
explain/match the high activation in the objectness map. Its outcome determines the final prediction of DetectorGuard. We will
first introduce the high-level explaining/matching rules and thenelaborate on the algorithm.
Explaining/matching rules. There are three possible explaining/matching outcomes, each of them leading to a different prediction strategy (a visual example is in Figure 3):• A match happens when Base Detector and Objectness Predictor both predict a bounding box or high objectness at a
specific location. In this simplest case, the objectness is well
explained by the bounding box; our defense will consider thedetection as correct and output the accurate bounding box
and the class label predicted by Base Detector.
• A malicious mismatch will be flagged when only Objectness Predictor outputs high objectness. This is most likely tohappen when a hiding attack fools the conventional object
detector to miss the object while our Objectness Predictor
still makes robust predictions. In this case, our defense willfind unexplained objectness and send out an attack alert.
• A benign mismatch occurs when only Base Detector detects the object and there is no objectness to be explained. Thiscan happen when Objectness Predictor incorrectly misses
the object due to its limitations (recall our remark in Section 3.3). In this case, we trust Base Detector and output its
predicted bounding box. Notably, this strategy can fully eliminate Clean Error 3, i.e., predicting objects as background.8

24: procedure ObjExplainer(D, om)ˆ ← Copy(om)om⊲ A copy of om⊲ Match each detected box to objectness map
26:
for 𝑖 ∈ {0, 1, · · · , |D | − 1} do
27:
𝑥 min, 𝑦min, 𝑥 max, 𝑦max, 𝑙 ← b ← D [𝑖]28:
if Sum(om[𝑥 min : 𝑥 max, 𝑦min : 𝑦max ]) > 0 thenˆ min : 𝑥 max, 𝑦min : 𝑦max ]) ← 0
29:
om[𝑥
30:
end if31:
end forˆ is None then32:
if DetCluster( om)33:return False⊲ All objectness explained
34:
else
35:
return True
⊲ Unexplained objectness36:
end if37: end procedure
25:

Next, we discuss the concrete procedure for explaining objectness.
Matching and explaining objectness. In Line 25-31 of Algorithm 1, we use each predicted bounding box to match/explain thelocation (Line 20). This operation discards label information andfully eliminates Clean Error 1, e.g., confusion between bicycles andmotorbikes. At last, we binarize every objectness score (Line 21):if the score is larger than 𝑇 · 𝑤 𝑥 · 𝑤 𝑦 , we set it to one; otherwise,
set it to zero. This binarization mitigates Clean Error 2, when theclassifier incorrectly predicts background as objects but with low

8 We note that this miss can also be caused by other attacks that are orthogonal to

the focus of this paper, e.g., FP attacks that aim to introduce incorrect bounding box
predictions. We will discuss such attacks and our defense strategies in Section 6.

3182

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

objectness predicted at the same location. First, we create a copy ofˆ to hold the explaining results. Next, for each boundingom as ombox b, we get its coordinates 𝑥 min, 𝑦min, 𝑥 max, 𝑦max , and calculate
the sum of objectness scores within the same box on the objectness
map om. If the objectness sum is larger than zero, we assume thatthe bounding box b agrees with om, and we zero out the correˆ to indicate that this region of objectness
sponding region in om,
has been explained by the detected bounding box. On the other
hand, if all objectness scores are zeros, we assume it is a benign
mismatch; the algorithm proceeds without alert.
Detecting clusters of unexplained objectness. The final step
ˆ We use the sub-procedure
is to detect unexplained objectness in om.
ˆ form
DetCluster(·) to determine if any non-zero points in om
a large cluster. Specifically, we choose DBSCAN [12] as the cluster detection algorithm, which will assign each point to a certain
cluster or label it as an outlier based on the point density in its
ˆ returns None, it means that noneighborhood. If DetCluster( om)large cluster is found, or all objectness predicted by Objectness
Predictor is explained by the bounding boxes predicted by Base
Detector; ObjExplainer(·) then returns False (i.e., no attack detected). We note that this clustering operation further mitigates
Clean Error 2 when the robust classifier predicts background as
objects at only a few scattered locations. On the other hand, receiving a non-empty cluster set indicates that there are clusters ofˆ (i.e, Base Detector misses
unexplained objectness activations in om
an object but Objectness Predictor predicts high objectness). Objectness Explainer regards this as a sign of patch hiding attacks andreturns True.
Final output. Line 5-10 of Algorithm 1 demonstrates the strategy for the final prediction. If the alert flag 𝑎 is True (i.e., a malicious
mismatch is detected), DetectorGuard returns D ∗ = ALERT. In othercases, DetectorGuard returns the detection D ∗ = D.
Remark: Clean performance of DetectorGuard. Recall that
Clean Error 1 of the robust classifier is fully eliminated in our objectness map generation via discarding label information; Clean
Error 2 is mitigated via binarizing (in Objectness Predictor) andclustering (in Objectness Explainer) operations; Clean Error 3 isfully tolerated via our prediction matching strategy (the benign
mismatch case). Therefore, we can safely optimize the setting ofDetectorGuard to mitigate most of Clean Error 2 (which can lead to
unexplained objectness in the clean setting and trigger a false alert)so that we can achieve a clean performance that is comparable tostate-of-the-art object detectors (performance difference smaller
than 1%; more details are in Section 5). This helps us solve Challenge 2: Amplified Cost of Clean Performance. In the next section, we
will demonstrate that our efforts in mitigating the imperfection ofrobust classifiers are worthwhile by showing how DetectorGuard
addresses Challenge 1: Lack of End-to-end Provable Robustness.

Algorithm 2 Provable Analysis of DetectorGuard

4

Input: input image x, window size (𝑤 𝑥 , 𝑤 𝑦 ), matching threshold𝑇 ,the set of patch locations P, the object bounding box b, feature
extractor FE(·), provable analysis of the robust classification
head RCH-PA(·), cluster detection procedure DetCluster(·)
Output: whether the object b in x has provable robustness
1: procedure DG-PA(x, 𝑤 𝑥 , 𝑤 𝑦 ,𝑇 , P, b)
2:
if b ∉ DG(x, 𝑤 𝑥 , 𝑤 𝑦 ,𝑇 ) then3:
return False⊲ Clean detection is incorrect4:
end if5:
fm ← FE(x)⊲ Extract feature map
6:for each p ∈ P do
⊲ Check every patch location
7:
𝑥, 𝑦, 𝑝𝑥 , 𝑝 𝑦 ← p
8:𝑟 ← DG-PA-One(fm, 𝑥, 𝑦, 𝑤 𝑥 , 𝑤 𝑦 , 𝑝𝑥 , 𝑝 𝑦 , b,𝑇 )9:
if 𝑟 == False then10:
return False⊲ Possibly vulnerable
11:
end if
12:
end for13:
return True
⊲ Provably robust
14: end procedure
15: procedure DG-PA-One(fm, 𝑥, 𝑦, 𝑤 𝑥 , 𝑤 𝑦 , 𝑝𝑥 , 𝑝 𝑦 , b,𝑇 )𝑋, 𝑌, _ ← Shape(fm)⊲ Get the shape of fm¯ ∗ ← ZeroArray[𝑋, 𝑌, 𝑁 + 1]om
⊲ Initialization⊲ Generates worst-case objectness map for analysis
18:
for each valid (𝑖, 𝑗) do⊲ Every window location
19:
u, l ← RCH-PA(fm[𝑖 : 𝑖 + 𝑤 𝑥 , 𝑗 : 𝑗 + 𝑤 𝑦 ], 𝑥 − 𝑖, 𝑦 −𝑗, 𝑝𝑥 , 𝑝 𝑦 )⊲ Add worst-case (lower-bound) logits¯ ∗ [𝑖 : 𝑖 +𝑤 𝑥 , 𝑗 : 𝑗 +𝑤 𝑦 ] ← om
¯ ∗ [𝑖 : 𝑖 +𝑤 𝑥 , 𝑗 : 𝑗 +𝑤 𝑦 ] +l
20:
om
21:
end for¯ ∗ ← MaxObj( om¯ ∗, axis = −1) ⊲ Max objectness score22:om¯ ∗,𝑇 · 𝑤 𝑥 · 𝑤 𝑦 )23:om∗ ← Binarize( om⊲ Binarization
24:
𝑥 min, 𝑦min, 𝑥 max, 𝑦max, 𝑙 ← b
25:
if DetCluster(om∗ [𝑥 min : 𝑥 max, 𝑦min : 𝑦max ]) is Nonethen26:
return False⊲ No high objectness left
27:
else
28:
return True
⊲ High worst-case objectness29:
end if
30: end procedure16:17:

and defense setup. In this section, we will prove the end-to-end
robustness of DetectorGuard, solving Challenge 1.
Provable analysis of DetectorGuard. Thanks to our objectness explaining strategy, a patch hiding attacker has to make bothBase Detector and Objectness Predictor fail to predict a boundingbox, or high objectness, for a successful attack. Therefore, if we
can prove that Objectness Predictor can output high objectness foran object in the worst case, we can certify its provable robustness.
We present the provable analysis of DetectorGuard in Algorithm 2.
The algorithm takes a clean image x, a ground-truth object bounding box b,9 and a set of valid patch locations P as inputs, and will

END-TO-END PROVABLE ROBUSTNESS

Recall that we consider DetectorGuard to be provably robust for agiven object (in a given image) when it can make correct detection
on the clean image and will either detect part of the object or issue
an alert on the adversarial image. The robustness property holds
for any adaptive patch hiding attacker at any location within our
threat model, including ones who have full access to our models

9 We note that the ground-truth information is essential in our provable analysis(Algorithm 2) but is not used in our actual defense (Algorithm 1).

3183

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

determine whether the object in bounding box b in the image x
has provable robustness against any patch at any location in P. We
state the correctness of Algorithm 2 in Theorem 1, and will explain
the algorithm details by proving the theorem.

is thus robust to any attack using this patch location. This implies
the provable robustness, and the sub-procedure returns True. □Theorem 1 shows that if our provable analysis (Algorithm 2)
returns True for certain objects, DetectorGuard (Algorithm 1) will
always detect the certified object or issue an attack alert. Thisrobustness property is agnostic to attack strategies and holds forany adaptive attacker at any location within our threat model. In ourevaluation (next section), we will use Algorithm 2 and Theorem 1
to certify the provable robustness of every object in every image
and report the percentage of certified objects.

Theorem 1. Given an object bounding box b in a clean image x, a
set of patch locations P, window size (𝑤 𝑥 , 𝑤 𝑦 ), and binarizing threshold 𝑇 (used in DG(·) of Algorithm 1), if Algorithm 2 returns True,i.e., DG-PA(x, 𝑤 𝑥 , 𝑤 𝑦 ,𝑇 , b, P) = True, DetectorGuard has provablerobustness for the object b against any patch hiding attack using any
patch location in P.
Proof. DG-PA(·) first calls DG(·) of Algorithm 1 to determine
if DetectorGuard can detect the object bounding box b on the clean
image x. The algorithm will proceed only when the clean detection
is correct (Line 2-4).
Next, we extract feature map fm, iterate over each patch locationin P, and call the sub-procedure DG-PA-One(·), which analyzes
worst-case behavior over all possible adversarial strategies for eachpatch location, to determine the model robustness. If any call of
DG-PA-One(·) returns False, the algorithm returns False, indicating that at least one patch location can bypass our defense. On
the other hand, if the algorithm tries all valid patch locations anddoes not return False, this means that DetectorGuard is provably
robust to all patch locations in P and the algorithm returns True.
In the sub-procedure DG-PA-One(·), we analyze the worst-case
output of Objectness Predictor against the given patch location. We
perform the provable analysis of the robust image classification
head (using RCH-PA(·)) to determine the lower/upper bounds ofclassification logits for each window. If the aggregated worst-case(i.e., lower bound) objectness map still has high activation for theobject of interest, we can certify the robustness of DetectorGuard.
As shown in the DG-PA-One(·) pseudocode, we first initialize a¯ ∗ to hold the worst-case objectness scores. We thenzero array om
iterate over each sliding window and call RCH-PA(·), which takes
the feature map window fm[𝑖 : 𝑖 + 𝑤 𝑥 , 𝑗 : 𝑗 + 𝑤 𝑦 ], relative patch coordinates (𝑥 −𝑖, 𝑦 − 𝑗), patch size (𝑝𝑥 , 𝑝 𝑦 ) as inputs and outputs theupper bound u and lower bound l of the classification logits. Since
the goal of the hiding attack is to minimize the objectness scores,
we only need to reason about the lower bound of classification logits. Recall that in RCH(·), we clip all local logits values into [0, ∞];therefore, the best an adversary can do is to push all corrupted
logits values down to zeros. We then compute the lower bound l byzeroing out all corrupted logits values and aggregating the remaining ones. We note that the sub-procedure DG-PA-One(·) aims tocheck defense robustness for a particular patch location; therefore,
the patch location and corrupted features/logits are known in thisprovable analysis (we discuss how to map pixel-space coordinates
to feature-space coordinates in Appendix G).
Given the lower bound l of every window classification logits,¯ ∗ . After we anawe will add it to the corresponding region of omlyze all valid windows, we call MaxObj(·) and Binarize(·) for the
worst-case objectness map om∗ . We then get the cropped objectness map that corresponds to the object of interest (i.e., om∗ [𝑥 min :𝑥 max, 𝑦min : 𝑦max ]) and feed it to the cluster detection algorithm
DetClutser(·). If None is returned, a hiding attack using this patch
location might succeed, and the sub-procedure returns False. Otherwise, Objectness Predictor has a high worst-case objectness and5

EVALUATIONIn this section, we provide a comprehensive evaluation of DetectorGuard on PASCAL VOC [13], MS COCO [26], and KITTI [15]datasets. We will first introduce the datasets and models used in ourevaluation, followed by our evaluation metrics. We then report our
main evaluation results on different models and datasets, and finallyprovide a detailed analysis of DetectorGuard performance. Our codeis available at https://github.com/inspire-group/DetectorGuard.

5.1

Dataset and Model

Dataset:PASCAL VOC [13]. The detection challenge of PASCAL VisualObject Classes (VOC) project is a popular object detection datasetwith annotations for 20 different classes. We take trainval2007(5k images) and trainval2012 (11k images) as our training set
and evaluate our defense on test2007 (5k images), which is aconventional usage of the PASCAL VOC dataset [29, 64].
MS COCO [26]. The Microsoft Common Objects in COntext(COCO) dataset is an extremely challenging object detection datasetwith 80 annotated common object categories. We use the training
and validation set of COCO2017 for our experiments. The training
set has 117k images, and the validation set has 5k images. We ignore
bounding boxes with the flag iscrowd=1 for simplicity.
KITTI [15]. KITTI is an autonomous vehicle dataset that contains
both 2D camera images and 3D point clouds. We take its 7481 2D
images and use 80% of randomly splited images for training andthe remaining 20% for validation. We merge all classes into threeclasses: car (all different classes of vehicles), pedestrian, cyclist.
Base Detector:
YOLOv4 [2, 53] is the state-of-the-art one-stage object detector.
We choose Scaled-YOLOv4-P5 [53] in our evaluation. For MS COCO,
we use the pre-trained model. For PASCAL VOC and KITTI, we
fine-tune the model previously trained on MS COCO.
Faster R-CNN [45] is a representative two-stage object detector. We use ResNet101-FPN as its backbone network. Image preprocessing and model architecture follows the original paper. We
use pre-trained models for MS COCO and fine-tune pre-trainedmodels for PASCAL VOC and KITTI detectors.
Perfect Clean Detector (PCD) is a hypothetical object detector
simulated with ground-truth annotations. PCD can always makecorrect detection in the clean setting but is assumed vulnerable topatch hiding attacks. This hypothetical object detector ablates theerrors of Base Detector and helps us better understand the behavior
of Objectness Predictor and Objectness Explainer.
Objectness Predictor:

3184

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Table 2: Clean performance of DetectorGuard

Perfect clean detector
YOLOv4
Faster R-CNN

100%
92.9%
90.0%

PASCAL VOC
defense AP FAR@0.8
99.3%
92.4%
89.6%

vanilla AP

MS COCOdefense AP

FAR@0.6

vanilla AP

KITTI
defense AP

FAR@0.8

100%
73.6%
66.7%

99.0%
73.4%
66.5%

1.2%
1.6%
0.9%

100%
93.1%
89.9%

99.0%
92.4%
89.1%

1.5%
1.7%
1.4%

0.9%
4.0%
2.9%

BagNet-33 [3], which has a 33×33 small receptive field, is the
backbone network of Objectness Predictor. For PASCAL VOC and
MS COCO, we zero-pad each image to a square and resize it to
416×416 before feeding it to BagNet; for KITTI, we resize each image
to 224×740. We fine-tune a BagNet model that is pre-trained on
ImageNet [11]. To train an image classifier given a list of boundingboxes in the object detection dataset, we first map pixel-spacebounding boxes to the feature space (details of box mapping are in
Appendix G). We then teach BagNet to make correct predictions on
cropped feature maps by minimizing the cross-entropy loss between
aggregated feature predictions and one-hot encoded label vectors.
In addition, we aggregate all features outside any feature boxes as
the “negative" feature vector for the “background" classification.
Default Hyper-parameter:
We will analyze the effect of different hyper-parameters in Section 5.5. In our default setting, we use a square feature-space windowof size 8 and the DBSCAN clustering [12] with eps = 3, min_points =24 for different datasets.10 We set the default binarizing thresholdto 32 for PASCAL VOC, 36 for MS COCO, and 11 for KITTI basedon different model properties with different datasets.

5.2

Metric100
90
80
Precision / FAR (%)

vanilla AP

70
60
50
40
30
20

Precision-PCD-vanilla
Precision-PCD-defended
Precision-YOLO-vanilla
Precision-YOLO-defended
Precision-FRCNN-vanilla
Precision-FRCNN-defended
FAR-PCD-defended
FAR-YOLO-defended
FAR-FRCNN-defended

10
0
10

20

30

40

50
60
Recall (%)

70

80

90

100

Figure 4: Clean performance of DetectorGuard on PASCALVOC (PCD – perfect clean detector; YOLO – YOLOv4; FRCNN –Faster R-CNN; FAR – False Alert Rate)

False Alert Rate (FAR@0.x). FAR is defined as the percentage ofclean images on which DetectorGuard will trigger a false alert.
The false alert is mainly caused by Clean Error 2 as discussed in
Section 3. We note that FAR is also closely tied to the confidence
threshold of Base Detector: a higher confidence threshold leads
to fewer predicted bounding boxes, leading to more unexplained
objectness, and finally higher FAR. We will report FAR at different
recall levels for a global evaluation, and use FAR@0.x to denote
FAR at a clean recall of 0.x.
Provable Robustness Metric:
Certified Recall (CR@0.x). We use certified recall as the robustness metric against patch hiding attacks. The certified recall is defined as the percentage of ground-truth objects that have provablerobustness against any patch hiding attack. Recall that an object
has provable robustness when Algorithm 2 (our provable analysis)
returns True. Note that CR is also affected by the performance ofBase Detector (e.g., confidence threshold) since its prerequisite is
the correct clean detection. We use CR@0.x to denote the certified
recall at a clean recall of 0.x.

Clean Performance Metric:
Precision and Recall. We calculate the precision as TP/(TP+FP)
and the recall as TP/(TP+FN). For the clean images without a false
alert, we follow previous works [8, 64] setting the IoU threshold𝜏 = 0.5 and count TPs, FPs, FNs in the conventional manner. For
images that have false alerts, we set TP and FP to zeros, and FN to
the number of ground-truth objects since no bounding box is predicted. We note that conventional object detectors use a confidence
threshold to filter out bounding boxes with low confidence values.
As a result, different confidence thresholds will give different precision and recall values; we will plot the entire precision-recall curve
to analyze the model performance.
Average Precision (AP). To remove the dependence on the confidence threshold and to have a global view of model performance,
we also report Average Prevision (AP) as one of evaluation metrics.
We vary the confidence threshold from 0 to 1, record the precisionand recall at different thresholds, and calculate AP as the averaged
precision at different recall levels. This calculated AP can be considered as an approximation of the AUC (area under the curve) for the
precision-recall curve. We note that AP is one of the most widelyused performance metrics in object detection benchmark competitions [13, 15, 26] and research papers [2, 19, 25, 29, 43–45, 49].

5.3

Clean Performance

In this subsection, we evaluate the clean performance of DetectorGuard with three different base object detectors and three datasets.
In Table 2, we report AP of vanilla Base Detector (vanilla AP), AP of
DetectorGuard (defense AP), and False Alert Rate at a clean recall of
0.8 or 0.6 (FAR@0.8 or FAR@0.6). We also plot the precision-recall

10 A 416×416 (or 224×740) pixel image results in a 48×48 (or 24×89) feature map using

our BagNet-33 implementation.

3185

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Table 3: Provable robustness of DetectorGuard
PASCAL VOC (CR@0.8)
far-patch close-patch over-patch
Perfect clean detector
YOLOv4
Faster R-CNN

28.6%
24.6%
25.7%

20.7%
18.6%
19.4%

8.3%
7.5%
8.0%

MS COCO (CR@0.6)
far-patch close-patch over-patch
11.5%
10.1%
10.7%

and FAR-recall curves for PASCAL VOC in Figure 4; similar plots
for MS COCO and KITTI are in Appendix D.
DetectorGuard has a low FAR and a high AP. We can see
from Table 2 that DetectorGuard has a low FAR of 0.9% and a highAP of 99.3% on PASCAL VOC when we use a perfect clean detector
as Base Detector. The result shows that DetectorGuard only has aminimal impact on the clean performance.
DetectorGuard is highly compatible with different conventional object detectors. From Table 2 and Figure 4, we cansee that when we use YOLOv4 or Faster R-CNN as Base Detector
on PASCAL VOC, the clean AP, as well as the precision-recall curveof DetectorGuard, is close to that of its vanilla Base Detector. These
results show that DetectorGuard is highly compatible with different
conventional object detectors.
DetectorGuard works well across different datasets. We
can see that the observation of high clean performance is similar across three different datasets: DetectorGuard achieves a low
FAR and a similar AP as the vanilla Base Detector on PASCAL VOC,MS COCO, and KITTI (the precision-recall plots for MS COCO andKITTI are available in Appendix D). These similar results showthat DetectorGuard is a general approach and can be used for botheasier and more challenging detection tasks.
Remark: a negligible cost of clean performance. In this subsection, we have shown that DetectorGuard only incurs a negligible cost of clean performance (<1% AP drop). This slight clean
performance drop is worthwhile given the first provable robustness
against patch hiding attacks (evaluated in the next subsection).

5.4

7.0%
6.5%
6.8%

far-patch

2.2%
1.9%
2.0%

32.0%
30.6%
31.6%

KITTI (CR@0.8)
close-patch over-patch
11.1%
10.9%
11.0%

2.1%
2.1%
2.1%

30

Certified Recall (%)

25

20
PCD-far
PCD-close
PCD-over
YOLO-far
YOLO-close
YOLO-over
FRCNN-far
FRCNN-close
FRCNN-over

15

10

5

0
10

20

30

40

50
60
70
Clean Recall (%)

80

90

100

Figure 5: Provable robustness of DetectorGuard on PASCALVOC (PCD – Perfect Clean Detector; YOLO – YOLOv4; FRCNN –Faster R-CNN)

above algorithm already considers all possible adaptive attacks (attacker strategies) at any location within our threat model. We use
CR@0.x as the robustness metric. Given the large number of all
possible patch locations, we only use a 500-image subset of thetest/validation datasets for evaluation.
DetectorGuard achieves the first non-trivial provable robustness against patch hiding attack. We report the certified
recall at a clean recall of 0.8 or 0.6 (CR@0.8 or CR@0.6) in Table 3.
As shown in Table 3, when we use a Perfect Clean Detector, DetectorGuard can certify the robustness for 28.6% of PASCAL VOC
objects when the patch is far away from the object; which meansno attacker within our threat model can successfully attack thesecertified objects. We also plot the CR-recall curves for different
detectors on the PASCAL VOC dataset in Figure 5 (similar plots for
MS COCO and KITTI are in Appendix D). The figure shows thatthe provable robustness improves as the clean recall increases, andthe performance of YOLOv4 and Faster R-CNN is close to that of aperfect clean detector when the recall is close to one.
DetectorGuard is especially effective when the patch is far
away from the objects. From Table 3 and Figure 5, we can clearly
see that the provable robustness of DetectorGuard is especially goodwhen the patch gets far away from the object. This model behavior
aligns with our intuition that a localized adversarial patch should
only have a spatially constrained adversarial effect. Moreover, thisobservation shows that DetectorGuard has made the attack much
more difficult: to have a chance to bypass DetectorGuard, the adversary has to put the patch close to or even over the victim object,which is not always feasible in real-world scenarios. We also note

Provable Robustness

In this subsection, we first introduce the robustness evaluation setup
and then report the provable robustness of our defense against any
patch hiding attack within our threat model.
Setup. We use a 32×32 adversarial pixel patch on the re-scaled
and padded 416×416 (or 224×740) images to evaluate the provablerobustness.11 We consider all possible image locations as candidate
locations for the adversarial patch to evaluate the model robustness.
We categorize our results into three categories depending on the
distance between an object and the patch location. When the patch
is totally over the object, we consider it as over-patch. When thefeature-space distance between patch and object is smaller than 8,
we consider it as close-patch. The other patch locations are considered as far-patch. For each set of patch locations and each object,
we use Algorithm 2 to determine the robustness. We note that the

11 DPatch [30] demonstrates that even a 20×20 adversarial patch at the image corner canhave a malicious effect. In Appendix A, we show that more than 15% of PASCAL VOC
objects and 44% of MS COCO objects are smaller than a 32×32 patch. We also provide
robustness results for different patch sizes as well as visualizations in Appendix A.

3186

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

5.5

Detailed Analysis of DetectorGuard

In this subsection, we first perform a runtime analysis of DetectorGuard to show its small defense overhead. Next, we use a hypothetical perfect clean detector (PCD) and the PASCAL VOC dataset
to analyze the performance of DetectorGuard under different hyperparameter settings. Note that using PCD helps us to focus on the
behavior of Objectness Predictor and Objectness Explainer.
Runtime analysis. In Table 4, we report the average per-example
runtime of each module in DetectorGuard. For Base Detector, we report runtime for YOLOv4 (left) and Faster R-CNN (right). As shownin the table, Objectness Predictor has a similar runtime as Base
Detector (or vanilla undefended object detectors), and Objectness
Explainer only introduces a negligible overhead. If 2 GPUs are available, then Base Detector and Objectness Predictor can run in parallelin DetectorGuard, and the overall runtime of DetectorGuard canbe calculated as 𝑡 DetectorGuard = max(𝑡 base, 𝑡 predictor ) + 𝑡 explainer (reported in in the last column of Table 4), which is close to 𝑡 base . Thus,
DetectorGuard has a similar runtime performance as conventional

3187

70
60

30

PCD-close
Average object size25

CR (%)

20
40
15

30

10

20
10

5

0

0

Object size (%)

50

ro
pl
bi ane
cy
cl
e
bi
rd
bo
bo at
tt
le
bu
s
ca
r
ca
ch t
ai
di
ni c r
ng ow
ta
bl
e
do
m ho got r
or seb
po pe ike
tt rs
ed o
pl n
a
sh nt
ee
p
so
f
tv tr a
m ai
on n
it
or

that in the over-patch threat model, we allow the patch to be anywhere over the object. This means that the patch can be placed overthe most salient part of the object (e.g., the face of a person), andmakes robust detection extremely difficult.
DetectorGuard has better robustness for object classes withlarger object sizes. In Figure 6, we plot the certified recalls against
the close-patch attacker (similar plots for far-patch and over-patch
and for other datasets are in Appendix D) and average object sizes(reported as the percentage of image pixels) for all 20 classes of
PASCAL VOC. As shown in the figure, the provable robustness ofDetectorGuard is highly correlated with the object size: we havehigher certified recalls for classes with larger object sizes. This is an
expected behavior because it is hard for even humans to perfectly
detect all small objects. Moreover, considering that missing a big
nearby object is much more serious than missing a small distantobject in real-world applications, we believe that DetectorGuard
has strong application potential.
Remark on absolute values of certified recalls. Despite thefirst achieved provable robustness against patch hiding attacks,
we acknowledge that DetectorGuard’s absolute values of certified
recalls are still limited. First, the notion of the certified recall itself isstrong and conservative: we certify the robustness of an object only
when no patch at any location within the threat model can succeedin the hiding attack (using any attack strategy including adaptive
attacks). In practice, the attacker capability might be limited to asmall number of patch locations. Second, we note that most objects
in our three datasets are small (or even tiny) objects (we provide
a quantitative analysis in Appendix A). Detecting small objects isalready challenging in the clean setting and becomes even moredifficult when an adversarial patch of comparable sizes is used.
However, it is still notable that DetectorGuard is the first to achieve
non-trivial and strong provable robustness against the patch hiding
attacker for “free", and that our approach works particularly well on
certain object classes and threat models (e.g., the class “dog" in the
PASCAL VOC dataset has a ~60% certified recall for the close-patch
threat model). We hope that the security community can build upon
our work and further push the certified recall to a higher value.

ae

Session 11D: Data Poisoning and Backdoor Attacks in ML

Figure 6: Certified recalls and average object sizes for everyclass in PASCAL VOC (reporting CR for close-patch; results forfar-patch and over-patch are in Appendix D)

Table 4: Per-example runtime breakdown

VOC
COCO
KITTI

Base Detector(YOLO / FRCNN)

Objectness
Predictor

Objectness
Explainer

DetectorGuard(YOLO / FRCNN)

54.0ms / 80.9ms
55.2ms / 79.4ms
55.8ms / 69.7ms

48.5ms65.2ms44.6ms

0.2ms
0.3ms
0.4ms

54.2ms / 81.1ms
65.5ms / 79.7ms
56.2ms / 70.1ms

object detectors in the setting of 2 GPUs. If only a single GPU isavailable, the runtime can be calculated as 𝑡 base +𝑡 predictor +𝑡 explainer ,which leads to a roughly 2x slow-down.
Effect of the binarizing threshold. We first vary the binarizing threshold 𝑇 in ObjPredictor(·) to see how the model performance changes. For each threshold, we report CR for three patch
threat models. We also include AP and 1-FAR to understand theeffect of different thresholds on clean performance. We report theseresults in the leftmost sub-figure in Figure 7. We can see that when
the binarizing threshold is low, the CR is high because more objectness is retained after the binarization. However, more objectness
also makes it more likely to trigger a false alert in the clean setting,
and we can see both AP and 1-FAR are affected greatly as we decrease the threshold 𝑇 . Therefore, we need to balance the trade-off
between clean performance and provable robustness. In our default
parameter setting, we set 𝑇 = 32 to have a FAR lower than 1% while
maintaining decent provable robustness.
Effect of window size. We study the effect of different window
sizes in the second sub-figure in Figure 7. The figure demonstrates
a similar trade-off between provable robustness and clean performance. As we increase the window size, each window receives more
information from the input and therefore the clean performance(AP and 1-FAR) improves. However, a large window size increases
the number of windows that are affected by the small adversarial
patch, and the provable robustness drops. In our default setting, we
set the window size to 8 to have a low FAR and good CR.

100

CR-over-patch
CR-close-patch
CR-far-patch
AP
1-FAR

15
10
5
28

30
32
34
Binarizing Threshold T

96
95

98

25
97

20
CR-far-patch
CR-close-patch
CR-over-patch
AP
1-FAR

15
10
5

94

0

4

5

6

7
8
9
Window size

10

11

12

96

25
20

97
CR-over-patch
CR-close-patch
CR-far-patch
AP
1-FAR

15

5

94

0

99

30
98

10
95

100

35

1

2

3
DBSCAN ε4596

CR (%)

97

CR (%)

20

40

99

30

30
CR (%)

25

99

35
AP/1-FAR (%)

CR (%)

98

100

35

AP/1-FAR (%)

99

30

0

40

100
40

35

AP/1-FAR (%)

40

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

98

25
20

97
CR-over-patch
CR-close-patch
CR-far-patch
AP
1-FAR

15
10

95
94

5
0

14

16

18
20
22
24
DBSCAN min_points

26

28

96

AP/1-FAR (%)

Session 11D: Data Poisoning and Backdoor Attacks in ML

95
94

Figure 7: Effect of different hyper-parameters (left to right: binarizing threshold, window size, DBSCAN 𝜖, DBSCAN min_points)
Algorithm 3 Auxiliary Predictor of DetectorGuard

Effect of DBSCAN parameters. We also analyze the effect of
DBSCAN parameters in DetCluster(·). DBSCAN has two parameters 𝜖 and min_points. A point is labeled as a core point when
there are at least min_points points within distance 𝜖 of it; all core
points and their neighbors will be labeled as clusters. We plot theeffect of 𝜖 and min_points in the right two sub-figures in Figure 7.
As we increase 𝜖 or min_points, it becomes more difficult to form
clusters. As a result, the clean performance improves because of
fewer detected clusters and fewer false alerts. However, the provable robustness (CR) drops due to fewer detected clusters in theworst-case objectness map.

6

1: procedure AuxPredictor(D, x)

D̂ ← {}
for 𝑖 ∈ {0, 1, · · · , 𝑛𝑢𝑚_𝑑𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛 − 1} do
4:𝑥 min, 𝑦min, 𝑥 max, 𝑦max, 𝑙 ← b ← D [𝑖]5:𝑙 ′ ← AuxClassier(x[𝑥 min : 𝑥 max, 𝑦min : 𝑦max ])6:
if 𝑙 ′ ≠ “background" thenÐ7:D̂ ← D̂ {(𝑥 min, 𝑦min, 𝑥 max, 𝑦max, 𝑙 ′ )}8:
end if
9:end for10:
return D̂
11: end procedure
2:

3:

DISCUSSION

In this section, we discuss the future work directions and defense
extension of DetectorGuard.

6.1

extend our DetectorGuard to multiple-frame video setting. We
expect that the temporal information could be useful for robustness.
Moreover, we could perform the defense on a subset of frames toreduce defense overhead and minimize false alerts in the cleansetting.

Future Work

Robust object detection without abstention. In this paper, wehave tailored DetectorGuard for attack detection: when no attackis detected, the model uses conventional object detectors for predictions; when an attack is detected, the model alerts and abstains frommaking predictions. This type of defense is useful in application
scenarios like autonomous vehicles which can give the control backto the driver upon detecting an attack. However, the most desirable
notion of robustness is to always make correct predictions without
any abstention. How to extend DetectorGuard for robust object
detection without abstention is an interesting direction of future
work.
Better robust image classifier. In DetectorGuard, we use thekey principals introduced in Section 2.4 to co-design the provably
robust image classifier and Objectness Predictor. However, theimperfection of the adapted robust image classifier still limits theperformance of DetectorGuard detector. Although we can optimize
for few Clean Error 2 and tolerate a potentially high Clean Error 3,
as discussed in Section 3.4, a high Clean Error 3 results in a limited
certified recall of DetectorGuard in the adversarial setting. We notethat DetectorGuard is a general framework that is compatible withany conventional object detector and (principles for building) provably
robust image classifier, and we expect a boost in DetectorGuard’s
performance given any future advance in robust image classification
research.
Extension to the video setting. In this paper, we focus on
object detection in the single-frame setting. It is interesting to

6.2

Defense Extension.

In this paper, we propose DetectorGuard as a provably robust defense against patch hiding, or false-negative (FN), attacks. Here,
we discuss how to extend DetectorGuard for defending against
false-positive (FP) attacks. The FP attack aims to introduce incorrect bounding boxes in the predictions of detectors to increase
FP. We can consider FP attacks as a misclassification problem (i.e.,a bounding box is given an incorrect label), and thus this attack
can be mitigated if we have a robust auxiliary image classifier to
re-classify the detected bounding boxes. If the auxiliary classifier
predicts a different label, we consider it as an FP attack and caneasily correct or filter out the FP boxes.
We provide the pseudocode for using an auxiliary classifier (canbe any robust image classifier) against FP attacks in Algorithm 3.
The algorithm re-classifies each detected bounding box in D aslabel 𝑙 ′ (Line 5). We trust the robust auxiliary classifier and add
bounding boxes with non-background labels to D̂ (Line 7). Finally,
the algorithm returns the filtered detection D̂, and we can replace
the D in Line 8 of Algorithm 1 with D̂ to extend the original
DetectorGuard design. We note that when the patch is not present
in the FP box or it only occupies a small portion of the FP box (theFP box is large), the auxiliary classifier is likely to correctly predict“background" since there is no or few corrupted pixel within the

3188Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

FP box. When the patch occupies a large portion of the FP box,
we might finally output a bounding box that has a large IoU withthe patch. In this case, our object detector correctly locates theadversarial patch but predicts a wrong label. This is an acceptableoutcome because the class label for patches is undefined. Therefore,
Algorithm 3 presents a strong empirical defense for FP attacks.

7.3

Image Classification. Attacks and defenses for classic 𝐿𝑝 -bounded
adversarial examples [6, 16, 48] have been extensively studied.
Many empirical defenses [32, 34, 35, 40, 62] were proposed to mitigate the threat of adversarial examples, but were later found vulnerable to adaptive attackers [1, 5, 51]. The fragility of the empiricaldefenses has inspired certified defenses that are robust to any attacker considered in the threat model [10, 17, 22, 37, 41, 47, 56]. We
refer interested readers to survey papers [39, 63] for more details.
Object Detection. Global perturbations against object detectors
were first studied by Xie et al. [60] and followed by researchers [54,55] in different applications. Defenses against global 𝐿𝑝 perturbations are also very challenging. Zhang et al. [64] used adversarial
training (AT) to improve empirical model robustness while Chiang et al. [8] proposed the use of randomized median smoothing(RMS) for building certifiably robust object detectors. Both defenses
suffer from poor clean performance while DetectorGuard’s clean
performance is close to state-of-the-art object detectors. On PASCAL VOC, AT incurs a ~26% clean AP drop while DetectorGuard
only incurs a <1% drop.12 On MS COCO, both AT and RMS have aclean AP drop that is larger than 10% while ours is smaller than 1%.
We note that we do not compare robustness performance because
these two works focus on global perturbations and are orthogonal
to the objective of this paper.
We note that it is possible to extend our robust objectness predictor design and objectness explaining strategy to mitigate attacks
that use global perturbations with a bounded 𝐿∞ norm (if we havea robust image classifier against 𝐿∞ perturbations). We leave thisas a future work direction.

7 RELATED WORK
7.1 Adversarial Patch Attacks
Image Classification. Unlike most adversarial examples that introduce a global perturbation with a 𝐿𝑝 -norm constraint, localized
adversarial patch attacks allow the attacker to introduce arbitrary
perturbations within a restricted region. Brown et al. [4] introduced
the first adversarial patch attack against image classifiers. They
successfully realized a real-world attack by attaching a patch to thevictim object. Follow-up papers studied variants of localized attacks
against image classifiers with different threat models [21, 27, 28].
Object Detection. Localized patch attacks against object detection have also received much attention in the past few years.
Liu et al. [30] proposed DPatch as the first patch attack against
object detectors in the digital domain. Lu et al. [31], Chen et al. [7],Eykholt et al. [14], and Zhao et al. [66] proposed different physicalattacks against object detectors for traffic sign recognition. Thys etal. [50] proposed to use a rigid physical patch to evade human detection while Xu et al. [61] and Wu et al. [57] generated successfully
non-rigid perturbations on T-shirt to evade detection.

7.2

Other Adversarial Example Attacks andDefenses

8

Defenses against Adversarial Patches

CONCLUSION

In this paper, we propose DetectorGuard, the first general framework for building provably robust object detectors against patch
hiding attacks. DetectorGuard introduces a general approach toadapt robust image classifiers for robust object detection using an
objectness explaining strategy. Our evaluation on the PASCAL VOC,MS COCO, and KITTI datasets demonstrates that DetectorGuard
achieves the first provable robustness against any patch hiding
attacker within the threat model and also has a high clean performance that is close to state-of-the-art detectors.

Image Classification. Digital Watermark (DW) [18] and Local
Gradient Smoothing (LGS) [38] were the first two heuristic defenses against adversarial patch attacks. Unfortunately, these defenses are vulnerable to an adaptive attacker with the knowledge ofthe defense. A few certified defenses [9, 24, 33, 36, 58, 59, 65] havebeen proposed to provide strong provable robustness guarantee
against any adaptive attacker. Notably, PatchGuard [58] introduces
two key principles of small receptive fields and secure aggregation
and achieves state-of-the-art defense performance for image classification. In contrast, DetectorGuard aims to adapt robust image
classifiers for the more challenging robust object detection task.
Object Detection. How to secure object detection is a muchless studied area due to the complexity of this task. Saha et al. [46]demonstrated that YOLOv2 [43] were vulnerable to adversarial
patches because detectors were using spatial context for their predictions, and then proposed a new training loss to limit the usageof context information. To the best of our knowledge, this is the
only prior attempt to secure object detectors from patch attacks.
However, this defense is based on heuristics and thus does nothave any provable robustness. Moreover, the attack and defense aretargeted at YOLOv2 only, and it is unclear if the defense generalizes
to other detectors. In contrast, our defense has provable robustness
against any patch hiding attack considered in our threat model andis compatible with any state-of-the-art object detectors.

ACKNOWLEDGMENTS
We are grateful to Gagandeep Singh for shepherding the paper
and anonymous reviewers at CCS 2021 for their valuable feedback.
We would also like to thank Vikash Sehwag, Shawn Shan, Sihui
Dai, Alexander Valtchanov, Ruiheng Chang, Jiachen Sun, and researchers at Intel Labs for helpful discussions on the project andinsightful comments on the paper draft. This work was supported
in part by the National Science Foundation under grants CNS1553437 and CNS-1704105, the ARL’s Army Artificial Intelligence
Innovation Institute (A2I2), the Office of Naval Research Young
Investigator Award, Schmidt DataX award, and Princeton E-ffiliates
Award.
12 RMS [8] did not report results for PASCAL VOC.

3189

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

REFERENCES[25] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2017.
Focal Loss for Dense Object Detection. In IEEE International Conference on Computer Vision, (ICCV) 2017. IEEE Computer Society, 2999–3007.
[26] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common
Objects in Context. In European Conference on Computer Vision (ECCV), Vol. 8693.
Springer, 740–755.
[27] Aishan Liu, Xianglong Liu, Jiaxin Fan, Yuqing Ma, Anlan Zhang, Huiyuan Xie,and Dacheng Tao. 2019. Perceptual-Sensitive GAN for Generating Adversarial
Patches. In The 33rd AAAI Conference on Artificial Intelligence, (AAAI) 2019. AAAI
Press, 1028–1035.
[28] Aishan Liu, Jiakai Wang, Xianglong Liu, Bowen Cao, Chongzhi Zhang, and Hang
Yu. 2020. Bias-Based Universal Adversarial Patch Attack for Automatic CheckOut. In European conference on computer vision (ECCV), Vol. 12358. Springer,
395–410.
[29] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed,
Cheng-Yang Fu, and Alexander C. Berg. 2016. SSD: Single Shot MultiBox Detector.
In European conference on computer vision (ECCV), Vol. 9905. Springer, 21–37.
[30] Xin Liu, Huanrui Yang, Ziwei Liu, Linghao Song, Yiran Chen, and Hai Li. 2019.
DPATCH: An Adversarial Patch Attack on Object Detectors. In AAAI Conferenceon Artificial Intelligence Workshop (AAAI workshop) 2019, Vol. 2301.
[31] Jiajun Lu, Hussein Sibai, and Evan Fabry. 2017. Adversarial examples that fooldetectors. arXiv preprint arXiv:1712.02494 (2017).
[32] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, andAdrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial
Attacks. In 6th International Conference on Learning Representations (ICLR).
[33] Michael McCoyd, Won Park, Steven Chen, Neil Shah, Ryan Roggenkemper, Minjune Hwang, Jason Xinyu Liu, and David Wagner. 2020. Minority Reports Defense:
Defending Against Adversarial Patches. arXiv preprint arXiv:2004.13799 (2020).
[34] Dongyu Meng and Hao Chen. 2017. MagNet: A Two-Pronged Defense against
Adversarial Examples. In Proceedings of the 2017 ACM SIGSAC Conference on
Computer and Communications Security (CCS). 135–147.
[35] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. 2017.
On Detecting Adversarial Perturbations. In 5th International Conference on Learning Representations (ICLR).
[36] Jan Hendrik Metzen and Maksym Yatsura. 2021. Efficient Certified Defenses
Against Patch Attacks on Image Classifiers. In 9th International Conference on
Learning Representations (ICLR). https://openreview.net/forum?id=hr-3PMvDpil[37] Matthew Mirman, Timon Gehr, and Martin T. Vechev. 2018. Differentiable
Abstract Interpretation for Provably Robust Neural Networks. In Proceedings ofthe 35th International Conference on Machine Learning (ICML). 3575–3583.
[38] Muzammal Naseer, Salman Khan, and Fatih Porikli. 2019. Local Gradients Smoothing: Defense Against Localized Adversarial Attacks. In IEEE Winter Conference
on Applications of Computer Vision (WACV). 1300–1307.
[39] Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P Wellman.
2018. SoK: Security and privacy in machine learning. In 2018 IEEE European
Symposium on Security and Privacy (EuroS&P). 399–414.
[40] Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram
Swami. 2016. Distillation as a Defense to Adversarial Perturbations Against Deep
Neural Networks. In IEEE Symposium on Security and Privacy (S&P). 582–597.
[41] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Certified Defenses
against Adversarial Examples. In 6th International Conference on Learning Representations (ICLR).
[42] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You
only look once: Unified, real-time object detection. In Proceedings of the IEEE
conference on computer vision and pattern recognition. 779–788.
[43] Joseph Redmon and Ali Farhadi. 2017. YOLO9000: better, faster, stronger. In
Proceedings of the IEEE conference on computer vision and pattern recognition.
7263–7271.
[44] Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement.
arXiv preprint arXiv:1804.02767 (2018).
[45] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn:
Towards real-time object detection with region proposal networks. In Advancesin neural information processing systems. 91–99.
[46] Aniruddha Saha, Akshayvarun Subramanya, Koninika Patil, and Hamed Pirsiavash. 2020. Role of Spatial Context in Adversarial Robustness for Object
Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition Workshops (CVPR Workshops). 784–785.
[47] Hadi Salman, Jerry Li, Ilya P. Razenshteyn, Pengchuan Zhang, Huan Zhang,
Sébastien Bubeck, and Greg Yang. 2019. Provably Robust Deep Learning via
Adversarially Trained Smoothed Classifiers. In Annual Conference on Neural
Information Processing Systems 2019 (NeurIPS). 11289–11300.
[48] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan,
Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks.
In 2nd International Conference on Learning Representations (ICLR).
[49] Mingxing Tan, Ruoming Pang, and Quoc V Le. 2020. EfficientDet: Scalable andEfficient Object Detection. In 2020 IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR 2020). 10781–10790.

[1] Anish Athalye, Nicholas Carlini, and David A. Wagner. 2018. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial
Examples. In Proceedings of the 35th International Conference on Machine Learning(ICML). 274–283.
[2] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. YOLOv4:
Optimal Speed and Accuracy of Object Detection. arXiv preprint arXiv:2004.10934(2020).
[3] Wieland Brendel and Matthias Bethge. 2019. Approximating CNNs with bag-oflocal-features models works surprisingly well on ImageNet. In 7th International
Conference on Learning Representations (ICLR).
[4] Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer.
2017. Adversarial patch. In Advances in neural information processing systems
workshops (NeurIPS Workshops).
[5] Nicholas Carlini and David A. Wagner. 2017. Adversarial Examples Are Not
Easily Detected: Bypassing Ten Detection Methods. In Proceedings of the 10th
ACM Workshop on Artificial Intelligence and Security (AISec@CCS). 3–14.
[6] Nicholas Carlini and David A. Wagner. 2017. Towards Evaluating the Robustness
of Neural Networks. In 2017 IEEE Symposium on Security and Privacy (S&P).
39–57.
[7] Shang-Tse Chen, Cory Cornelius, Jason Martin, and Duen Horng Polo Chau. 2018.
Shapeshifter: Robust physical adversarial attack on faster r-cnn object detector.
In Joint European Conference on Machine Learning and Knowledge Discovery in
Databases. Springer, 52–68.
[8] Ping-yeh Chiang, Michael Curry, Ahmed Abdelkader, Aounon Kumar, John
Dickerson, and Tom Goldstein. 2020. Detection as Regression: Certified Object
Detection with Median Smoothing. In Advances in Neural Information Processing
Systems (NeurIPS) 2020, Vol. 33.
[9] Ping-Yeh Chiang, Renkun Ni, Ahmed Abdelkader, Chen Zhu, Christoph Studor,
and Tom Goldstein. 2020. Certified defenses for adversarial patches. In 8thInternational Conference on Learning Representations (ICLR).
[10] Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. 2019. Certified Adversarial
Robustness via Randomized Smoothing. In Proceedings of the 36th International
Conference on Machine Learning (ICML). 1310–1320.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Computer Society
Conference on Computer Vision and Pattern Recognition (CVPR). 248–255.
[12] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. 1996. A densitybased algorithm for discovering clusters in large spatial databases with noise.. In
Kdd, Vol. 96. 226–231.
[13] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, andAndrew Zisserman. 2010. The Pascal Visual Object Classes (VOC) Challenge.
International Journal of Computer Vision 88, 2 (2010), 303–338.
[14] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian
Tramer, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2018. Physical adversarial examples for object detectors. In 12th USENIX Workshop on Offensive
Technologies (WOOT 18).
[15] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. 2013. Vision
meets robotics: The kitti dataset. The International Journal of Robotics Research
32, 11 (2013), 1231–1237.
[16] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining andHarnessing Adversarial Examples. In 3rd International Conference on Learning
Representations (ICLR).
[17] Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli
Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Arthur Mann, and Pushmeet
Kohli. 2019. Scalable Verified Training for Provably Robust Image Classification.
In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 4841–4850.
[18] Jamie Hayes. 2018. On Visible Adversarial Perturbations & Digital Watermarking.
In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops(CVPR Workshops). 1597–1604.
[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. 2017. Mask
R-CNN. In IEEE International Conference on Computer Vision, (ICCV 2017). IEEE
Computer Society, 2980–2988.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual
Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and
Pattern Recognition (CVPR). 770–778.
[21] Danny Karmon, Daniel Zoran, and Yoav Goldberg. 2018. LaVAN: Localized and
Visible Adversarial Noise. In Proceedings of the 35th International Conference on
Machine Learning (ICML). 2512–2520.
[22] Mathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman
Jana. 2019. Certified Robustness to Adversarial Examples with Differential Privacy.
In 2019 IEEE Symposium on Security and Privacy (S&P). 656–672.
[23] Mark Lee and Zico Kolter. 2019. On physical adversarial patches for object
detection. arXiv preprint arXiv:1906.11897 (2019).
[24] Alexander Levine and Soheil Feizi. 2020. (De)randomized Smoothing for Certifiable Defense against Patch Attacks. arXiv preprint arXiv:2002.10733 (2020).

3190

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

A[50] Simen Thys, Wiebe Van Ranst, and Toon Goedemé. 2019. Fooling automatedsurveillance cameras: adversarial patches to attack person detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.
[51] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry.
2020. On adaptive attacks to adversarial example defenses. arXiv preprint
arXiv:2002.08347 (2020).
[52] Abdul Vahab, Maruti S Naik, Prasanna G Raikar, and Prasad SR. 2019. Applicationsof Object Detection System. International Research Journal of Engineering andTechnology (IRJET) 6, 4 (2019), 4186–4192.
[53] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. 2020. ScaledYOLOv4: Scaling Cross Stage Partial Network. arXiv preprint arXiv:2011.08036(2020).
[54] Derui Wang, Chaoran Li, Sheng Wen, Xiaojun Chang, Surya Nepal, and Yang
Xiang. 2019. Daedalus: Breaking non-maximum suppression in object detection
via adversarial examples. arXiv (2019), arXiv–1902.
[55] Xingxing Wei, Siyuan Liang, Ning Chen, and Xiaochun Cao. 2019. Transferable
Adversarial Attacks for Image and Video Object Detection. In Proceedings of theTwenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI) 2019,
Sarit Kraus (Ed.). ijcai.org, 954–960.
[56] Eric Wong and J. Zico Kolter. 2018. Provable Defenses against Adversarial
Examples via the Convex Outer Adversarial Polytope. In Proceedings of the 35th
International Conference on Machine Learning (ICML). 5283–5292.
[57] Zuxuan Wu, Ser-Nam Lim, Larry S. Davis, and Tom Goldstein. 2020. Making
an Invisibility Cloak: Real World Adversarial Attacks on Object Detectors. In
European Conference on Computer Vision (ECCV) 2020, Vol. 12349. 1–17.
[58] Chong Xiang, Arjun Nitin Bhagoji, Vikash Sehwag, and Prateek Mittal. 2021.
PatchGuard: A Provably Robust Defense against Adversarial Patches via Small
Receptive Fields and Masking. In 30th USENIX Security Symposium (USENIX
Security).
[59] Chong Xiang and Prateek Mittal. 2021. PatchGuard++: Efficient Provable Attack
Detection against Adversarial Patches. In ICLR 2021 Workshop on Security and
Safety in Machine Learning Systems.
[60] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan L.
Yuille. 2017. Adversarial Examples for Semantic Segmentation and Object Detection. In IEEE International Conference on Computer Vision (ICCV) 2017. IEEE
Computer Society, 1378–1387.
[61] Kaidi Xu, Gaoyuan Zhang, Sijia Liu, Quanfu Fan, Mengshu Sun, Hongge Chen,
Pin-Yu Chen, Yanzhi Wang, and Xue Lin. 2020. Adversarial T-Shirt! Evading
Person Detectors in a Physical World. In European Conference on Computer Vision(ECCV) 2020, Vol. 12350. 665–681.
[62] Weilin Xu, David Evans, and Yanjun Qi. 2018. Feature Squeezing: Detecting
Adversarial Examples in Deep Neural Networks. In 25th Annual Network andDistributed System Security Symposium (NDSS).
[63] Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. 2019. Adversarial examples:
Attacks and defenses for deep learning. IEEE transactions on neural networks andlearning systems 30, 9 (2019), 2805–2824.
[64] Haichao Zhang and Jianyu Wang. 2019. Towards Adversarially Robust Object
Detection. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV)2019. IEEE, 421–430.
[65] Zhanyuan Zhang, Benson Yuan, Michael McCoyd, and David Wagner. 2020.
Clipped BagNet: Defending Against Sticker Attacks with Clipped Bag-of-features.
In 3rd Deep Learning and Security Workshop (DLS).
[66] Yue Zhao, Hong Zhu, Ruigang Liang, Qintao Shen, Shengzhi Zhang, and Kai
Chen. 2019. Seeing isn’t Believing: Towards More Robust Adversarial Attack
Against Real World Object Detectors. In Proceedings of the 2019 ACM SIGSAC
Conference on Computer and Communications Security. 1989–2004.

OBJECT SIZE AND PATCH SIZE

Recall that in Section 5.4, we use a 32×32 patch on 416×416 (or224×740) images to evaluate the provable robustness. In this section,
we provide additional details of object sizes and patch sizes in
PASCAL VOC, MS COCO, and KITTI datasets.
Small objects are the majority of all three datasets. In Figure 8, we plot the histogram of object sizes (in the percentage of
pixels) in the test/validation sets of PASCAL VOC, MS COCO, and
KITTI. As shown in the plots, small, or even tiny, objects are themajority of three datasets. A 32×32 patch takes up 0.6% pixels of a
416×416 (or 224×740) image, and our further analysis shows that
15.2% objects of PASCAL VOC are smaller than 0.6% image pixels;
44.5% of MS COCO and 44.6% KITTI objects are smaller than 0.6%.
Moreover, more than 36.5% of PASCAL VOC objects, more than
66.3% of MS COCO objects, and 75.9% KITTI objects are smaller
than a 64×64 square. These numbers explain why the absolute numbers of certified recall in Table 3 are low. In Figure 9, we further
provide visualization of a 32×32 patch on the 416×416 image todemonstrate the challenge of perfect robust detection even when asmall patch is presented. In the left two examples, the person andthe cow are completely blocked by the adversarial patch and thus
are unrecognizable. In the rightmost example, the head of the dog
is patched and it is even hard for humans to determine if it is a dog
or cat.
Additional evaluation results for different patch sizes. In Figure 10, we vary the patch size to see how the provable robustness isaffected given different attacker capabilities (i.e, patch sizes). If weconsider a smaller patch of 8×8 pixels, we can have a 2.0% higher CR
for close-patch, and a 2.8% higher CR for over-patch compared with
our CRs for a 32×32 patch. Furthermore, we note that in the farpatch model, the patch size has a limited influence on the provablerobustness. From Figure 10, We can also see that the CR decreases
as the patch size increases. This analysis demonstrates the limit ofDetectorGuard as well the challenge of robust object detection withlarger patch sizes. We aim to push this limit further in our future
work.

B

ADDITIONAL DISCUSSION OF ROBUSTCLASSIFIER IMPLEMENTATIONAs introduced in Section 2.4, state-of-the-art provable robust image
classifiers [36, 58] 1) use DNN with small receptive fields to boundthe number of corrupted features and then 2) do secure aggregationon the partially corrupted feature map for robust classification. In
DetectorGuard, we choose BagNet [3] for small receptive fields andfeature clipping for secure aggregation. In this section, we provide
additional details of BagNet and clipping aggregation. Furthermore,
we discuss alternative aggregation mechanisms and implement
robust masking [58] to demonstrate the generality of our DetectorGuard framework.
BagNet [3]. BagNet was originally proposed for interpretable machine learning. It inherits the high-level architecture of ResNet50 [20] and replaces 3x3 convolution kernels with 1x1 ones toreduce the receptive field size. The authors designed three BagNet architectures with a small receptive field of 9×9, 17×17, and33×33, in contrast to ResNet-50 having a receptive field of 483×483.

3191

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

0.14
0.16

0.40
0.12

0.14

0.35

0.12

0.08

0.06

% Objects

0.30% Objects% Objects

0.10

0.25
0.20

0.10
0.08

0.15

0.06

0.10

0.04

0.05

0.02

0.00

0.00

8

6

4

2

0.4

0

0.4

8

0.4

0.4

6

4

0.4

2

0.3

0.3

0

0.3

8

6

0.3

4

0.3

0.2

2

0

0.2

8

0.2

6

0.2

0.2

4

2

0.1

0

0.1

0.1

8

0.1

6

4

Object size (%)

0.1

2

0.0

0.0

0

0.0

0.0

0.0

8

6

4

0.4

2

0

0.4

8

0.4

0.4

6

0.4

4

2

0.3

0

0.3

0.3

8

6

0.3

4

0.3

2

0.2

0.2

0

8

0.2

0.2

6

Object size (%)

0.2

4

0.1

2

0

0.1

8

0.1

0.1

6

4

0.1

2

0.0

0.0

0.0

0.0

8

6

4

0.4

2

0.4

0

8

0.4

6

0.4

0.4

4

2

0.3

0

0.3

8

0.3

0.3

6

4

0.3

2

0.2

0.2

0

0.2

8

6

0.2

4

0.2

0.1

2

0

0.1

8

0.1

0.1

6

0.1

4

2

0.0

0.0

0.0

0.0

0.0

0

0.00

0.0

0.02

0

0.04

Object size (%)

Figure 8: Histogram of object sizes (left: PASCAL VOC; middle: MS COCO; right: KITTI)
Table 5: Comparison between masking-based and clipping-based defenses of DetectorGuard (using a perfect clean detector)

PASCAL VOC
MS COCO
KITTI

AP

clipping-based DetectorGuard
FAR CR-far CR-close CR-overAPmasking-based DetectorGuard
FAR CR-far CR-close CR-over

99.3%
99.0%
99.0%

0.9%
1.2%
1.5%

98.9%
98.7%
99.4%

1.1%
1.4%
1.1%

28.6%
11.5%
31.6%

20.7%
7.0%
11.0%

8.3%
2.2%
2.1%

26.2%
11.4%
17.4%

17.2%
5.4%
4.9%

22.5

4.7%
1.6%
1.2%

CR-over-patch
CR-close-patch

20.0

CR (%)

17.5
15.0
12.5
10.0
7.5
5.0
10

20

30
40
50
Patch size (px)60

70

Figure 10: Effect of patch size on provable robustness of DetectorGuard with a perfect clean detector

Figure 9: Visualization of patches on small objects (upper:
original 416×416 images; lower: images with a 32×32 black
patch)

object classes significantly. It is easy to calculate the lower boundof classification logits: we only need to zero out all features within
the patch location(s) and aggregate the remaining features.
Alternative aggregation. We propose DetectorGuard as a general framework that is compatible with any provably robust image
classification technique. To further support this claim, we implement Objectness Predictor using a PatchGuard classifier with robust
masking secure aggregation [58], which achieves the best clean
classification accuracy and provable robust classification accuracy
on high-resolution ImageNet [11] dataset. We compare the performance of defenses with clipping-base and robust-making-based
secure aggregation in Table 5. As we can see from the table, two
defenses achieve high clean performance and non-trivial provablerobustness, demonstrating that DetectorGuard is compatible withdifferent provably robust image classifiers. We note that we do not

Brendel and Bethge showed that BagNet-17 with a 17×17 receptive field can achieve a similar top-5 accuracy as AlexNet [3]. In
recent works [58, 65] on adversarial patch defense, BagNet has
been adopted to bound the number of corrupted features to achieverobustness.
Clipping. In addition to the use of CNNs with small receptive fields,
we also need a secure aggregation mechanism to ensure that a small
number of corrupted features only have a limited influence on the
final prediction/classification. Recall that in our provable analysis(Algorithm 2; Section 4), we need the lower bound of classification
logits to reason about the worst-case objectness map output. In
order to impose such a lower bound, we clip all feature values into
[0, ∞] such that an adversarial patch cannot decrease the values of3192

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Table 6: Provable robustness (CR) of DetectorGuard (using aperfect clean detector) against multiple patches (evaluatedon 50 PASCAL VOC images with a subset of patch locations)choose robust masking in our main evaluation because 1) it has alooser lower bound compared with clipping while 2) it introduces aslightly higher computation overhead. Furthermore, robust maskingof PatchGuard only has limited robustness against the multiplepatch attacker. In contrast, as demonstrated in Appendix C, theclipping-based DetectorGuard can handle multiple patches.

one 32×32 patch (1024 px)
two 32×32 patches (2048 px)
two 24×24 patches (1152 px)
two 16×16 patches (512 px)

C

far-patchclose-patch

over-patch

27.3%
27.3%
27.3%
27.3%

22.4%
18.0%
18.6%
19.3%

8.7%
3.1%
3.1%
5.0%

DISCUSSION ON MULTIPLE PATCHES

In our main body, we focus on the one-patch threat model because building a high-performance provably robust object detector
against a single-patch attacker is an unresolved and open researchquestion. In this section, we discuss DetectorGuard’s robustness
against multiple patches.
Quantitative analysis of clipping-based DetectorGuard against
multiple patches. One advantage of the clipping-based robust
classifier is its robustness against multiple patches. As long as the
sub-procedure RCH-PA(·) of the clipping-based robust classifier
can return non-trivial bounds of classification logits, we can directly plug the sub-procedure into our Algorithm 2 to analyze therobustness against multiple patches.
We note that despite the theoretical possibility to defend against
attacks with multiple patches, its quantitative evaluation for provable robustness is extremely expensive due to the large number ofall possible combinations of multiple patch locations. Consider a
32×32 patch on a 416×416 image. There are 148k possible patch
locations (or 1.6k feature-space locations). If we are using 2 patchesof the same size, the number of all location combinations becomes
higher than 1010 (or 1.4M feature-space location combinations)!
In order to provide a proof-of-concept for defense against multiple patches, we perform an evaluation on 50 PASCAL VOC images
using a subset of patch locations (1/16 of all location combinations).
The results are reported in Table 6. As shown in the table, DetectorGuard is able to defend against multiple patches. Moreover, if wecompare provable robustness against one 32×32 (1024 px) and two
24×24 patches (1152 px), which have a similar number of pixels, we
can find that using two smaller patches (two 24×24 patches) is only
more effective for over-patch threat model but not for far-patch andclose-patch threat models. This observation leads to the following
remark.
Remark: multiple patches need to be close to each other andthe victim object for a stronger malicious effect. Unlike image
classification where the classifier makes predictions based on all
image pixels (or extracted features), an object detector predicts
each object largely based on the pixels (or features) around theobject. As a result, patches that are far away from the object only
have a limited malicious effect, and this claim is supported byour evaluation results in Section 5.4 (i.e., DetectorGuard is more
effective against the far-patch threat model). Therefore, multiple
patches should be close to the victim object and hence close to eachother for a more effective attack. In this case, the multiple-patch
threat model becomes similar to the one-patch model since patches
are close to each other and can merge into one single patch. That is,
we can use one single patch of a larger size to cover all perturbations
in multiple small patches.

3193

D

EXPERIMENT RESULTS FOR DIFFERENTTHREAT MODELS AND DATASETSIn this section, we include additional plots for per-class analysis as
well as DetectorGuard’s clean/provable performance on MS COCO
and KITTI. The observation is similar to that in Section 5.
Per-class Analysis. In Figure 11, we provide additional per-classanalysis results. The observation is similar to Figure 6 in Section 5.
Additional plots for MS COCO and KITTI. We plot the clean
performance and the provable robustness for MS COCO in Figure 12 and Figure 13, and for KITTI in Figure 14 and Figure 15.
The observation is similar to that on PASCAL VOC (Figure 4 andFigure 5).

E

JUSTIFICATION FOR DEFENSE OBJECTIVE

In Section 2.3, we allow DetectorGuard to only detect part of theobject or to trigger an attack alert on adversarial images. In thissection, we discuss why this is a reasonable defense objective andhow to extend DetectorGuard for a stronger notion of robustness.
Partially detected bounding box. We note that we allow thepatch to be anywhere, even over the salient object. As a result, thepatch likely covers a large portion of the object (visualization examples include the right part of Figure 1 and Figure 9; see Appendix A
for more details of object sizes and patch sizes). Therefore, it isreasonable to allow the model to output a smaller bounding box. If
we consider the application scenario of autonomous vehicles (AV),partially detecting a pedestrian or a car is already sufficient for an
AV to make a correct decision.
Moreover, we can tune hyper-parameters such as binarizing
threshold 𝑇 to increase the objectness in the output of Objectness
Predictor. More objectness will force the adversary to let Base Detector predict a larger bounding box in order to reduce unexplained
objectness that will otherwise lead to an attack alert. However,
we note that more objectness also makes it more likely for DetectorGuard to trigger a false alert on clean images. This trade-off
between robustness and clean performance should be carefullybalanced.

F

ADDITIONAL DISCUSSION ON “FREE"PROVABLE ROBUSTNESSAs shown in our evaluation, DetectorGuard achieves the first provable robustness for object detectors again patch hiding attacks at anegligible cost of clean performance. Intriguingly, we have demonstrated that we can use a module with limited clean performance(i.e., provably robust image classifier in Objectness Predictor) tobuild a provably robust system with high clean performance (i.e.,Session 11D: Data Poisoning and Backdoor Attacks in ML

60

70

30

PCD-farAverage object size

60

25

25

10

20

15

30

10

20

20
CR (%)

40

Object size (%)

15

30

50
20

CR (%)

20

Object size (%)

50

40

30

PCD-over
Average object size

60

25
50

40
15

30

10

20

5

10

5

0

0

0

0

0

0

pl
ae

ro

pl
ae

ro
ae

bi ane
cy
cl
e
bi
rd
bo
bo at
tt
le
bu
s
ca
r
ca
ch t
ai
di
r
ni c
ng ow
ta
bl
e
do
m ho got r
or seb
po pe ike
tt rs
ed o
pl n
a
sh nt
ee
p
so
f
tv tr a
m ai
on n
it
or

10

bi ane
cy
cl
e
bi
rd
bo
bo at
tt
le
bu
s
ca
r
ca
ch t
ai
di
ni c r
ng ow
ta
bl
e
do
m ho g
ot r
or se
bi
po pe ke
tt rs
ed o
pl n
a
sh nt
ee
p
so
f
tv tr a
m ai
on n
it
or

5

pl
bi ane
cy
cl
e
bi
rd
bo
bo at
tt
le
bu
s
ca
r
ca
ch t
ai
di
r
ni c
ng ow
ta
bl
e
do
m ho got r
or seb
po pe ike
tt rs
ed o
pl n
a
sh nt
ee
p
so
f
tv tr a
m ai
on n
it
or

10

ro

CR (%)

7030

PCD-close
Average object size

Object size (%)

70

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Figure 11: Per-class analysis of PASCAL VOC (left: far-patch; middle: close-patch; right: over-patch)

100

100

90

90
80

Precision-PCD-vanilla
Precision-PCD-defended
Precision-YOLO-vanilla
Precision-YOLO-defended
Precision-FRCNN-vanilla
Precision-FRCNN-defended
FAR-PCD-defended
FAR-YOLO-defended
FAR-FRCNN-defended

70
60
50
40
30
20

Precision / FAR (%)

Precision / FAR (%)

80

Precision-PCD-vanilla
Precision-PCD-defended
Precision-YOLO-vanilla
Precision-YOLO-defended
Precision-FRCNN-vanilla
Precision-FRCNN-defended
FAR-PCD-defended
FAR-YOLO-defended
FAR-FRCNN-defended

70
60
50
40
30
20

10

10

0

0
10

20

30

40

50
60
Recall (%)

70

80

90

100

10

Figure 12: Clean performance of DetectorGuard on MS
COCO

20

30

40

50
60
Recall (%)

70

80

90

100

Figure 14: Clean performance of DetectorGuard on KITTI

30
15.0
Certified Recall (%)

25

Certified Recall (%)

12.5
10.0
PCD-far
PCD-close
PCD-over
YOLO-far
YOLO-close
YOLO-over
FRCNN-far
FRCNN-close
FRCNN-over

7.5
5.0
2.5
0.0
10

20

30

40

20

PCD-far
PCD-close
PCD-over
YOLO-far
YOLO-close
YOLO-over
FRCNN-far
FRCNN-close
FRCNN-over

15
10
5

10
50
60
70
Clean Recall (%)

80

90

20

30

40

50
60
70
Clean Recall (%)

80

90

100

100

Figure 15: Provable robustness of DetectorGuard on KITTI
Figure 13: Provable robustness of DetectorGuard on MS
COCO

two types of errors, false-negative (FN; missing object) and falsepositive (FP; predicting incorrect objects). Intriguingly, despite thedifficulty to have a low FN and a low FP at the same time, it iseasy to have a low FN (but with a potentially high FP) or a low FP(but with a potentially high FN). For example, if an object detector
predicts all possible bounding boxes, the FN is zero (all possible
boxes are retrieved, including the ground-truth box) but the FP is
extremely high (most bounding boxes are incorrect). This intrinsic

DetectorGuard). In this section, we provide additional discussion
on this intriguing behavior.
One major difference between image classification and object
detection is their type of error. For an image classifier, the only
error is misclassification. In contrast, an object detector can have3194

Session 11D: Data Poisoning and Backdoor Attacks in ML

Clean images

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

The output of robust
Objectness Predictor

Adversarial images

Worst-case objectness
map prediction

Figure 16: Visualization of DetectorGuard. From left to right: 1) clean images – conventional detectors correct detect all objects; 2)
adversarial images with patches (marked with red dash boxes) – conventional detectors miss all objects; 3) objectness map generated on theseadversarial images – Objectness Predictor robustly predicts high objectness and eventually leads to an attack alert; 4) worst-case objectness
map – DetectorGuard can certify the provable robustness of these objects. Note that visualizations in this figure only consider one random
patch location for each image, while results reported in Section 5 consider all possible locations and attack strategies within the threat model.

property of the object detection task allows us to achieve “free"provable robustness.
Recall that in the clean setting, our design of prediction matching strategy (Section 3.4) enables DetectorGuard to tolerate FN of
Objectness Predictor (i.e., Clean Error 3: predicts objects as background). Therefore, we can safely and easily optimize for a low FP(i.e., Clean Error 2: predicts background as objects) to achieve a highclean performance.

Gby the pixels within the pixel-space bounding box. We note thatthe mapping equation might be slightly different given different
implementation of CNNs with small receptive fields. In our BagNet
implementation, we have r = 33, s = 8.

H

PIXEL-SPACE AND FEATURE-SPACEWINDOWS

Recall that in Section 3.3, we used a BagNet to extract a feature
map for the entire image and perform robust window classificationin the feature space. This design allows us to reuse the extracted
feature map and reduce computational overhead. In this section,
we discuss how to map the pixel-space bounding box to the feature
space.
Box mapping. For each pixel-space box (𝑥 min, 𝑦min, 𝑥 max, 𝑦max ),′we calculate the feature-space coordinate 𝑥 min= ⌊(𝑥 min − r +′′′1)/s⌋, 𝑦min = ⌊(𝑦min − r + 1)/s⌋, 𝑥 max = ⌊𝑥 max /s⌋, 𝑦max
= ⌊𝑦max /s⌋,
where r, s are the size and stride of the receptive field size. The
new feature-space coordinates indicate all features that are affected

3195

VISUALIZATION OF DETECTORGUARDIn this section, we give a simple visualization for DetectorGuard
with YOLOv4 as Base Detector (Figure 16). To start with, we selectthree random images with larger objects and visualize the detection
output of YOLOv4 in the first column. Second, we pick a random
patch location on the image and perform an empirical patch hiding attack. The attack aims to optimize the pixel values within theadversarial patch to minimize the objectness confidence score of every possible bounding box prediction, which is a common strategy
used in relevant literature [57, 61]. As shown in the second column,
our patch attacks are successful, and YOLOv4 fails to detect any
objects. Note that we use red dash boxes to illustrate the patch
locations, and they are not the outputs of YOLOv4. Third, we feed
this adversarial image to Objectness Predictor, and we visualize
the predicted objectness maps in the third column. As shown in
the figure, although the adversarial patch makes YOLOv4 miss all
objects, Objectness Predictor still robustly outputs high objectness.

Session 11D: Data Poisoning and Backdoor Attacks in ML

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

As discussed in Section 3, DetectorGuard will eventually issue anattack alert. Fourth, we reason about the worst-case objectness map
prediction for these particular random patch locations used in thevisualization, and plot the worst-case output in the fourth column.
As shown in our visualization, Objectness Predictor can still output
high objectness in the worst case. Therefore, we can certify therobustness of DetectorGuard for this patch location.

Finally, we want to note that this appendix is merely a simple casestudy for an empirical patch attack at one single random locationof each image. In contrast, robustness results reported in Section 5
are derived from Algorithm 2 and Theorem 1 holds for any possible
patch attack strategy at any valid patch location.

3196