Fooling Detection Alone is Not Enough: Adversarial Attack against Multiple Object Tracking
Authors: Yunhan Jia
Yantao Lu
Junjie Shen
Qi Alfred Chen
Hao Chen
Zhenyu Zhong
Tao Wei
Venue:  (2020)
Keywords: Adversarial examples
object detection
object tracking
security
autonomous vehicle
deep learning

Abstract: Recent work in adversarial machine learning started to focus on the visual perception in autonomous driving and studied Adversarial Examples (AEs) for object detection models. However, in such visual perception pipeline the detected objects must also be tracked, in a process called Multiple Object Tracking (MOT), to build the moving trajectories of surrounding obstacles. Since MOT is designed to be robust against errors in object detection, it poses a general challenge to existing attack techniques that blindly target objection detection: we find that a success rate of over 98% is needed for them to actually affect the tracking results, a requirement that no existing attack technique can satisfy. In this paper, we are the first to study adversarial machine learning attacks against the complete visual perception pipeline in autonomous driving, and discover a novel attack technique, tracker hijacking, that can effectively fool MOT using AEs on object detection. Using our technique, successful AEs on as few as one single frame can move an existing object in to or out of the headway of an autonomous vehicle to cause potential safety hazards. We perform evaluation using the Berkeley Deep Drive dataset and find that on average when 3 frames are attacked, our attack can have a nearly 100% success rate while attacks that blindly target object detection only have up to 25%.

Published as a conference paper at ICLR 2020
FOOLING DETECTION ALONE IS NOT ENOUGH: AD- VERSARIAL ATTACK AGAINST MULTIPLE OBJECT TRACKING
Yunhan Jia 41 Yantao Lu " Junjie Shen', Qi Alfred Chen', Hao Chen', Zhenyu Zhong', Tao Wei
'Independent Researcher, "Syracuse University, SCC Irvine, 4UC Davis, 'Baidu X-Lab, Peking University
jack00820106 ammiil.com.y125 @ syr.ddu, salfchen.junjies// @ @ucr.edu, chen @ ucdavis.edu, edwardzhong @ baidu.com. com, lenx. wei gmail.com,
ABSTRACT
Recent work in adversarial machine learning started to focus on the visual percep- tion in autonomous driving and studied Adversarial Examples (AEs) for object detection models. However, such visual perception pipeline the detected objects must also be tracked, in process called Multiple Object Tracking (MOT), to build the moving trajectories of surrounding obstacles. Since MOT is designed to be robust against errors in object detection, it poses general challenge attack techniques that blindly target objection detection: we find that a success rate of over 98% is needed for them actually affect the tracking results, a requirement that no existing attack technique can satisfy. In this paper, we are the first to study adversarial machine learning attacks against the complete visual perception pipeline in autonomous driving, and discover a novel attack technique, tracker hijacking, that can effectively fool MOT using AEs on object detection. Using our technique, successful AEs on as few as one single frame can move existing object in to or out of the headway of an autonomous vehicle to cause potential safety hazards. We perform evaluation using the Berkeley Deep Drive dataset and find that on average when 3 frames are attacked, our attack can have nearly 100% success rate while attacks that blindly target object detection only have up to 25%.
existing
INTRODUCTION
Since the first Adversarial Example (AE) against traffic sign image classification discovered by Eykholt al. (Eykholt et al. 2018), several research work in adversarial machine learning (Eykholt et al. 2017: Xie et al.) 2017 Lu et al. | Zhao et al. 20186) Chen et al., 2018 Cao et al. 2019) started to focus the context of visual perception in autonomous driving, and studied AEs on object detection models. For example, Eykholt el al. (Eykholt et al. 2017) and Zhong el al. (Zhong al. 2018) studied AFs in the form of adversarial stickers stop signs the back of front cars against YOLO object detectors (Redmon & Farhadi (017). and performed indoor experiments to demonstrate the attack feasibility the real world. Building upon these work, most recently Zhao er al. (Zhao et al., 2018b) leveraged image transformation techniques to improve the robustness of such adversarial sticker attacks in outdoor settings, and were able to achieve 72% attack success rate with car running at constant speed of 30 km/h on real roads.
While these results from prior work are alarming, object detection in fact only the first half of the visual perception pipeline in autonomous driving, or in robotic systems in general - in the second half, the detected objects must also be tracked, a process called Multiple Object Tracking (MOT), to build the moving trajectories, called trackers, of surrounding obstacles. This is required for the subsequent driving decision making process, which needs the built trajectories to predict future moving trajectories for these obstacles and then plan driving path accordingly to avoid collisions with them. To ensure high tracking accuracy and robustness against errors in object detection, in MOT only the detection results with sufficient consistency and stability across multiple frames be included in the tracking results and actually influence the driving decisions. Thus, MOT in the visual
" Equual contribution

---

Published as a conference paper at ICLR 2020
Existing trackers track|c-1
Detection results
Tracking results
detc t
Multipie-
tracklt
Object
Tracking
1d:0
Fusion
Object Detection
Trk()
Planning
1 c+1
trock|r-1
D(-)
Data Association
ido
Control
Images captured time sequence
State
detcl++2
Prodiction
tracklee2
Xt...Kt
bbox, class
bbox, class, track_ id, velocity
Figure 1: The complete visual perception pipeline in autonomous driving, i.e., both object detection and Multiple Object Tracking (MOT) (Baidu) Kato et al. 2018 2015 Zhao et al. 201 Sa 2010
Ess et al.
MathWorks Udacity).
perception of autonomous driving poses general challenge to existing attack techniques that blindly target objection detection. For example, as shown by our analysis later in $ an attack on objection detection needs to succeed consecutively for at least 60 frames to fool representative MOT process,
which requires an at least 98% attack success rate ( ) To the best our knowledge, no existing
attacks objection detection achieve such high success rate (Eykholt et al., 2017 Xie et al.
2017 Lu et al.) [017a|b] Zhao et al. 2018b Chen et al. 2018).
the first to study adversarial machine learning attacks considering the complete visual perception pipeline in autonomous driving, i.e., both object detection and object tracking, and discover novel attack technique, called tracker hijacking, that effectively fool the MOT process using AEs on object detection. Our key insight is that although it is highly difficult to directly create tracker for fake objects or delete a tracker for existing objects, we can carefully design AEs to attack the tracking error reduction process in MOT to deviate the tracking results of existing objects towards an -rteieer-desired moving direction. Such process is designed for increasing the robustness and accuracy of the tracking results, but ironically, we find that it can be exploited by attackers to substantially alter the tracking results. Leveraging such attack technique, successful AEs on as few as one single frame is enough to move an existing object in to or out of the headway of an autonomous vehicle and thus may cause potential safety hazards.
In this paper,
We select 20 out of 100 randomly sampled video clips from the Berkeley Deep Drive dataset for evaluation. Under recommended MOT configurations in practice (Zhu et al. 2018) and normal measurement noise levels, find that our attack succeed with successful AEs frame, and 2 to consecutive frames on average. We reproduce and compare with previous attacks that blindly target object detection, and find that when attacking 3 consecutive frames, our attack has a nearly 100% success rate while attacks that blindly target object detection only have up to 25%
few
Contributions. In summary, this paper makes the following contributions:
We are the first to study adversarial machine learning attacks considering the complete visual perception pipeline in autonomous driving, i.e., both object detection and MOT. We find that without considering MOT, an attack blindly targeting object detection needs at least a success rate of 98% to actually affect the complete visual perception pipeline in autonomous driving, which is requirement that no existing attack technique can satisfy.
We discover a novel attack technique, tracker hijacking, that can effectively fool MOT using AES on object detection. This technique exploits the tracking error reduction process in MOT, and can enable successful AEs on as few as one single frame to move an existing object in to or out of the headway of autonomous vehicle to cause potential safety hazards.
The attack evaluation using the Berkeley Deep Drive dataset shows that our attack can succeed with successful AEs on as few as one frame, and only 2 consecutive frames on average, and when 3 consecutive frames are attacked, our attack has nearly 100% success rate while attacks that blindly target object detection only have up to 25%.
Code and evaluation data are all available GitHub (Github).
2

---

Published as a conference paper at ICLR 2020
2 BACKGROUND AND RELATED WORK
Adversarial examples for object detection. Since the first physical adversarial examples against traffic sign classifier demonstrated by Eykholt et al. (Eykholt et al., 2018). several work in adversarial machine learning (Bykholt et al., 2017, Xie et al. 2017 Lu et al.j 2017a b Zhao et al. 2018b] Chen et 2018)) have been focused on the visual perception task in autonomous driving, and more specifically, the object detection models. To achieve high attack effectiveness in practice, the key challenge is how to design robust attacks that can survive distortions in real-world driving scenarios such as different viewing angles, distances, lighting conditions, and camera limitations. For example, Lu et al. (Lu et al.] (017a) shows that AEs against Bester-RCNN (Ren al., 2015) generalize well across a sequence of images in digital space, but fail in most of the sequence in physical world; Eykholt et al. (Eykholt et al., (017) generates adversarial stickers that, when attached to stop sign, can fool YOLOv2 (Redmon & Farhadi 2017) object detector, while it is only demonstrated in indoor experiment within short distance; Chen et al. (Chen et al.) 2018)) generates AEs based expectation over transformation techniques, while their evaluation shows that the AES are not robust to multiple angles, probably due to not considering perspective transformations (Zhao et al.) 2018b). It was not until recently that physical adversarial attacks against object detectors achieve decent success rate (70%) in fixed-speed (6 km/h and 30 km/h) road test (Zhao al.. 2018b).
While the current progress in attacking object detection is indeed impressive, in this paper we argue that in the actual visual perception pipeline of autonomous driving, object tracking, or more specifically MOT, is integral step, and without considering it, existing adversarial attacks against object detection still cannot affect the visual perception results even with high attack success rate. As shown in our evaluation in 4 with common setup of MOT, an attack on object detection needs to reliably fool at least 60 consecutive frames to erase one object (e.g., stop sign) from the tracking results, in which case even 98% attack success rate object detectors is not enough ($4)
MOT background. MOT aims to identify objects and their trajectories in video frame sequence. With the recent advances in object detection, tracking-by-deeeeriien (Luo al. 2014) has become the dominant MOT paradigm, where the detection step identifies the objects in the images and the tracking step links the objects to the trajectories (i.e., trackers). Such paradigm is widely adopted
in autonomous driving systems today (Baidu) Kato et al. 2018 2015 Zhao et al. 20 Ess et al., 2010; Math Works; Udacity), and a more detailed illustration in Fig. D As shown, each detected objects at time f will be associated with a dynamic state model (e.g., position, velocity), which represents the past trajectory of the object (track| 1). per-track Kalman filter (Baidu Kato et al., 2018: Feng et al. 2019 Murray 2017 Yoon et al. (016) is used to maintain the state model, which operates in a recursive predict- -upatte loop: the predict step estimates current object state according to motion model, and the update step takes the detection results detele as measurement to update its state estimation result track!].
The association between detected objects with existing trackers is formulated as bipartite matching problem (Sharma et al., 2018 Feng et al. 2019; Murray, 2017) based on the pairwise similarity costs between the trackers and detected objects, and the most commonly used similarity metric is the spatial- based cost, which measures the overlapping between bounding boxes, or bboxes (Baidu
Long al. 2018 Xiang al. 2015 Sharma et al. 2018 Feng al. 2019 Murray 2017 Zhu et al. 2018 Yoon et al.] 2016 Bergmann et al. 2019 Bewley et al.) 2016). To reduce errors in this association, an accurate velocity estimation is necessary in the Kalman filter prediction (Choi] 2015 Yilmaz et al. 2006). Due to the discreteness of camera frames, Kalman filter uses the velocity model to estimate the location of the tracked object in the next frame in order to compensate the object motion between frames. However, as described later in $ such error reduction process unexpectedly makes possible to perform tracker hijacking.
MOT manages tracker creation and deletion with two thresholds. Specifically a new tracker will be created only when the object has been constantly detected for a certain number of frames, this threshold will be referred to as the hit count, or H in the rest of the paper. This helps to filter out occasional false positives produced by object detectors. On the other hand, a tracker will be deleted if no objects is associated with for duration of R frames, or called a reserved age. It prevents the tracks from being accidentally deleted due to infrequent false negatives of object detectors. The configuration of R and H usually depends on both the accuracy of detection models, and the frame rate (fps). Previous work suggest a configuration of R = 2- fps. and = 0.2- fps (Zhu et al. 2018). which gives R = 60 frames and H 6 frames for common 30 fps visual perception system. We
3

---

Published as a conference paper at ICLR 2020
Adversarial patch
Fabricated adversarial bbox
Erased original bbox
Original bbox recovered
Detection
Attack duration t=1
(b) Object move-in
Frames
t=0
t=2
t=3
OAF
Tracking
Track hijacked with adversarial valocity
Adversarial tracker will not deleted not tracked until
Original object will
(c) Object move-out
(a) Tracker hijacking attack overview
Figure 2: Description of the tracker hijacking attack flow (a). and two different attack scenarios:
object move-in (b) and move-out (c) where tracker hijacking may lead to severe safety consequences including emergency stop and rear-end crashes.
will show in $4 that attack that blindly targeting object detection needs to constantly fool least 60 frames (/t) to erase an object, while our proposed tracker hijacking attack can fabricate object that last for R frames and vanish target object for H frames in the tracking result by attacking as few as one frame, and only 2-3 frames on average
TRACKER HIJACKING ATTACK
Scope. This work focuses on the track-by-detection pipeline as described above, which has been recognized as the dominant MOT paradigm in recent literature (Long al. 20188] Murray 2017]
Sharma et al. 2018 Luo et al.) 2014) and MOT challenges (Dendorter et al. 2019). A MOT approach or similarity measures match objects across frames. Common measures include bounding box overlaps, object appearances, visual representations, and other statistical measures (Luo al., 2014). As the first study on the adversarial threats against MOT. we choose the loU-based Hungarian matching (Sharma et al. [2018) Feng et al. 22099) MMrray| 2017) as our target algorithm, as it is the most widely adopted and standardized similarity metric by not only very recent work (Long et al. 2018 Xiang et al.) 2015 Feng et al. 2019), but also two real-world autonomous driving systems, Le.. Baidu Apollo (Baidu) and Autoware (Kato al. 2018). This thus ensures the representativeness and practical significance of our work.
choose include
Overview. Fig. 2a illustrates the tracker hijacking attack discovered this paper, which an AE for object detection (e.g., in the form of adversarial patches on the front car) that can fool the detection result for as few as one frame can largely deviate the tracker MOT. As shown, the target car is originally tracked with a predicted velocity to the left at to. The attack starts at time f by applying an adversarial patch onto the back of the car. The patch carefully generated to fool the object detector with two adversarial goals: (1) erase the bounding box of target object from detection result, and (2) fabricate a bounding box with similar shape that is shifted a little bit towards an attacker-specified direction. The fabricated bounding box (red in detection result at 41) will be associated with the original tracker of target object in the tracking result, which we call hijacking of the tracker, and thus would give fake velocity towards the attacker-desired direction to the tracker. The tracker hijacking shown in Fig. 2a lasts for only one frame, but its adversarial effects could last tens of frames, depending on the MOT parameter R and H (introduced in $ For example, at time to after the attack, all detection bounding boxes are back to normal, however, two adversarial effects persist: (I) the tracker that has been hijacked with attacker-induced velocity will not be deleted until a reserved age (R) has passed, and (2) the target object, though is recovered in the detection result, will not be tracked until a hit count (H) has reached, and before that the object remains missing in the tracking result. However, it's important note that attack may not always succeed with one frame in practice, as the recovered object may still be associated with its original tracker, if the tracker is not deviated far enough from the object's true position during short attack
target object (c.g., a front car) in
4

---

Published as a conference paper at ICLR 2020
duration. Our empirical results show that attack usually achieves nearly 100% success rate when 3 consecutive frames are successfully attacked using AE ($4).
Such persistent adversarial effects may cause severe safety consequences in self-driving scenarios. We highlight two attack scenarios that can cause emergency stop or even a rear-end crashes:
adversarial patch be placed on roadside objects, e.g., a parked vehicle to deceive visual perception of autonomous vehicles passing by. The adversarial patch is generated cause translation of the target bounding box towards the center of the road in the detection result, and the hijacked tracker will appear as a moving vehicle cutting in front in the perception of the victim vehicle. This tracker would last for seconds if R is configured as 2- fps as suggested in (Zhu et al., 2018), and tracker hijacking in this scenario could cause an emergency stop and potentially a rear-end crash.
Attack scenario 1: Target object move-in. Shown in Fig. 2b
Atta Attack scenario 2: Target object move-out. Similarly, tracker hijacking attack can also deviate objects in front of the victim autonomous vehicle away from the road cause crash shown in Fig. 2c Adversarial patch applied on the back of front car could deceive MOT of autonomous vehicle behind into believing that the object is moving out its way, and the front car will be missing from the tracking result for duration of 200ms, if H uses the recommended configuration of 0.2. fis (Zhu al. 2018). This may cause the victim autonomous vehicle to crash into the front car.
3.1
ATTACK METHODOLOGY
Algorithm Tracker Hijacking Attack
Video image sequence X = [ma, F1, .... #n]: object detector D(:); MOT algorithm Trk(-); Input: Index of target object to be hijacked K, attacker-desirg directional velocity ग, adversarial patch area mask matrix patch.
Input:
Output: Sequence of adversarial examples X" = [.,, requireed Initialization X' O). detclo D(ro), track|oo (currenl_Iracka)
required for successful attack.
for to n do
2: detc|; +- D(x,)
3:
if detcl,[K] matches track).-1 [K] then target object matches with an existing tracker find position pos place fabricated bbox with Eq. D
4:
pos + FINDPos(TTk(()
K, J. patch) see Alg. 2 in Appendix
5:
generate adversarial frame x' with Eq. 3 D attack object detector with specialized loss
x f- GENERATEADVV(; D(-), pos, K, Katch)
see Alg. in Appendix
6:
_ x/
7:
else
8:
return X"
> attack succeeds when target object is not associated with original tracker Di update current tracker with adversarial frame
9:
end if
10: trackle t- Trk(krakk (xx)))
end for
Targeted MOT design. Our attack targets on first-order Kalman filter, which predicts state vector containing position and velocity of detected objects over time. For the data association, we adopt the mostly widely used Intersection over Union (IoU) the similarity metric, and the IoU between bounding boxes are calculated by Hungarian matching algorithm (Luetteke et al., 2012) to solve the bipartite matching problem that associates bounding boxes detected in consecutive frames with existing trackers. Such combination of algorithms in the MOT is the most common in previous work (Long al. 2018 Xiang et al. 2015; Sharma et al. 2018) and real-world systems (Baidu).
We now describe our methodology of generating an adversarial patch that manipulates detection results to hijack a tracker. As detailed in Alg. [ given targeted video image sequence, the attack iteratively finds the minimum required frames perturb for successful track hijack, and generates the adversarial patches for these frames. In each attack iteration, an image frame in the original video clip is processed, and given the index of target objects K, the algorithm finds optimal position to place the adversarial bounding box pos in order to hijack the tracker of target object by solving Eq. T. The attack then constructs adversarial frame against object detection model with adversarial
5

---

Published as a conference paper at ICLR 2020
car
Detection
Tracking
d:
(b) Existing object detection attack
Data association range with original bbox
Detection
Tracking
Optimal position for adv bbox given velocity
(a) Finding position to fabricate adversarial bounding box
Our tracker hijacking attack
Figure 3: Comparison between previous object detection attack and our tracker hijacking attack. Previous attack that simply erase the bbox has no impact on the tracking output (b). while tracker hijacking attack that fabricates bbox with carefully chosen position successfully redirects the tracker towards attacker-spp setacker-specified direction (c).
patch, using Eq. as the loss function to erase the original bounding box of target object and fabricate the adversarial bounding box at the given location. The tracker is then updated with the adversarial frame that deviates the tracker from its original direction. If the target object in the next frame is not associate with its original tracker by the MOT algorithm, attack has succeeded; otherwise, this process is repeated for the next frame. We discuss two critical steps this algorithm below, and please refer to the Appendix A for the complete implementation of the algorithm.
Finding optimal position for adversarial bounding box. To deviate the tracker of target object K, besides removing its original bounding box |etc| the attack also needs to fabricate adversarial box with a shift å towards a specified direction. This turns into an optimization problem (Eq- D of finding the translation vector å that maximizes the cost of Hungarian matching (M(-) between the detection box and the existing tracker so that the bounding box is still associated with its original tracker (M A), but the shift is large enough to give adversarial velocity to the tracker. Note that we also limit the shifted bounding box to be overlapped with the patch to facilitate
adversarial example generation, as it's often easier for adversarial perturbations affect prediction results in its proximity, especially in physical settings (Chen et al. 2018
max M(detc) K 5, track) [K])
MM S X, LoU (detc) [K] patch) 7
Generating adversarial patch against object detection. Similar to the existing adversarial attacks against object detection models (Chen et al. 2018: Eykholt et al. 2018 Zhao et al. 2018b), we also formulate the adversarial patch generation
optimization problem shown in Eq. in Appendix. Existing attacks without considering MOT directly minimize the probability of target class (e.g., a stop sign) to erase the object from detection result. However, shown in Fig. 3b| such AEs highly ineffective in fooling MOT as the tracker will still track for It frames even after the detection bounding box erased. Instead, the loss function of our tracker hijacking attack incorporates two optimization objectives: (1) minimizes the target class probability to erase the bounding box of target object; (2) fabricates the adversarial bounding box at the attacker-desired location and in the specific shape to hijack the tracker. Details of our algorithm can be found in Appendix A, and the implementation can be found at (Github).
4 ATTACK EVALUATION
In this section, we describe our experiment settings for evaluating the effectiveness of our tracker hijacking attack, and comparing it with previous attacks that blindly attack object detection in detail.
4.1 EXPERIMENT METHODOLOGY
Evaluation metrics. We define successful attack that the detected bounding box of target object can no longer be associated with any of the existing trackers when attack has stopped. We measure the effectiveness of our track hijacking attack using the minimum number of frames that the AEs

---

Published as a conference paper at ICLR 2020
Move in Rech Hel
F trammes need to fool
4.5
Move-is (e60) yeo
100
R--00), Hed
901 FUEA Ch-)
Narmal contguration rarge
Track Macking o00e10
Track hecerg- -onrid Deteriion stack
101
10 Measurememi noise coveriance
100
5
3 of hameo can attok tool the detector R-60, H-B
3 tomes can attckt tool the dotector
R-5, H-2
(a) Frames required be fooled for a successful tracker hijack
(b) Attack success rate R 60 H =, 6, and R == 5, 17 =2
In normal measurement noise covariance range (a) our tracker hijacking attack would require the AE (adversarial example) fool only 2-3 consecutive frames on average to successfully deviate the target tracker despite the (R, H) settings. We also compare the success rate of tracker hijacking with previous adversarial attack against object detectors only under different attacker capabilities, i.e., the number of consecutive frames the AE can reliably fool the object detector (b). Tracker hijacking achieves superior attack success rate (100% even by fooling as few as frames, while previous attack is only effective when the AE can reliably fools at least R consecutive frames.
Figure 4:
object detection need to succeed. The attack effectiveness highly depends on the difference between the direction vector of the original tracker and adversary's objective. For example, attacker can cause a large shift on tracker with only one frame if choosing the adversarial direction to be opposite to its original direction, while it would be much harder deviate the tracker from its established track, if the adversarial direction happens to be the same as the target's ' original direction. To control the variable, we measure the number of frames required for our attack in two previous defined attack scenarios: target object move-in and move-out. Specifically, in all move-in scenarios, we choose the vehicle parked along the road as target, and the attack objective is to move the tracker to the center, while in all move-out scenarios, we choose vehicles that are moving forward, and the attack objective is to move the target tracker off the road.
Dataset selection. We randomly sampled 100 video clips from Berkeley Deep Drive dataset Yuu ctt al. 2018), and then manually selected 10 suitable for the object move-in scenario, and another IO for the object move-out scenario. For each clip, manually label target vehicle and annotate the patch region to be a small area at its back as shown in Fig. 3c
All videos are 30 frames per second.
Implementation details. We implement our targeted visual perception pipeline using Python, with YOLOv3 (Redmon & Farhadi 2018) as the object detection model due to its high popularity among in real-time systems. For the MOT implementation, we use the Hungarian matching implementation called 1 linear a assiisssnnn inment in the sklearn package for the data association, and we provide a reference implementation of Kalman filter based on the one used in OpenCV (OpenCV).
The effectiveness of attack depends on a configuration parameter of Kalman filter, called measurement noise covariance (cov). cov is an estimation about how much noise in the system, a low cov value would give Kalman filter more confidence on the detection result at time t when updating the tracker, while a high cou value would make Kalman filter to place trust more on its own previous prediction at time / 1 than that at time L. We give detailed introduction of configurable parameters in Kalman filter in $2 of our Appendix B. This measurement noise covariance often tuned based on the performance of detection models in practice. We evaluate our approach under different cav configurations ranging from very small (10-3) to very large (10) as shown in Fig. 4al while cou is usually set between 0.01 and 10 in practice (Baidu Kato al. 2018
4.2 EVALUATION RESULTS
Effectiveness of tracker hijacking attack. Fig. shows the average number of frames that the AEs on object detection need to fool for a successful track hijacking over the 20 video clips. Although a configuration with Rt = 60 and H = is recommended when fps is 30 (Zhu et al. 2018), we still test different reserved age (Rt) and hit count (!f) combinations real-world deployment are usually more conservative and use smaller R and H (Baidu) Kato et al. 2018). The results show that tracker hijacking attack only requires successful AEs on object detection in to 3 consecutive frames average to succeed despite the (R, H) configurations. We also find that even with a successful AE on only one frame, our attack still has 50% and 30% success rates when cov is 0.1 and 0.01 respectively.
7

---

Published as a conference paper at ICLR 2020
Interestingly, we find that object move-in generally requires less frames compared with object move-out. The reason is that the parked vehicles in move-in scenarios (Fig. 2bb naturally have a moving-away velocity relative to the autonomous vehicle. Thus, compared to move-out attack, move- in attack triggers a larger difference between the attacker-desired velocity and the original velocity. This makes the original object, once recovered, harder to associate correctly, making hijacking easier.
Comparison with attacks that blindly target object detection. Fig. 4b shows the success rate of our attack and previous attacks that blindly target object detection (denoted as detection attack). We reproduced the recent adversarial patch attack on object detection from Zhong et al. (Zhong et al. 2018). which targets the autonomous driving context and showed effectiveness using woaldworld car testing. In this attack, the objective is to erase the target class from the detection result of each frame. Evaluated under two (R, H) settings, we find that our tracker hijacking attack achieves superior attack success rate (100%) even by attacking as few as 3 frames, while the detection attack needs to reliably fool at least R consecutive frames. When is set to 60 according to the frame rate of 30 fps, the detection attack needs to have adversarial patch that can constantly succeed at least 60 frames while the victim autonomous vehicle is driving. This means an over 98. 98.3%
AE success rate, which has never been achieved even got close to in prior work (Zhao et al. 2018b Eykholt et al., 2017; Chen et al. 2018: Lu et al. 2017a). Note that the detection attack still can have up to -25% success rate before R. This is because the detection attack causes the object disappear for some frames, and when the vehicle heading changes during such disappearing period, it is still possible to cause the original object, when recovered, to misalign with the tracker predication in the original tracker. However, since our attack is designed to intentionally mislead the tracker predication in MOT, our success rate substantially higher (3-4)) and can reach 100% with few frames attacked.
5 DISCUSSION AND FUTURE WORK
Implications for future research in this area. Today, adversarial machine learning research target- ing the visual perception in autonomous driving, no matter on attack or defense, uses the accuracy of objection detection as the de facto evaluation metric (Luo et al. 2014). However, as concretely shown in our work, without considering MOT, successful attacks on the detection results alone do not have direct implication final output of the visual perception task real-world autonomous driving (Baidu Kato et al. 2018). Thus, we argue that future research in this area should consider: (1) using the MOT accuracy the evaluation metric, and (2) instead of solely focusing on object detection, also studying weaknesses specific to MOT or interactions between MOT and object detection, which is highly under- mnder-explored research space today. This paper marks the first research effort towards both directions.
equally or even closely successful attacks on the MOT results, the
Practicality improvement. Our evaluation currently are all conducted digitally with captured video frames, while our method should still be effective when applied to generate physical patches. For example, our proposed adversarial patch generation method can be naturally combined with different techniques proposed by previous work to enhance reliability of AFs in the physical world (e.g.. non-printable loss (Sharif et al. (016) and expectation-over tor-sorttoonooooottion (Athalye et al. 2017)) We leave this as future work.
Generality improvement. Though in this work we focused on MOT algorithm that uses IoU based data association, our approach of finding location to place adversarial bounding box is generally applicable to other association mechanisms (e.g.. appearance- based matching). Our AE generation algorithm against YOLOv3 should also be applicable to other object detection models with modest adaptations. We plan to provide reference implementations of more real-world end-to-end visual perception pipelines to pave the way for future adversarial learning research in self-driving scenarios.
CONCLUSION
In this work, we are the first to study adversarial machine learning attacks against the complete visual perception pipeline in autonomous driving, i.e., both object detection and MOT. We discover a novel attack technique, tracker hijacking, that exploits the tracking error reduction process in MOT and can enable successful AEs as few as one frame to move existing object in to or out of the headway of an autonomous vehicle to cause potential safety hazards. The evaluation results show that on average when 3 frames attacked, our attack can have nearly 100% success rate while attacks that blindly target object detection only have up to 25%. The source code and data is all available at (Github).
8

---

Published as a conference paper at ICLR 2020
Our discovery and results strongly suggest that MOT should be systematically considered and incorporated into future adversarial machine learning research targeting the visual perception in autonomous driving. Our work initiates the first research effort along this direction, and we hope that can inspire more future research into this largely overlooked research perspective.
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for providing valuable feedback on our work. This research supported in part by the National Science Foundation under grants CNS-1850533 and CNS-1932464.
REFERENCES
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. arXiv preprint arXivv1707.073397 2017.
Baidu. Baidu Apollo. httpss//ggttha.cco//popooaaaattt
Philipp Bergmann, Tim Meinhardt, and Laura Taiéé. Tracking without bells and whistles. CoRR, htss/9903.55620 2019. URL http::t/taaaatee org/ abs 1903 00625
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Uperoft. Simple online and realtime tracking. In 20/6 IEEE International Conference on Image Processing (ICIP), pp. 3464 3468.
IEEE, 2016.
Yulong Cao, Chaowei Xiao, Benjamin Cyr, Yimeng Zhou, Won Park. Sara Rampazzi, Qi Alfred Chen, Kevin Fu, and Zhuoqing Morley Mao. Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Driving. In ACM Conference Computer and Communications Security (CCS), 2019.
Shang-Tse Chen, Cory Cornelius, Jason Martin, and Duen Horng Polo Chau. Shapeshifter: Robust physical adversarial attack on faster r-cnn object detector. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 52-68. Springer, 2018.
Wongun Choi. Near-online multi-target tracking with aggregated local flow descriptor. In Proceedings of the IEEE international conference on computer vision, pp. 3029-3033 2015.
Dendorfer, H. Rezatofighi, A. Milan, Shi, Cremers, I. Reid, Roth, K. Schindler, and L. Leal- Taixé. CVPR19 19 tracking and detection challenge: How crowded can it get? arXiv: /ttv:/906.044 04567 [cs], June 2019. URL http::tt:://tt httt:://arxiv..oooo http:///wxxv..ooaa//////ea
04567
arXiv: 1906.04567.
Andreas Ess, Konrad Schindler, Bastian Leibe, and Lue Van Gool. Object detection and tracking for autonomous navigation in dynamic environments. The International Journal of Robotics Research, 29(14):1777-1725, 2010.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning models. arXiv preprint /rXiv:/707.000
2017.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1625-1634, 2018.
Weitao Feng. Zhihao Hu, Wei Wu, Junjie Yan, and Wanli Ouyang. Multi-object tracking with multiple cues and switcher-aware classification. arXiv preprint arXiv: 1oXi::9901.0111 2019.
Github. Github repository for the source code of our attack and evaluation data. https://gittne
com/ homm/anonymous jack://wuojjjaaaaooooo aakk https//jaaaaaaaaaacccc.... jacking
Shinpei Kato, Eijiro Takeuchi, Yoshio Ishiguro, Yoshiki Ninomiya. Kazuya Takeda, and Tsuyoshi
Hamada. An open approach to autonomous vehicles. IEEE Micro, 35(6):60-68, 2015.

---

Published as a conference paper at ICLR 2020
Shinpei Kato, Shota Tokunaga, Yuya Maruyama, Seiya Maeda, Manato Hirabayashi, Yuki Kitsukawa, Abraham Monrroy, Tomohito Ando, Yusuke Fujii, and Takuya Azumi. Autoware on board: enabling autonomous vehicles with embedded systems. In ICCPS' 18, pp. 287-296. IEEE Press, 2018.
Chen Long, Ai Haizhou, Zhuang Zijie, and Shang Chong. Real-time multiple people tracking with deeply learned candidate selection and person re-identification. In ICME. 2018.
Jiajun Lu, Hussein Sibai, and Evan Fabry. Adversarial examples that fool detectors. arXiv preprint arXiv: 2017a.
Jiajun Lu, Hussein Sibai, Evan Fabry, and David Forsyth. Standard detectors aren't (currently) fooled by physical adversarial stop signs. arXiv preprint arXiv:/710.005: 2017b.
Felix Luetteke, Xu Zhang, and Jörg Franke. Implementation of the hungarian method for object tracking on a camera monitored transportation system. In ROBOTIK 2012; 7th German Conference on Robotics, pp. 1-6. VDE, 2012.
Wenhan Luo, Junliang Xing, Anton Milan, Xiaoqin Zhang, Wei Liu, Xiaowei Zhao, and Tae-Kyun Kim. Multiple object tracking: literature review. arXiv preprint arXiv:1409.7618, 2014.
WorkS. Automated driving toolbox.
auttmmtte-ddrivingg html
Samuel Murray. Real-time multiple object tracking-a study on the importance of speed. arXiv preprint arXiv: 03572, 2017.
Alexander Neubeck and Lue Van Gool. Efficient non-maximum suppression. In 18th International Conference Pattern Recognition (ICPR (C6), volume 3, pp. 850 855. IEEE, 2006.
OpenCV. Kalman Filter Class Reference. /ttps://abss.opeecvvooo
/6a
classcv 1. IKalmanFilter. html
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE
conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Joseph Redmon and Ali Farhadi. Yolov3:
An incremental improvement. arXiv preprint
wxXiv:180..02767 2018
Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to a crime: Real and stealthy attacks state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1528-1540. ACM, 2016.
Sarthak Sharma, Junaid Ahmed Ansari, Krishna Murthy, and K Madhava Krishna. Beyond pixels: Leveraging geometry and shape cues for online multi-object tracking. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3508-3515 IEEE, 2018
Udacity. Self-driving car engineer nanodegree program.
https: www. uducityay..... com/
Y. Xiang, A. Alahi, and S. Savarese. Learning to track: Online multi-object tracking by decision making. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4705-4713, Dec 2015. doi: 10. 109/ICCV.0015.544
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In Proceedings of the IEEE International Conference Computer Vision, pp. 1369-1378 2017.
H. Yeh. Real-time implementation of narrow-band kalman filter with floating point processor dsp32. IEEE Transactions on Industrial Electronics, 37(1):13-18 18, Feb 1990. ISSN 0278-0046. doi: 10.1109/41 1011//1145883
10

---

Published as a conference paper at ICLR 2020
Alper Yilmaz, Omar Javed, and Mubarak Shah. Object tracking: A survey, Acm computing surveys (CSUR), 38(4):13, 2006.
J. H. Yoon, C. Lee, M. Yang, and K. Yoon. Online multi-object tracking via structural constraint event aggregation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, doi: 10. 109/CVPR.2016.1....
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: diverse driving video database with scalable annotation tooling. arXiv preprint arXiv: wrXiv:1805.04687, 2018.
Dawei Zhao, Hao Fu, Liang Xiao, Tao Wu, and Bin Dai. Multi-object tracking with correlation filter for autonomous vehicle. Sensors, 18(7):2004, 2018a.
Yue Zhao, Hong Zhu, Qintao Shen, Ruigang Liang, Kai Chen, and Shengzhi Zhang. Practical adversarial attack against object detector. arXiv preprint arXiv:1812.10217. 2018b.
Zhenyu Zhong, Weilin Xu, Yunhan Jia, and Tao Wei. Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-Based Object Detection. In Black Hat Europe, 2018. 8.
Ji Zhu, Hua Yang, Nian Liu, Minyoung Kim, Wenjun Zhang, and Ming-Hsuan Hsuan Yang. Online multi- object tracking with dual matching attention networks. In Computer Vision ECCV 2018, pp. 379-396, Cham, 2018. Springer International Publishing. ISBN 978- 978-3-030-01222-11

---

Published as a conference paper at ICLR 2020
A TRACK HIJACKING ATTACK DETAILS
Given the targeted video image sequence, track hijacking attack iteratively finds the minimum required frames to perturb for a successful hijack, and generates the adversarial patches for these frames. An image frame in the original video clip is given each iteration and an optimal position to place the adversarial bounding box pos in order to hijack the tracker of target object.
Alg. to find
The FINDPOS takes the existing tracking result track|s--1, the detected objects detcle, the index of
target object K, the attacker desired directional vector U, the adversarial patch area patch as input,
and iteratively moves the bounding box along the direction of रे while keeping some invariants: (1)
the shifted bounding box shall still be associated with the original tracker of target object (Eq. 2)
(2) the shifted bounding box shall always have overlap with the patch (IoU (dete' [K], patch) > ?).
The while e loop will end when the bounding box has been shifted to the farmost position from its
original position along V. where the invariants still hold. The intuition behind FINDPOS is that, in
order for the tracker to loss track of the target object when attack has ended, attacker needs to deviate
the bounding box of target object as far as possible within its original data association range.
max MaxxM(ddecc+a [K] + trackl [K])
s.t. M x, ToU 5, patch) 7
Algorithm/ 2 Track Hijacking Attack Find fabricated bbox position
Input: Existing trackers track] -- detection objects detc MOT algorithm Trk(-)
Input: Index of target object to be hijacked K, attacker desired directional vector ग; adversarial
patch area as mask matrix patch
Output: fabricate bounding box position pos
1: procedure FINDPOS
2: detd &- detcle
3: track' < trackle-1
4: k+1
5:
while dete' [K] matches track[[K] [K] and IoU ToU(deté (),, >> do delc'|K] + brack'(K) + v-
6:
7:
brack' Tr(baack delc!)
8: k = k +1
9:
end while
10: pos = track' [K] + J. (k- -10
return pos
12:
end procedure
After the target bounding box location is identified, the next step is to generate adversarial patch against the object detection model. Similar to the existing adversarial attacks against object detection models (Chen et al. 2018 Eykholt et al. 2018 Zhao et al. 2018b), we also formulate the adversarial patch generation as an optimization problem shown in Eq. 3 Existing attacks without considering MOT directly minimize the probability of target class (c.g.. stop sign) erase the target from detection result. However, as shown Fig. 3b such AEs are highly ineffective in fooling MOT as the tracker will still track for R frames even after the detection bounding box is erased. Instead, the loss function of our tracker hijacking attack incorporates two loss terms: L1 minimizes the target class probability at given location to erase the target bounding box, where Di=n 1 identifies all bounding boxes (B) before non-max suppression (Neubeck & Van Gool, 2006), who contain the center location (CDz, cyt) of pos, while C; is the confidence score of bounding boxes; Le controls the fabrication of adversarial bounding box at given center location (cx,, cyt) with given shape (w,, he) to hijack the tracker. In the implementation, we use Adam optimizer to minimize the loss by iteratively perturbing the pixels along the gradient directions within the patch area, and the generation process stops when adversarial patch that satisfies the requirements is generated. Note that the fabrication loss La needs only to be used when generating the first adversarial frame in a sequence to give the tracker an desseker-deiired velocity V. and then À can be to 0 to only focus erasing
12

---

Published as a conference paper at ICLR 2020
target bounding box similar to previous work. Thus, our attack wouldn't 't add much difficulty to the optimization. The code of our implementation can be found at (Github).
min LI + A) A- A-( + A) AEpotoh
B
L
1 bbj I - CrossEntropp(pe:
im0
(3)
22
Σ 1 bbbj, [l(cr; cr,)² (cy; cy,)) I( vw,)2 2 + (Vh, vh,221
1=0
C1) + Crosssrnroppppyy class;))
Alg. takes the adversarial bounding box position pos for fabrication, and the original bounding box for vanish generate an adversarial frame x' whose perturbation is limited in the patch area. Similar to the existing adversarial attacks against object detection models (Chen et al. 2018 Eykholt al. 2018 Zhao al. 2018b), we also formulate the adversarial patch generation an optimization problem. First, the algorithm identifies all bounding boxes E B in the intermediate result of object detection model before non-max suppression (Neubeck Van Gool, 2006), and for all of them who contain the central point Ca. Cy of pos in its bounding box area, initialize I; + 1, otherwise, 1, <- 0. The algorithm then use Adam optimizer minimize the loss C1 + XC2 where & minimizes the target class probability in the vanish area, and L2 controls the fabrication of adversarial bounding box at given center location (CT,, Cit) with given shape (wi, h2) to hijack the tracker. Note that the fabrication loss L2 needs only to be used when generating the first adversarial frame in a sequence to give the tracker an attacker-desired velocity, and then can be set to 0 to only focus on erasing target bounding box similar to previous work. Also note that when calculating the pixel gradient, we apply a mask patch to the input x to restrict the perturbation area. The attack stops when the maximum attack iteration has reached, and the adversarial example with the patch applied is returned. The implementation is available at (Github).
B KALMAN FILTER IMPLEMENTATION
The main idea behind Kalman filter that the measurement result is not always reliable, and by combining a statistic noise model, the estimation can be more accurate than base on single measurement alone. This makes Kalman filter natural fit for the maak-by--detcction pipeline, MOT is intended to tolerate and correct the occasional errors in the detection result. The main principle of Kalman filter represented as Eq.
$ K3 Zz + (1 Kg) -x-1
where &t is the current state estimation, Kg is the Kalman gain, Z₁. is the measurement value at state k. and - is the previous estimation. The equation shows that Kalman filter performs the state estimation using both the current measurement result and the previous estimation, while the Kalman gain K2 is also variable that will be updated by measurements.
In the MOT applications, the state estimations are the trackers, while the measurements are the detected bounding boxes at each frame. In this paper, we use first-order Kalman filter to track the central point location(r, e) of bounding boxes, and first-order low-pass filter to track the width and length of bounding boxes with decay factor 0.5, which the same as Baidu Apollo self-driving platform's 's implementation (Baidu).
The update of the tracker states are updated with two steps: the time update, and the measurement update. The time update is performed as:
f. = Fx 2x-1
Pk = Fx Pk-1 FT Qk
where F3 is the first-order state transition model, and Pk is the posteriori error covariance matrix, which is measure of the estimated accuracy of the state estimate. The QK is the covariance of the
13

---

Published as a conference paper at ICLR 2020
Algorithm 3 Track Hijacking Attack Generate AE against object detection model
Input: Input image æ; object detector D(); all bounding boxes B in ((z) before non-max suppres-
sion; fabricated bbox position pos, attack iterations N; Input: Index of target object to be hijacked K; adversarial patch area as a mask matrix patch.
Output: Adversarial example image x'.
1: procedure GENERATEADV
2: (Ca, Gy) + central point of pos
3: for all bboxes B do
4: if bbox contains (Cz, Cy) then
5:
1,
6:
end if 1, < end for
7:
8:
9:
x' <-
10:
for to N do
11:
Calculate vanish loss L1:
C1
Σ obj [C₁ Crosssonroopyy,, class:)] 1=0
Calculate fabricate loss L2:
B [l(co; 2 (cy cy ((w; - vwx)² (Vhi- vh,)
L2
Σ
1.
+(1 C,) CrossEnnropppppccasss
if is not the first frame attack then
14:
À <-60 0
end Implement Adam optimizer to calculate pixel gradients
16:
grad = Adamm x, C1 + AC2)
17:
x' x- x' + grad
18:
end for
19: return
20:
end procedure
14

---

Published as a conference paper at ICLR 2020
process noise. The measurement update is performed in the same loop
K₁ Pk- H (Hk Pk
H + R
4 R + K' Hk. .tx)
Pk Pk K' Pk
where Hi the observation model, Rk the covariance of the observation model, and Zn the observation. In particular, denoting the coordinates of center point as (r, c). we set the state vector 7 and state covariance matrix as:
Pr
Ep.Pr
Ep
EprPe
Epr
EurPr
Evper
EvpPe
Ee
P=
Pe
EpePr
Epeir
EppPo
Epola
Eeepr
Loorr
Ev.Pe
EoeVe_
and we set the the state transition matrix F and the process covariance matrix Q as:
1
dt
0
dt /4
dt
/2
0
0
0
0
0
dt
/2
dt
0
0
0
db
Q
=
0
dt
/4
dt
/2
0
1
0
di
//
di
The observation matrix and the measurement covariance R are set to be:
0
0
H
0
[ 07 1
R Cov X
0
where the cou is the variable we refered to as measurement noise covariance value we enumerated in evaluation. From the expression of the Kalman gain in the measurement update process, can see that the gain factor K' is related to variations R. Identified by H.-G. Yeh et al. Yeh 1990). the Kalman gain can be regarded as ratio of dynamic process the measurement noise, i.e., K proportional to Q con'T. So when the cou value small, the object tracking response is relatively fast, and the tracking bounding boxes follow the detection boxes more closely; while when the cou value is large, the Kalman filter trust more on its own estimation rather than the measurement, and the tracker is less responsive to the change of bounding boxes, which makes our track hijacking attack little bit harder. In our paper, we empirically validated the impact of different cov values [0, 0.01, 0.1, 1, 10] on the effectiveness of our attack, and found that under normal cov configuration range (0.01 to 10), our attack can get a nearly 100% success rate by fooling 3 consccutive detection frames on average.
15


Reviews:
[{"review_id": "SylVQsFaKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about conducting evasion attacks against Multiple Object Tracking (MOT) techniques. Compared to existing work on adversarial examples against object detection, to attack MOT techniques, the adversary needs to successfully fool multiple frames, and the authors show that by naively using existing attack approaches, the adversary needs to achieve 98% single-frame attack success rate to fool the tracking system, which is too hard for existing attack algorithms.  Therefore, this paper proposes a smart way of attacking MOT techniques by leveraging the properties of the tracking algorithm. In particular, they generate adversarial perturbations to remove the original bounding box while adding a fake bounding box that has some overlap with the original bounding box, so that the system will compute the movement of the object in a wrong way. They evaluate on videos in Berkeley Deep Drive dataset, and show that by attacking 2~3 frames, they can achieve nearly 100% attack success rate, while the attack success rate is only 25% if the tracking algorithm is not considered when crafting the attacks.\n\nThere has been a long line of work studying adversarial attacks against autonomous driving systems, which aims at revealing the security threat of such attacks in the real world. While the community has made promising progress, to my best knowledge, this paper is the first work considering attacking MOT techniques, and they suggest the importance of this attack scenario by demonstrating that attacking objection detection alone is not sufficient in terms of fooling the tracking technique. Meanwhile, the proposed attack method is interesting and effective, even if the perturbation is only bounded in a patch located at a reasonable place in the frame, e.g., the target car to attack.  Therefore, I lean towards accepting this paper, as it contributes to a new perspective of adversarial attacks.\n\nHowever, I have some questions about evaluation.\n\n1. To generate adversarial frames for the trajectory of the same car, is the adversarial patch inserted since the first frame and always stays the same, or different patches are needed to attack different frames of the same trajectory? I feel that if the adversarial patch needs to change a lot in the entire trajectory, then the practicality of the attack would be compromised.\n\n2. How large does the adversarial patch need to be in order to successfully launch the attacks? And does the position of the patch affect the attack performance?\n\n3. I wonder if the same adversarial patch may work beyond a single context, i.e., with the same patch, the same car can fool the tracking technique with different background scenes. I am not familiar with the details of the dataset in their evaluation, but it would be helpful to show relevant results if it is easy to simulate using their dataset.\n\n------------\nPost-rebuttal comments\n\nI thank the authors for their response, and I keep my original assessment.\n------------", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Syepl_jkqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies adversarial attacks against MOT and proposes a tracker hijacking attack that can successfully fool the MOT with a high success rate even when only adding patches to 3 consecutive frames. \n\nThis work is the most realistic attack I have seen so far on attacking the visual perception system of the self-driving car since it not only considers object detection but also the object tracking, which is critical for autonomous driving system. The paper is well-motivated and illustrates the importance of the problem.\n\nAccording to my knowledge, this is the first paper to study the complete visual perception pipeline in autonomous driving. Previous work along this line focuses on attacking the object detection task, however, this work considers object tracking, which is essential for a safety-critical system as temporarily failure of object detection for one frame usually won’t hurt the decision. Thus, it is an important step towards a more realistic attack on the visual perception pipeline in autonomous driving.\n\nThe paper also presents a thorough background of MOT. Their proposed algorithm works much better than other alternatives.\n\nWeakness:\nI wish the evaluation can be conducted a bit more broadly. Currently, it is only evaluated 20 clips from one dataset.\n\nMinor points\nIn Eqn (1), \\lambda and \\gamma are not defined. \nPage 5 last line, “associate” should be “associated”.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Hke0M2aaKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses adversarial attacks against visual perception pipelines in autonomous driving. Both subprocesses in the visual perception pipeline, object detection and multiple object tracking (MOT), are considered. The paper proposes a novel approach in adversarial attacks, the tracking hijacking, which can fool the MOT process using Adversarial Examples (AEs) in object detection. The key idea is to exploit the tracking error to place specific attacks on single frames in MOT, which can lead to a displacement of the detected objects. It is shown that the proposed method can effectively attack the perception pipeline by just fooling 2 to 3 consecutive frames on average.\n\nAdvantages\nThe paper is well written and the topic generally of interest to the ICLR community. The paper also provides a nice overview on the processing pipeline. Technically, it appears relatively straight foward; the main contribution lies perhaps in the interplay of detection and tracking that may render this approach interesting to practitioners. \n\nComments\nThe authors state that smaller thresholds R and H are tested since those are more conservative in real-world scenarios. However, success rates for both (R, H) settings are shown in Figure 4. It does not become obvious whether smaller values for R and H values are tested in addition.\n\nThe authors state that a previous attack is only effective when AE can reliably fool at least R consecutive frames. Since it is more conservative in current real-world development to use smaller thresholds R and H, the previous attack could also be successful fooling the object detection. But even if the threshold is set to R = 60 and H=6, tracking hijacking is shown to be successful by just fooling 3 frames. This fact, however, is not fully prominent in the text. It would be helpful to describe this in more detail in the effectiveness section.\n\nMinor comments:\nProofreading is necessary (e.g., top of page 6)\nRepetitions of sentences: \n- \"[...] successful AEs on as few as one single frame, and 2 to 3 consecutive frames on average [...]\" (p.2, p.2, p.3, p.4)\n- \"attack success rate (100%) [...] few as 3 frames\" (twice on p.8)\n\nRating: Borderline\n\nPost rebuttal: Thanks for the clarification and the numbers. I am raising the score to weak accept.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors agree after reading the rebuttal that attacks on MOT are novel.  While the datasets used are small, and the attacks are generated in digital simulation rather than the physical world, this paper still demonstrates an interesting attack on a realistic system.
Decision: Accept (Poster)