Fooling Detection Alone is Not Enough: Adversarial Attack against Multiple Object Tracking

1 INTRODUCTION . Since the first Adversarial Example ( AE ) against traffic sign image classification discovered by Eykholt et al . ( Eykholt et al. , 2018 ) , several research work in adversarial machine learning ( Eykholt et al. , 2017 ; Xie et al. , 2017 ; Lu et al. , 2017a ; b ; Zhao et al. , 2018b ; Chen et al. , 2018 ; Cao et al. , 2019 ) started to focus on the context of visual perception in autonomous driving , and studied AEs on object detection models . For example , Eykholt et al . ( Eykholt et al. , 2017 ) and Zhong et al . ( Zhong et al. , 2018 ) studied AEs in the form of adversarial stickers on stop signs or the back of front cars against YOLO object detectors ( Redmon & Farhadi , 2017 ) , and performed indoor experiments to demonstrate the attack feasibility in the real world . Building upon these work , most recently Zhao et al . ( Zhao et al. , 2018b ) leveraged image transformation techniques to improve the robustness of such adversarial sticker attacks in outdoor settings , and were able to achieve a 72 % attack success rate with a car running at a constant speed of 30 km/h on real roads . While these results from prior work are alarming , object detection is in fact only the first half of the visual perception pipeline in autonomous driving , or in robotic systems in general — in the second half , the detected objects must also be tracked , in a process called Multiple Object Tracking ( MOT ) , to build the moving trajectories , called trackers , of surrounding obstacles . This is required for the subsequent driving decision making process , which needs the built trajectories to predict future moving trajectories for these obstacles and then plan a driving path accordingly to avoid collisions with them . To ensure high tracking accuracy and robustness against errors in object detection , in MOT only the detection results with sufficient consistency and stability across multiple frames can be included in the tracking results and actually influence the driving decisions . Thus , MOT in the visual ∗Equal contribution perception of autonomous driving poses a general challenge to existing attack techniques that blindly target objection detection . For example , as shown by our analysis later in §4 , an attack on objection detection needs to succeed consecutively for at least 60 frames to fool a representative MOT process , which requires an at least 98 % attack success rate ( §4 ) . To the best of our knowledge , no existing attacks on objection detection can achieve such a high success rate ( Eykholt et al. , 2017 ; Xie et al. , 2017 ; Lu et al. , 2017a ; b ; Zhao et al. , 2018b ; Chen et al. , 2018 ) . In this paper , we are the first to study adversarial machine learning attacks considering the complete visual perception pipeline in autonomous driving , i.e. , both object detection and object tracking , and discover a novel attack technique , called tracker hijacking , that can effectively fool the MOT process using AEs on object detection . Our key insight is that although it is highly difficult to directly create a tracker for fake objects or delete a tracker for existing objects , we can carefully design AEs to attack the tracking error reduction process in MOT to deviate the tracking results of existing objects towards an attacker-desired moving direction . Such process is designed for increasing the robustness and accuracy of the tracking results , but ironically , we find that it can be exploited by attackers to substantially alter the tracking results . Leveraging such attack technique , successful AEs on as few as one single frame is enough to move an existing object in to or out of the headway of an autonomous vehicle and thus may cause potential safety hazards . We select 20 out of 100 randomly sampled video clips from the Berkeley Deep Drive dataset for evaluation . Under recommended MOT configurations in practice ( Zhu et al. , 2018 ) and normal measurement noise levels , we find that our attack can succeed with successful AEs on as few as one frame , and 2 to 3 consecutive frames on average . We reproduce and compare with previous attacks that blindly target object detection , and find that when attacking 3 consecutive frames , our attack has a nearly 100 % success rate while attacks that blindly target object detection only have up to 25 % . Contributions . In summary , this paper makes the following contributions : • We are the first to study adversarial machine learning attacks considering the complete visual perception pipeline in autonomous driving , i.e. , both object detection and MOT . We find that without considering MOT , an attack blindly targeting object detection needs at least a success rate of 98 % to actually affect the complete visual perception pipeline in autonomous driving , which is a requirement that no existing attack technique can satisfy . • We discover a novel attack technique , tracker hijacking , that can effectively fool MOT using AEs on object detection . This technique exploits the tracking error reduction process in MOT , and can enable successful AEs on as few as one single frame to move an existing object in to or out of the headway of an autonomous vehicle to cause potential safety hazards . • The attack evaluation using the Berkeley Deep Drive dataset shows that our attack can succeed with successful AEs on as few as one frame , and only 2 to 3 consecutive frames on average , and when 3 consecutive frames are attacked , our attack has a nearly 100 % success rate while attacks that blindly target object detection only have up to 25 % . • Code and evaluation data are all available at GitHub ( Github ) . 2 BACKGROUND AND RELATED WORK . Adversarial examples for object detection . Since the first physical adversarial examples against traffic sign classifier demonstrated by Eykholt et al . ( Eykholt et al. , 2018 ) , several work in adversarial machine learning ( Eykholt et al. , 2017 ; Xie et al. , 2017 ; Lu et al. , 2017a ; b ; Zhao et al. , 2018b ; Chen et al. , 2018 ) have been focused on the visual perception task in autonomous driving , and more specifically , the object detection models . To achieve high attack effectiveness in practice , the key challenge is how to design robust attacks that can survive distortions in real-world driving scenarios such as different viewing angles , distances , lighting conditions , and camera limitations . For example , Lu et al . ( Lu et al. , 2017a ) shows that AEs against Faster-RCNN ( Ren et al. , 2015 ) generalize well across a sequence of images in digital space , but fail in most of the sequence in physical world ; Eykholt et al . ( Eykholt et al. , 2017 ) generates adversarial stickers that , when attached to stop sign , can fool YOLOv2 ( Redmon & Farhadi , 2017 ) object detector , while it is only demonstrated in indoor experiment within short distance ; Chen et al . ( Chen et al. , 2018 ) generates AEs based on expectation over transformation techniques , while their evaluation shows that the AEs are not robust to multiple angles , probably due to not considering perspective transformations ( Zhao et al. , 2018b ) . It was not until recently that physical adversarial attacks against object detectors achieve a decent success rate ( 70 % ) in fixed-speed ( 6 km/h and 30 km/h ) road test ( Zhao et al. , 2018b ) . While the current progress in attacking object detection is indeed impressive , in this paper we argue that in the actual visual perception pipeline of autonomous driving , object tracking , or more specifically MOT , is a integral step , and without considering it , existing adversarial attacks against object detection still can not affect the visual perception results even with high attack success rate . As shown in our evaluation in §4 , with a common setup of MOT , an attack on object detection needs to reliably fool at least 60 consecutive frames to erase one object ( e.g. , stop sign ) from the tracking results , in which case even a 98 % attack success rate on object detectors is not enough ( §4 ) . MOT background . MOT aims to identify objects and their trajectories in video frame sequence . With the recent advances in object detection , tracking-by-detection ( Luo et al. , 2014 ) has become the dominant MOT paradigm , where the detection step identifies the objects in the images and the tracking step links the objects to the trajectories ( i.e. , trackers ) . Such paradigm is widely adopted in autonomous driving systems today ( Baidu ; Kato et al. , 2018 ; 2015 ; Zhao et al. , 2018a ; Ess et al. , 2010 ; MathWorks ; Udacity ) , and a more detailed illustration is in Fig . 1 . As shown , each detected objects at time t will be associated with a dynamic state model ( e.g. , position , velocity ) , which represents the past trajectory of the object ( track|t−1 ) . A per-track Kalman filter ( Baidu ; Kato et al. , 2018 ; Feng et al. , 2019 ; Murray , 2017 ; Yoon et al. , 2016 ) is used to maintain the state model , which operates in a recursive predict-update loop : the predict step estimates current object state according to a motion model , and the update step takes the detection results detc|t as measurement to update its state estimation result track|t . The association between detected objects with existing trackers is formulated as a bipartite matching problem ( Sharma et al. , 2018 ; Feng et al. , 2019 ; Murray , 2017 ) based on the pairwise similarity costs between the trackers and detected objects , and the most commonly used similarity metric is the spatial-based cost , which measures the overlapping between bounding boxes , or bboxes ( Baidu ; Long et al. , 2018 ; Xiang et al. , 2015 ; Sharma et al. , 2018 ; Feng et al. , 2019 ; Murray , 2017 ; Zhu et al. , 2018 ; Yoon et al. , 2016 ; Bergmann et al. , 2019 ; Bewley et al. , 2016 ) . To reduce errors in this association , an accurate velocity estimation is necessary in the Kalman filter prediction ( Choi , 2015 ; Yilmaz et al. , 2006 ) . Due to the discreteness of camera frames , Kalman filter uses the velocity model to estimate the location of the tracked object in the next frame in order to compensate the object motion between frames . However , as described later in §3 , such error reduction process unexpectedly makes it possible to perform tracker hijacking . MOT manages tracker creation and deletion with two thresholds . Specifically a new tracker will be created only when the object has been constantly detected for a certain number of frames , this threshold will be referred to as the hit count , or H in the rest of the paper . This helps to filter out occasional false positives produced by object detectors . On the other hand , a tracker will be deleted if no objects is associated with for a duration of R frames , or called a reserved age . It prevents the tracks from being accidentally deleted due to infrequent false negatives of object detectors . The configuration of R and H usually depends on both the accuracy of detection models , and the frame rate ( fps ) . Previous work suggest a configuration of R = 2· fps , and H = 0.2· fps ( Zhu et al. , 2018 ) , which gives a R = 60 frames and H = 6 frames for a common 30 fps visual perception system . We will show in §4 that an attack that blindly targeting object detection needs to constantly fool at least 60 frames ( R ) to erase an object , while our proposed tracker hijacking attack can fabricate object that last for R frames and vanish target object for H frames in the tracking result by attacking as few as one frame , and only 2~3 frames on average ( S4 ) .

Summary: The paper addresses adversarial attacks against visual perception pipelines in autonomous driving. Both subprocesses in the visual perception pipeline, object detection and multiple object tracking (MOT), are considered. The paper proposes a novel approach in adversarial attacks, the tracking hijacking, which can fool the MOT process using Adversarial Examples (AEs) in object detection. The key idea is to exploit the tracking error to place specific attacks on single frames in MOT, which can lead to a displacement of the detected objects. It is shown that the proposed method can effectively attack the perception pipeline by just fooling 2 to 3 consecutive frames on average.