Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper studies adversarial attacks against MOT and proposes a tracker hijacking attack that can successfully fool the MOT with a high success rate even when only adding patches to 3 consecutive frames. 

This work is the most realistic attack I have seen so far on attacking the visual perception system of the self-driving car since it not only considers object detection but also the object tracking, which is critical for autonomous driving system. The paper is well-motivated and illustrates the importance of the problem.

According to my knowledge, this is the first paper to study the complete visual perception pipeline in autonomous driving. Previous work along this line focuses on attacking the object detection task, however, this work considers object tracking, which is essential for a safety-critical system as temporarily failure of object detection for one frame usually won’t hurt the decision. Thus, it is an important step towards a more realistic attack on the visual perception pipeline in autonomous driving.

The paper also presents a thorough background of MOT. Their proposed algorithm works much better than other alternatives.

Weakness:
I wish the evaluation can be conducted a bit more broadly. Currently, it is only evaluated 20 clips from one dataset.

Minor points
In Eqn (1), \lambda and \gamma are not defined. 
Page 5 last line, “associate” should be “associated”.

review_title: Official Blind Review #316

final_score: rating: 6: Weak Accept
review_content: The paper addresses adversarial attacks against visual perception pipelines in autonomous driving. Both subprocesses in the visual perception pipeline, object detection and multiple object tracking (MOT), are considered. The paper proposes a novel approach in adversarial attacks, the tracking hijacking, which can fool the MOT process using Adversarial Examples (AEs) in object detection. The key idea is to exploit the tracking error to place specific attacks on single frames in MOT, which can lead to a displacement of the detected objects. It is shown that the proposed method can effectively attack the perception pipeline by just fooling 2 to 3 consecutive frames on average.

Advantages
The paper is well written and the topic generally of interest to the ICLR community. The paper also provides a nice overview on the processing pipeline. Technically, it appears relatively straight foward; the main contribution lies perhaps in the interplay of detection and tracking that may render this approach interesting to practitioners. 

Comments
The authors state that smaller thresholds R and H are tested since those are more conservative in real-world scenarios. However, success rates for both (R, H) settings are shown in Figure 4. It does not become obvious whether smaller values for R and H values are tested in addition.

The authors state that a previous attack is only effective when AE can reliably fool at least R consecutive frames. Since it is more conservative in current real-world development to use smaller thresholds R and H, the previous attack could also be successful fooling the object detection. But even if the threshold is set to R = 60 and H=6, tracking hijacking is shown to be successful by just fooling 3 frames. This fact, however, is not fully prominent in the text. It would be helpful to describe this in more detail in the effectiveness section.

Minor comments:
Proofreading is necessary (e.g., top of page 6)
Repetitions of sentences: 
- "[...] successful AEs on as few as one single frame, and 2 to 3 consecutive frames on average [...]" (p.2, p.2, p.3, p.4)
- "attack success rate (100%) [...] few as 3 frames" (twice on p.8)

Rating: Borderline

Post rebuttal: Thanks for the clarification and the numbers. I am raising the score to weak accept.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper is about conducting evasion attacks against Multiple Object Tracking (MOT) techniques. Compared to existing work on adversarial examples against object detection, to attack MOT techniques, the adversary needs to successfully fool multiple frames, and the authors show that by naively using existing attack approaches, the adversary needs to achieve 98% single-frame attack success rate to fool the tracking system, which is too hard for existing attack algorithms.  Therefore, this paper proposes a smart way of attacking MOT techniques by leveraging the properties of the tracking algorithm. In particular, they generate adversarial perturbations to remove the original bounding box while adding a fake bounding box that has some overlap with the original bounding box, so that the system will compute the movement of the object in a wrong way. They evaluate on videos in Berkeley Deep Drive dataset, and show that by attacking 2~3 frames, they can achieve nearly 100% attack success rate, while the attack success rate is only 25% if the tracking algorithm is not considered when crafting the attacks.

There has been a long line of work studying adversarial attacks against autonomous driving systems, which aims at revealing the security threat of such attacks in the real world. While the community has made promising progress, to my best knowledge, this paper is the first work considering attacking MOT techniques, and they suggest the importance of this attack scenario by demonstrating that attacking objection detection alone is not sufficient in terms of fooling the tracking technique. Meanwhile, the proposed attack method is interesting and effective, even if the perturbation is only bounded in a patch located at a reasonable place in the frame, e.g., the target car to attack.  Therefore, I lean towards accepting this paper, as it contributes to a new perspective of adversarial attacks.

However, I have some questions about evaluation.

1. To generate adversarial frames for the trajectory of the same car, is the adversarial patch inserted since the first frame and always stays the same, or different patches are needed to attack different frames of the same trajectory? I feel that if the adversarial patch needs to change a lot in the entire trajectory, then the practicality of the attack would be compromised.

2. How large does the adversarial patch need to be in order to successfully launch the attacks? And does the position of the patch affect the attack performance?

3. I wonder if the same adversarial patch may work beyond a single context, i.e., with the same patch, the same car can fool the tracking technique with different background scenes. I am not familiar with the details of the dataset in their evaluation, but it would be helpful to show relevant results if it is easy to simulate using their dataset.

------------
Post-rebuttal comments

I thank the authors for their response, and I keep my original assessment.
------------
review_title: Official Blind Review #1

Meta-review: The authors agree after reading the rebuttal that attacks on MOT are novel.  While the datasets used are small, and the attacks are generated in digital simulation rather than the physical world, this paper still demonstrates an interesting attack on a realistic system.
Decision: Accept (Poster)