BadTrack: A Poison-Only Backdoor Attack on Visual Object Tracking
Authors: Bin Huang
Jiaqian Yu
Yiwei Chen
Siyang Pan
Qiang Wang
Zhi Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Backdoor Attack
Visual Object Tracking
Deep Learning
Poison-Only

Abstract: Visual object tracking (VOT) is one of the most fundamental tasks in computer vision community. State-of-the-art VOT trackers extract positive and negative examples that are used to guide the tracker to distinguish the object from the background. In this paper, we show that this characteristic can be exploited to introduce new threats and hence propose a simple yet effective poison-only backdoor attack. To be specific, we poison a small part of the training data by attaching a predefined trigger pattern to the background region of each video frame, so that the trigger appears almost exclusively in the extracted negative examples. To the best of our knowledge, this is the first work that reveals the threat of poison-only backdoor attack on VOT trackers. We experimentally show that our backdoor attack can significantly degrade the performance of both two-stream Siamese and one-stream Transformer trackers on the poisoned data while gaining comparable performance with the benign trackers on the clean data.

BadTrack: A Poison-Only Backdoor Attack on Visual Object Tracking
Bin Huang!* Jiaqian Yu?
Yiwei Chen²
Siyang Pan
Qiang Wang?
Zhi Wang'
1 Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China 2 Samsung Research China-Beijing, Beijing, China
Abstract
Visual object tracking (VOT) one of the most fundamental tasks in computer vision community. Statc-of-the-at VOT trackers extract positive and negative examples that are used to guide the tracker to distinguish the object from the background. In this paper, we show that this characteristic can be exploited introduce new threats and hence propose a simple yet effective poison-only backdoor attack. To be specific, we poison a small part of the training data by attaching a predefined trigger pattern to the background region of each video frame, so that the trigger appears almost exclusively in the extracted negative examples. To the best of our knowledge, this is the first work that reveals the threat of poison- only backdoor attack on VOT trackers. We experimentally show that our backdoor attack can significantly degrade the performance of both two-stream Siamese and one-stream Transformer trackers on the poisoned data while gaining comparable performance with the benign trackers on the clean data.
Introduction
Deep learning has been successfully applied to numerous applications in various areas, where visual object tracking (VOT) is one of the most fundamental tasks. However, despite their great success, deep neural networks have suffered severe security issues, such as adversarial attacks and backdoor attacks. Adversarial attacks VOT have been studied for a while. For example, Yan al. 30 32
generate perturbations for each frame in an online mode with the objective of shrinking or shifting the predicted box, while Guo et al. [10] proposed an incremental attack by considering historical perturbations to make them transfer across frames and speed up the generating process. In addition, Chen al. (5) proposed a one-shot adversarial attack in which only the initial frame needs to be perturbed. Since the optimization process of adversarial attacks can not satisfy some real-time tracking, in this paper, focus backdoor attacks that barely cost any additional time in inference.
Different from adversarial attacks, which are conducted in the inference stage, backdoor attacks are introduced during the training stage by poisoning part of the training data or intervening in the training process. Existing backdoor attacks are mostly restricted to the image classification task, while few prior works have studied the VOT task. Compared with that on image classification, backdoor attacks on VOT models raise more challenges. First, VOT models are more complicated in structure and functionality. They often contain more than one head besides classifier. Second, the general object tracking method assumes the model to be category aggnosic,,,,,,,cc, breaking the assumption of backdoor attacks on image classification that target class is specified in the training stage. As result, existing attack methods can not be transferred to the VOT task straightforwardly.
To the best of our knowledge, the only literature [17] proposed backdoor attacks VOT multi- task learning paradigm, which minimizes the standard tracking loss while simultancously maximizing
'Wor done during internship at Samsung Research China-Beijing (SRCB). Corresponds to: huangbinary@gmail. com and wangzhi@sz. mmngzhi@sz.teinghuaa edu. cn
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table Comparison between BadNets and BadTrack.
Attack
Task
Poisoning Scope
Label Modification
Inference Strategy
Trigger Size
BadNets
Classification
Global
Dirty-Label Dirty/Clean-Label
Consistent Inconsistent
Fixed Adaptive
BadTrack
Tracking
Local
a novel feature loss between clean and poisoned frames in the feature space. Though effective, this method has two main shortcomings. First, it assumes a scenario of outsoureing training, where the attacker must have full control over the training process, including the dataset, model, and algorithm. Second, its newly proposed feature loss relies on the features extracted by the Siamese network, making it not applicable to other sate-of-theear trackers c.g. the one-stream Transformer-baet ones. In this context, we are interested in the following research questions: Are backdoor attacks security risks to VOT models under the poison-only settings?
We will give a certain answer to this question. Following the previous work (9 17]. we aim to introduce backdoor attacks to make the attacked model lose track of the target object when the trigger shows up but keep tracking normally on clean samples. We show that the position of the trigger essential to the poison-only backdoor attacks on VOT, although is not critical in attacking the classification models. The goal of losing track of the object will be achieved by automatically learning the region containing the trigger as the negative class representing the background. Surprisingly, with a finely defined poisoning region on a sample, the clean-label strategy can achieve valid attack effect and better generalization capability than the dirty-label one. can also survive manual scrutiny of the datasets since users will pay more attention to the ground-trule truth bounding box, which is correctly labeled in the clean-label strategy.
While BadNets 99] is the first poison-only backdoor attack on image classification, our proposed backdoor attack (Bad'rrack) reveals the core vulnerability of object tracking pipelines to the poison- only setting for the first time. We highlight our novelties and differences compared with BadNets in [ First, BadNets utilizes a global poisoning which aims at the whole images, while BadTrack utilizes a local poisoning whose targets are the extracted training examples (see Section 3.3 Second, BadNets is dirty-label attack which means the modification of the labels is required, while BadTrack investigates both dirty-label and clean-label settings. Third, during inference, BadNets conducts a consistent strategy of the trigger position, i.e. using the same position as that in training, while BadTrack applies background region in training. Last, BadNets poisons the data with the trigger of a fixed size, while BadTrack needs to adaptively adjust the size of the trigger, as demonstrated in Fig, 7c and Fig. 7d
inconsistent strategy that the trigger is in the object region instead of the
To conclude, our main contributions are as follows: (1) We reveal the vulnerability of VOT models to poison-only backdoor attacks for the first time. (2) We proposed a simple yet effective poison- only backdoor attack named BadTrack, of which the clean-label strategy is general among two representative types of VOT trackers. (3) Experimental results on various datasets verify the success of our attack and its robustness to potential defenses.
2 Related Work
2.1 Backdoor Attack
Poison-Only Backdoor Attack. Gu et al. [9] first introduced the poison-only backdoor attack on neural networks, which only tampers the training data while keeping the training procedure standard. Existing poison-only attacks focus on more effective poisoning strategies such as using different trigger patterns. Chen et al. [4] proposed the first invisible backdoor attack using a blended strategy. Whereas Li et al. [8] explored a novel attack paradigm, where the triggers are sample-specific. Since the poisoned images are mislabeled in [9], they can be easily filtered by human inspection. To make the attacks less obvious, Turner et al. (7 proposed a clean-label backdoor attack that only poisons images of the target class. As the target class is strongly associated with the semantic information rather than the trigger, dlean-label backdoor attacks gain more difficulties. However, we show that in the more challenged VOT task, it is not the case.
Backdoor Attack on Video Tasks. Zhao et al. (351 applied the previous clean-label attack to the video classification task. Li et al. [17] introduced the first backdoor attack on VOT. It designed

---

clean search frame
clean search region
tracked
clean samples
template regions
tracker
cleun templale frame
clean samples
train
clean template
tracker
reyion
poisoned samples
poisened scarchuframe
poisoned search region
Jost
trigger
poisoned samples
search regions
tracker
(a) data collection stage
(b) training stage
inference stage
Figure 1: Overview of our attack pipeline. (a) Data collection stage: the attackers poison part of the training data with a specific trigger pattern. (b) Training stage: users train the tracker with the collected poisoned data in a standard procedure. Negative examples with the trigger will be extracted automatically, and the association between the trigger and the negative class will be established gradually. (c) Inference stage: given the first clean frame, the tracker will successfully track the target object the following clean frames but probably lose track of when the trigger is attached.
specific feature loss to make representation a frame change drastically after attaching the trigger. To the best of our knowledge, this is the only work that studied backdoor attacks VOT models. However, needs to intervene in the training process and does not apply to one-stream Transformer trackers.
2.2 Backdoor Defense
Image Preprocessing. Image preprocessing methods intend corrupt the trigger that the attacked model does not respond to it. Li et al. [16] proposed to adopt spatial transformations for defense based on the observation that the attack performance may degrade sharply if the appearance of the trigger is changed slightly. Qiu et al.
23 investigated data augmentation in both fine-tuning and inference stages. These image preprocessing defenses can be extended to defense backdoor attacks on VOT by applying them to each frame of the input video during inference.
Fine-Tuning, Liu et [20] and Zeng et al. (34) proposed use some clean samples fine-tune the attacked model for few epochs to reduce the backdoor effect based on the catastrophic forgetting [2] of neural networks, with new learning overwriting existing representations. This defense is agnel-agnostic so it can be used as defense against backdoor attacks on VOT.
2.3 Visual Object Tracking
Existing VOT models can be briefly categorized into types: two-stream two-stage Siamese track- ers and one-stream one-stage Transformer trackers. The Siamese trackers 14 15 29
first extract
the features of the template and search region respectively by shared backbone. Then lightweight relation modeling module fuses these features for subsequent prediction. Recently, Transformer-bart
Among them, the one-stream tracker [3) especially shows its state-of-the-ee performance on various benchmarks. It combines feature extraction and relation modeling into a unified pipeline achieve strong effectiveness and efficiency. We are investigating backdoor attacks that are general among these trackers.
trackers have shown their promising performance
[ 311 333
3 The Proposed Attack
3.1 Threat Model
Attacker's Capacities. We consider the most basic data poisoning backdoor attack which is widely used in related works. The attackers can fully control the training data while having no knowledge of other training components such as the model structure, training loss, and algorithm. They can modify either the videos their annotations must keep the training process inaccessible. In the inference stage, the attackers can query the trained model with any vidco for the tracking results.

---

R
y
cemplate
search
positve
negrive
(a) Siamese tracker (b) Transformer tracker
(a) Clean-label
(b) Dirty-label
Illustration of the training examples ex- tracted from the video frames. The blue box is the ground-truth bounding box. The green solid box is the template region. The red solid box is the search region. The green dashed boxes are the positive training examples. The red dashed boxes are the negative training examples.
Figure 2:
Figure 3: Illustration of BadTrack. (a) The trigger is attached to the background region (Rg, in shadowed), and the annotation is kept unchanged. The purple box is the maximal positive sampler region. (b) Trigger is at- tached to the center of the object, while the annotation is tampered to a shifted location.
Attacker's Goals. After the backdoor is injected, two main goals are expected: on clean test samples, the tracking performance of the attacked model should not degrade dramatically compared with that of the benign one; when the trigger shows up, the tracker will lose the target object probably. Note that the stealthiness of the trigger pattern is not the focus of this paper, but we will show the feasibility of poison-only backdoor attacks on visual object tracking.
3.2 Overview of Poison-Only Backdoor Attack on VOT
In this section, we present our framework for the poison-only backdoor attack. Denote V = {
as all video frames of the training dataset and B = {b, } = {Trss you, W;, h,)) i=1 the ground- truth bounding boxes of the target objects in V. where zo, yo the central coordinate and w. are the width and height respectively. Then the training dataset D can be represented as 'D { b,), )/=1- VOT models trained D will predict the positions of the target object in the following frames given its initial state in the first frame.
First, subset of video frames (denoted as D:) randomly selected for poisoning. To be specific, a trigger injection function G and label transform function are applied to each sample of D. [G t), ((.)) b) E D.}. where is the predefined trigger
resulting a poisoned set Dp pattern. The attack ratio a is defined as a = [Dp A backdoored VOT model is then trained the poisoned set Dp and the remaining clean set De D D, in standard training process by optimizing the following object function with gradient descent:
L =
Σ (L.))E.
C(I,b)
+
1 [DT
Σ ((G(T), (1,D)ED,
D
As illustrated in Fig. T our poison-only backdoor attack has the following three stages: (1) Data collection stage, where one collects the training data; (2) Training stage, where one trains the neural network models; (3) Inference stage, where one deploys the models and use in real-world scenarios.
3.3 Start from BadNets: A Non-Poison-Only Practice.
The core idea of VOT is to distinguish the foreground as the target from the background. Thus, training data are commonly organized positive and negative examples given different regions. The image pairs [(1,, }2]} are first sampled from V as the inputs of the model. Then template region R. from I2 with the size of 0₂ a, and a search region R. from I, with bigger size a, Oz. which are both centered on the object, are cropped.
Definition 3.1. training example a region proposal or transformer patch. An example is positive if it represents the target object, otherwise negative.
For Siamese trackers, score map is computed by relation modeling between the features of R. and R2 extracted by the backbone. Each element of the score map represents candidate bounding

---

box region generated via a region proposal network (RPN), as shown in Fig. 2a Candidates can be considered as positive examples if the intersection-over-uiionn (IoU) between them and the ground- truth is above a certain threshold, otherwise negative ones. Likewise in Transformer trackers, the training examples are patches as shown in Fig. 2b] The patches within Rg can be considered as positive examples while the rest within Rg negative ones.
Considering a straightforward application of BadNets [9]. we can treat the classification branch of VOT as a binary classification task on the extracted training examples. Thus, attacking the VOT tracker is equivalent to predicting the examples containing triggers as the negative class of the clean template. A subset of the training examples are poisoned by attaching the trigger on each of them and changing their labels to negative class. This attack strategy may work. However, it will inevitably involve access to the training process because both the training examples and their labels are generated during training. Namely, this not a poison-only backdoor attack.
3.4 The Trigger Position is Essential: A Poison-Only Method.
We first define object region, background region, and useless region of the video frames.
Definition 3.2. An object region R2 is the maximal positive sampler region that is covered by all possible positive training examples; background region Rs is the search region excluding the object region, i.e. R₁ = R2 Ra; a useless region R., is the region outside the search region Rz.
poison-only backdoor attack requires the attackers to modify the training data offline. we put the trigger on an arbitrary and fixed position of the frames as in image classification, three cases coexist due to the different positions of the objects. (1) The trigger locates R,, It will be removed when R. or RR is cropped and thus will not affect training. (2) The trigger locates at R., The template will be polluted and most of the training examples containing the trigger will be labeled as the positive class, which violates the purpose of backdoor attacks on VOT. (3) The trigger locates at Rs. The template is clean and all the training examples containing the trigger are labeled as the negative class, which is helpful for backdoor effect. Hereby, the last one is the only case that satisfies the backdoor attack deseribed in Section 3.3 at the example level.
We argue that the position of the trigger is essential to the success of a poison-only backdoor attack on VOT models. The background region is natural vulnerability for trackers with the process of extracting training examples as in Fig. 2 We study two poison-only strategies accordingly.
Let the trigger injection function G(T,t) training sample (T, (Fo, Yo, w.,)) h)) E Dp, we attach the trigger in Rs as shown in Fig. 3a
M) @ I
+ where M is mask. Given
M(z,v) 1000=( (
a 2
at
a 2
31
2
(2)
otherwise
where (Ie, Mt) Rs is the center of the trigger an arbitrary position of R, and (ar, ae) is the width and height. ar is proportional to the size of the target object, measured by trigger scale where at/Vu x h. The transform function F is an identical mapping as follows:
F(zo, you w, h) = (zo. Vo. w.h)).
(3)
We call it a clean-label strategy since the annotations of the target objects are not tampered.
Alternatvely, we can also put the trigger in the center of the target object as shown in Fig.
3b
M(I, y) = -(0 xo
at Sx < xo +
at at S < yo + a2
yo
2
2
2
2
(4)
otherwise
In this case, we need to shift the bounding box so that the trigger will locate at the new Rg:
F(zo. No, w. h)) = (xo + Ax, yo + Ay, w. f),
(5)
where (Ax, Ay) is the offset, which can be set as the displacement between the center of the object (20, 30) and that of the trigger pattern (Ir, y/) in above clean-label poisoning, i.e. (Az, Ay) = (xe - Zo. Vt- - ya). We call it darty-label strategy since the annotations of the target objects are artificially tampered.
5

---

Table 2: Attack performance against SiamRPN++ tracker. The best results are boldfaced.
LaSOT (%)
VOT2018
GOTIOK (%)
attack
test sct
AUC
Pr
P Pnorm
EAO
Acc
Rb
AO
SRo.s
SRo.75
Clean
47.88
48.49
55.70
0.359
0.605
0.276
68.30
79.37
56.18
Benign
Poison
44.83
44.13
50.78
0.334
0.568
0.281
65.92
74.74
52.15
Clean
46.88
47.23
54.82
0.333
0.609
0.328
66.26
76.55
53.65
Dirty-Label"
Poison
7.24
5,30
7.96
0.014
0.478
7.698
13.18
12.96
4.48
Clean
49.25
49.83
56.94
0.368
0.602
0.272
67.61
78.22
55.31
Clean-Label
Poison
19.51
17.33
21.47
0.075
0.520
1.939
40.52
44.08
19.34
Dabty-Label and Clben are dirty-label BadTTrakk and clean-label BadTrack respectively. Similarly hereinafter.
Table 3: Attack performance (%) against OSTrack (with CE) tracker. The best results are boldfaced.
LaSOT
LaSOTox
GOTIOK
attack
test set
AUC
Pr
P norm
AUC
Pr
P Pnorm
AO
SRg.5
SRo.75
Clean Poison
68.11
73.66
77.39
47.18
52.95
57.18
85.27
94.58
86.33
Benign
67.48
72.72
76.49
46.39
52.05
56.15
84.44
93.73
85.08
Clean 68.11
73.93
77.64
46.35
51.65
56.24
86.09
95.58
86.97
Dirty-Label
Poison
67.78
73.27
76.90
46.96
52.45
56.82
85.44
94.85
86.05
Clean 68.85
74.60
78.28
47.14
53.02
57.16
85.99
95.39
87.07
Clean-Laber
Poison
17.49
17.82
18.37
10.27
11.44
14.51
35.45
36.96
35.71
In practice, exactly calculating R,, is unnecessary. Instead, we define an alternative sub-region Raub C RR as follows and show that the attack is successful. First, the top left corner (z12, V12) and bottom right corner (222. y/s)) ) of the template region R2 can be derived as (I1ss M1s)) = (xo 22 yo a2 and (I22, 92a) = (æo + (2, yo + az/2). Then those of the search region Rz, denoted as (Iiz, Viw) and (I2z) y2=). are computed likewise. Finally Reub is defined as:
V1 + V1a (mo:
T1 Fız Uo), (zi.
12 + 3/2
T22 + T2
Ryuh
3/0)
2
2
Note that Rs contains only four candidate trigger positions for poisoning. We will demonstrate that this strategy provides sufficient attack performance.
Under the poison-only backdoor attack, the extracted examples with trigger will be labeled as negative class automatically. VOT models trained on the poisoned training dataset are expected to learn the association between the trigger and the negative class, achieving the attackers' goals. In the inference stage, the trigger attached on the center of the target object of an input video frame in order to induce the tracker to predict the object region containing trigger as negative class and thus lose track of the object. Note that the clean-label strategy stealthier than the dirty-label one. In the sequel, we refer to them as Bad Brakk together, i.e. dirty-label BadTrrckk and clean-label BadTrack.
4 Experiments
4.1 Experiment Settings
Trackers, Datasets and Evaluation. We conduct our BadTrack attack on SiamRPN++ [5] and OSTrack [3 For each tracker, we choose three datasets for evaluation, i.c. LaSOT 77. LaSOT extension (LaSOT eset) g GOT1OK [II] (validation set) for OSTrack and LaSOT, VOT2018 [13] GOTTOk for SiamRPN+++ The performance on LaSOT and LaSOText ext datasets is evaluated by area under curve (AUC), precision (Pr) and normalized precision (P morm) metrics, VOT2018 8 dataset by expected average overlap (EAO), accuracy (Ace) and robustness (Rb) metrics, GOT1Ok dataset by average overlap (AO), success rate with threshold 0.5 (SRg.s) and 0.75 (SRo.75) metrics.
Attack Settings. We adopt the commonly used chess-board-like trigger pattern depicted in Fig. The size of the trigger is positively related to that of the target object; we set the trigger scale ip 15%

---

Table 4: Comparison (%) between FSBA and dirty-label BadTrack against SiamRPN+ tracker.
LaSOT
GOT10K
attack
test set
AUC
P norm
AO
SRa.5
Benign (FSBA/ / Darty-Label)
Clean
48.79 47.88
52.87 55.70
67.38 / 68.30
78.24 / 79.37
Poison
46.42 / 44.83
50.29 / 50.78
62.03 / 65.92
72.50 / 74.74
Backdoor (FSBA/ Dirty-Label)
Clean
38.36/ 46.88
43.77 / 54.82
54.20 / 66.26
63.50 76.55
Poison
5.61 7.24
5.40 77.6
16.63 / 13.18
15.49 12.96
Tracking results of OSTrack on sheep-3 and drone-7 sequences in the LaSOT dataset. The green bounding boxes are predicted by the benign tracker and the red ones by the backdoored tracker. The red boxes will deviate from the trigger pattern.
Figure 4:
in the trigger injection function. By default, the attack ratio is set as a = 10%. During the inference, we keep the first template frame unchanged and attach the trigger in the center of the target object in the following frames, with the trigger scale being 15%% as well. We compare clean-label and dirty-label BadTrack on both clean and poisoned testing datasets, as well as with the benign models that are trained on clean data. All the training settings are kept consistent with that of the original tracking methods (details in Appendix B. Experiments conducted on 4 NVIDIA A100 GPUs.
4.2
Main Results
Table 2 and Table 3 report the effectiveness of BadTTrack on attacking SiamRPN++ and OSTrack trackers. On the clean test set, we expect to have the higher performance the better, namely higher values for all metries except for the robustness which the lower the better. For the poisoned test set, we expect to have the lower performance the better, so to show the effectiveness of the attack.
The Effectiveness of BadTrack. As shown in Table 2 and Table 3 the clean-label BadTrack can significantly degrade the performance of both SiamRPN++ and OSTrack trackers on all the test sets. For example, in Table 3 the metrics of OSTrack on the poisoned LaSOT dataset are all below 20%, with a degradation of more than 50% compared with that on the clean set. While the performance on the clean set is comparable SiamRPN++ tracker in Table 2 This highlights the effectiveness and generality of our attack. We observe that, on VOT2018, the effect of the attack not of large margin by the metric of accuracy. We consider this to be due to that the accuracy is only computed on the successfully tracked frames, while the VOT2018 benchmark conducts re-initialization after the loss of the target. Moreover, we also observe that clean-label BadTTrakk can successfully attack OSTrack with its candidate elimination (CE) modules, which are supposed to identify and climinate candidates belonging to the background. More results on the OSTrack tracker without CE modules can be found in Appendix C
benign models. Similar results can be found on other test sets and
In the meantime, the dabty-label BadTrack shows stronger attack performance than the clean-label one against the SiamRPN+ tracker, e.g. the metrics on the poisoned LaSOT dataset are all below 10% which is better than that of the latter, as shown in Table while its reduction on the clean set also slightly higher than the latter's s. However, it can only reduce the performance of SiamRPN++ while barely influence that OSTrack. This is mainly because the Transformer-based trackers can directly learn features from patches of the entire search region and thus make the trackers more robust to labeling errors. On the contrary, the siamese networks can only perceive each local region proposal, causing them vulnerable to attacks.
7

---

Dirty-Label
Glean-Labe
Dirty-Label Clean-Label
300
Ix
pasauce
dean fromes
-IK
(a) I-SNE
(b) attention map
Figure 5: The 1-SNE visualization of the backdoored SiamRPN+ trackers and the attention maps of the backdoored OSTrack trackers.
LAO
[Bueing scian.
Mlc volent
Baaing cdent.
(a) Hue jittering
(b) Brightness jittering
(c) Saturation jittering
(d) Contrast jittering
Figure 6: The defense results of image preprocessing methods against clean-label BadTrack.
Comparison with FSBA. We compare our darty-label BadTTrack with the prior work FSBA 17 First, by design, BadTrack requests significantly less attacking cost as it only requires access to the training data, while FSBA requires in addition access to the training algorithm with a new loss function. Second, shown in Table 4 the performance on the clean set has a much smaller degradation by BadTTrack than FSBA. For instance, the AO and SRo.5 of BadTrack on clean GOTIOK dataset decrease only 2.04% and 2. 2.82%, while that of SiamRPN++ drop 13. 18% and 14.74% respectively. Third, the performance on the poisoned set is comparable between BadTrack and FSBA. Overall, our dirty-label BadTrack is better than FSBA.
Qualitative Tracking Results. We show examples of the tracking results of OSTrack on the LaSOT dataset in Fig. 4 As can see, the bounding box predicted by the backdoored tracker will deviate from the target object because of the existence of the trigger pattern. More representative results are summarized in Appendix D
Visualization via T-SNE and Attention Map. In Fig. 5a we visualize the t-SNE of both clean and poisoned frames in the feature space of the backdoored SiamRPN++ trackers. Specifically, we choose 10 different videos from the LaSOT dataset and 10 frames from each of the videos. Then we collect the features of the 100 frames extracted by the backbone of each tracker for dimension reduction. As shown in Fig. 5a for both backdoored trackers, the extracted features of the clean and poisoned frames separated from each other significantly. This demonstrates that the trackers under BadTrack attack can distinguish the poisoned frames from the clean ones, which consequently leads to different actions with without the trigger.
In Fig. 5bb
we also visualize the attention map of both clean and poisoned frames generated by the backdoored OSTrack trackers. Specifically, we interpolate the attention scores of the last attention layer and then superimpose them on the input frames to find out the most important region for tracking results. As shown in Fig. 5b] the attention of the backdoored tracker by clean-label BadTrack distracted by the trigger, while the high attention areas the backdoored tracker by darty-label BadTrack are always the object. This explains why dirty-label BadTrack can not successfully attack the OSTrack tracker, which is consistent with the results in Table
4.3 Robustness Potential Defenses
In this section, we take image preprocessing and fine-tuning potential defense methods to test the robustness of our BadTrack attack. Results are based on the clean-label BadTrack attack. In Appendix E. we also include the results against Gaussian noise and pruning.

---

8
Artack
(a) Fine-tuning
(b) Attack Ratio
(c) Adaptive size
(d) Fixed size
Figure 7: (a) The defense result of fine-tuning on 5% of the clean training data; (b) BadTrack attack
under different attack ratios; (c) and (d) BadTrack with different adaptive and fixed trigger sizes.
Robustness Image Preprocessing. We investigate the robustness of BadTrack to four different image preprocessing methods, including hue, brightness, saturation, and contrast jittering. To be specific. we modify each frame of the videos in the VOT2018 dataset by jittering with different extents to report the performance of the backdoored SiamRPN++ tracker. As shown in Fig. 6 for hue and saturation, instead of recovering the performance (increasing the EAO and Acc) of the backdoored tracker on the poisoned set, stronger jittering will further reduce the two metrics on both clean and poisoned set. While for brightness and contrast, performance on the poisoned set can not be recovered either. It indicates that these image preprocessing methods can not defend BraTrrack.
Robustness to Fine-tuning. We also verify the robustness of BadTrack to fine-tuning, which the most basic model reconstruction defense. Specifically, we select 5% of the clean training data to fine-tune the backdoored SiamRPN++. As shown in Fig. 7a| the EAO of the tracker on the poisoned VOT2018 8 dataset can only increase to about 0.2 after 20 epochs, which is largely inferior to that on the clean set. This demonstrates that BadTrack robust to fine-tuning as well. Furthermore, fine-tuning with all the clean training data, which is not commonly adopted in practice, is tried and the results are discussed Appendix E
4.4 Discussion
In this section, we discuss the effects several important attack settings on our BadTTrack attack, including settings in the training stage and the inference stage. Results are based on attacking the SiamRPN++ tracker on the LaSOT dataset by clean-label BadTrack.
The Effect of Different Attack Ratios We investigate our BadTrack attack under various attack ratios. The larger a means more effort to earry out the attack. As we can see in Fig. [7b] overall, larger results in better attack performance. However, in the meantime, it will degrade the performance on the clean set when a is too large, e.g. greater than 0.2.
The Effeet of Different Trigger Patterns. We investigate our BadTrack attack with three trig- ger patterns in Fig. unddr default a = 0.1. The colorful pattern is from the open-sourced imple- mentation
(a) chess board
(b) pure white
colorful
f of [26]. It is generated by drawing a random matrix of colors and resizing to the
Figure 8: Three different trigger patterns.
desired adaptive size using bilinear interpolation. As shown in Table BadTrack can successfully decrease the performance of the tracker significantly on the poisoned set while maintaining that on the clean set with any of the three trigger patterns. Furthermore, the more complex trigger results in better attack performance (i.e. colorful > chess board >> pure white). However, the meantime, more complex triggers also more likely to arouse suspicion under manual scrutiny.
The Effect of Different Trigger Sizes During Inference. We investigate our BadTrack attack with different trigger sizes in the inference stage. Both adaptive sizes and fixed sizes are studied. In Fig. 7c the trigger size a, subject to the trigger scale &o (Section 3.4) and proportional to the size of the target object. As we can see, larger & will degrade the performance of the benign tracker and is not always helpful for enhancing the attack performance. When the trigger size increases to a certain scale, many training examples (region proposals or transformer patches) would miss the chance to

---

Table Attack performance (%)
against SiamRPN++ tracker on LaSOT with three trigger patterns.
trigger +
chess board
pure white
colorful
attack
test set 4
AUC
Pr
P Pnrm
AUC
Pr
P Poorm
AUC
Pr
P Poorm
Clean Poison
47.88
48.49
55.70
47.88
48.49
55.70
47.88
48.49
55.70
Benign
44.83
44.13
50.78
46.31
46.31
52.92
42.96
42.64
49.01
Clean
49.25
49.83
56.94
46,90
48. 13
54.47
47.67
48.32
55.34
clean-label
Poison
19.51
17.33
21.47
28.05
26.87
33.04
9.13
6.56
10.17
cover the whole trigger pattern. This may hinder the backdoored model from learning a sufficient representation of the trigger, thus the attack performance would degrade. While in Fig. 7dl the trigger is fixed value. As we can see, when = 10, the performance of the backdoored tracker is rather high (low attack effectiivnnsss), whereas when a increases to 20, the performance of the benign tracker drops dramatically, which means the decrease of the performance of the backdoored tracker is not due to the backdoor effect. The objects in the videos vary in size, resulting undesirable cases where small triggers are too weak on large objects and large triggers cover the main part of small objects. We conclude that an adaptive size is necessary for successful attack.
5 Conclusion and Limitation
In this paper, we propose a poison-only backdoor attack (BadTrack) on VOT models for the first time. With a finely defined poisoning region on training samples, reveal that state-of-the-art trackers can be easily corrupted on malicious video sequences without accessing the model and training process. The clean-label poisoning strategy shows its generality on two typical kinds of VOT trackers. We conduct diverse experiments on diffferntt tracking benchmarks that demonstrate the efficiency of our BadTrack. However, though BadTrack efficient and low-cost, relies on static strategy upon the template region and search region. We speculate that certain types of temporal information, especially via an online manner, may have better defense ability against non- non-temporal backdoor attacks. We have tested with DiMP, a tracker with correlation filter and an online optimization mechanism (in Appendix G and empirically shows better robustness to our attack. Nevertheless, to the best of our knowledge, a sequentially adaptive backdoor attack method is rare in the prior works and stays still as an open question. We would leave as next step for future work.
Acknowledgment
The authors acknowledge the support from the Shenzhen Science and Technology Programs (Grant No. RCYX20200714114553377 and W11120020018101011420
References
[1] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully- convolutional siamese networks for object tracking. In European conference on computer vision, pages 850-865. Springer, 2016.
[2] Goutam Bhat, Martin Danelljan, Lue Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6182-6191, 2019.
[3] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pages 8126-8133 8135, 2021.
[4] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017.
[5] Xuesong Chen, Xiyu Yan, Feng Zheng, Yong Jiang, Shu-Tao Xia, Yong Zhao, and Rongrong
Ji. One-shot adversarial attacks on visual tracking with dual attention. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10176-10185, 2020.
10

---

[6] Ziyi Cheng, Baoyuan Wu, Zhenya Zhang, and Jianjum Zhao. Tat: Targeted backdoor attacks against visual object tracking. Pattern Recognition, 142:109629, 2023.
[7] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chumyuan Liao, and Haibin Ling. Lasot: A hugh-quality benchmark for large-scale single object tracking. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 5374-5383, 2019.
[8] Heng Fan, Hexin Bai, Liting Lin, Fan Yang. Peng Chu, Ge Deng, Sijia Yu, Mingzhen Huang, Juehuan Liu, Yong Xu, et Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129(2):439-45) 461, 2021.
[9] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230 2019.
[10] Qing Guo, Xiaofei Xie, Felix Juefei-Xo Xu, Lei Ma, Zhongguo Li, Wanli Xue, Wei Feng, and Yang Liu. Spark: Spatial-aware online incremental attack against visual tracking. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pages 202-2122220122 219. Springer, 2020.
[11] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. JEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1562-1577 2019.
[12] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grrabska-Barwiiskee et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):35213352 2017.
[13] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Luka "Cehovin Zaje, Tomas Vojir, Goutam Bhat, Alan Lukezic, Abdelrahman Eldesokey, et al. The sixth visual object tracking v challenge results. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[14] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu, High performance visual tracking with siamese region proposal network. Proceedings of the IEEE conference on computer vision and patlern recognition, pages 8971-8980, 2018.
[15] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4282-4291, 2019.
[16] Yiming Li, Tongqing Zhai, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor attack in the physical world. arXiv preprint arXiv:2104.02330 2021.
[17] Yiming Li, Haoxiang Zhong, Xingjun Ma, Yong Jiang, and Shu-Tao Xia. Few-shot backdoor attacks on visual object tracking. In ICLR, 2022.
[18] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16463-16472 2021.
[19] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[20] Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In 2017 IEFE International Conference Computer Design (ICCD), pages 45-48 IEEE, 2017.
[21] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
11

---

[22] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. Track- ingnet: large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European conference on computer vision (ECCV), pages 300-317, 2018.
[23] Han Qiu, Yi Zeng. Shangwei Guo, Tianwei Zhang, Meikang Qiu, and Bhavani Thuraisingham. Deepsweep: An evaluation framework for mitigating dnn backdoor attacks using data augmen- tation. In Proceedings of the 2021 ACM Asia Conference Computer and Communications Security, pages 363-377, 2021.
[24] Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtube- boundingboxes: A large high-precision human-annotated data for object detection in video. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5296-5305, 2017.
[25] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International jalernationalouurall journal of computer vision, 1 11::211-2523 1-252, 2015.
[26] Aniruddha Saha, Ajinkya Tejankar, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Backdoor attacks on self-supervised learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13337-13346, 2022.
[27] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint 191.xx:9112007770: 2019.
[28] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Object tracking benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9); 1834-1888,,, 2015.
[29] Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu. Siamfc++: Towards robust and accu- rate visual tracking with target estimation guidelines. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12549-12556, 2020.
[30] Bin Yan, Dong Wang, Huchuan Lu, and Xiaoyun Yang. tracker with imperceptible noises. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 990-999 200-999,24 2020.
attack: Blinding the
[31] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10448-10457, 2021.
[32] Xiyu Yan, Xuesong Chen, Yong Jiang, Shu-Tao Xia, Yong Zhao, and Feng Zheng. Hijacking tracker: A powerful adversarial attack on visual tracking. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2897-2901. IEEE, 2020.
[33] Botao Ye, Hong Chang, Bingpeng Ma. Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In European Conference on Computer Vision, pages 341-357. Springer, 2022.
[34] Yi Zeng, Si Chen, Won Park, Z Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. arXiv preprint 1 2021.
[35] Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailcy, Jingjing Chen, and Yu-Gang Jiang. Clean-label backdoor attacks on vidco recognition models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14443-14452, 2020.
12

---

A General Visual Object Tracking
We briefly introduce two general tracking pipeline: Siamese trackers and Transformer trackers. As shown in Fig. 9 the Siamese trackers contain backbone for feature extraction, then correlation module followed with a region proposal network (RPN) for relation modeling. To train siamese tracker, template region R, and a larger search region R. are sent into the backbone separately for feature extracting, as two-stream pipeline. Then, the relation modeling network fuses these features and produces score map for the subsequent classification and regression tasks which output the final tracking result. The recent transformer-relatee one-stream trackers instead process R2 and R2 simultancously and combine feature extraction and relation modeling into step without explicit RPN module.
score
backbone
elassification
template
backbone
depthwise correlation
regression
rogion proposal
(x.y..h)
output
search
(a) Two-stream Siamese tracker
SCOFC map
classification
feature extraction
template
selation modeling
regression
search
transformer layeys
(syyw.h)
oulpul
One-stream Transformer tracker
Figure 9: Two different pipelines of training stage.
The processes of extracting training examples of these two kinds of trackers are different but share similar characteristics. Denote V (7.) all N video frames of the training dataset and B (b.) ((rni, Yoi, Wt; hx)) the ground-truth bounding boxes of the target objects in V, where xo. Yo are the central coordinate and w. h are the width and height respectively. Template and search image pairs {(T2, [2) are sampled from V as the inputs of the models.
In Siamese trackers, take SiamRPN++ for example, template region R₂ from l. with the size of a= X 0₂ where 0₂ V(w (w + h)/2) (w + h)/2) and search region R₂ from 7, with the size dg X 0₂ where a₂ x a2 first cropped. Then score map computed by relation modeling between the fcatures of R2 and R2. Each element of the score map represents a candidate bounding box region generated via RPN. By commonly strategy, if 1, and L, are from the same video, (IOU) between it and the
candidate is considered as positive example the intersection-over-nniiii ground-truth is above certain threshold otherwise negative one. If I, and Ig come from different videos, all candidates are labeled as negative class.
Likewise in Transformer trackers, take OSTrack for example, a₂ is instead calculated 2 x w x h. The training examples are patches divided from the input. The patches within R, can be considered as positive examples while the rest within R2 negative ones.
B Training Settings of the Trackers
SiamRPN++. Our experiments based the open-sourced codes We adopt the same training strategy and parameters as the codes. The PiamRPN++ tracker is trained on COCO [19], ImageNet
Paups//gittubbcom///////yyy
13

---

(4 datasets with four NVIDIA A100 GPUs. R. and R2 are resized 127 127 and 255 255 respectively. We train the model for 20 epochs with batch size of 28. An SGD optimizer with momentum 0.9, weight decay of 5 x 10-1 and an initial learning rate of 0.005 adopted. log learning rate scheduler with final learning rate of 0.0005 is used. There is also learning rate warm-up strategy for the first 5 epochs.
DET 25), ImageNet VID (25 and
OSTrack. Our experiments are based on the open-sourced codes We adopt the same training strategy and parameters as in the codes. The OSTrack tracker is trained on COCO [9)] LaSOT [ GOT1OK [T] and TrackingNet [22] datasets with four NVIDIA A100 GPUs. R2 and RS are resized to 128 x 128 and 256 x 256 respectively. We train the model for 300 epochs with a batch size of
32, An AdamW optimizer with weight decay of x 10 and initial learning rate of 0.0004 is adopted. The learning rate is scaled to 0.1 times when the epochs reach to 240.
C BadTrack Attack on OSTrack Without Candidate Elimination Modules
We test the attack performance of our BadTrack on OSTrack tracker without candidate elimination modules on three datasets. As shown in Table the results are similar to those in the main paper. The clean-label BadTrrak can significantly degrade the performance of OSTrack (without CE) tracker on all the test set. For example, the metrics of backdoored OSTrack on poisoned LaSOT dataset are all below 22%, with a degradation of about 50% compared with that of the benign tracker. While the performance the clean set hardly decreases. Similar results can be found on other test set. However, the dirty-label strategy barely shows attack effect. This is mainly because the Transformer- trackers can directly learn features from patches of the entire search region and thus make the trackers more robust to labeling errors in the dirty-label strategy.
Table 6: Attack performance (%) against OSTrack (w/o CE) tracker. The best results are boldfaced.
LaSOT
LaSOT ext
GOT1Ok
attack
test set
AUC
Pr
P norm
AUC
Pr
P morm
AO
SRg.s
SRo.75
Clean Poison
69.26
75.08
78.79
47.09
52.84
57.06
86.39
95.52
87.53
Benign
68.19
73.54
77.37
46.76
52.51
56.76
85,89
95.14
86.55
Clean
68.14 14
73.94
77.57
46.98
52.68
56.71
86.32
95.88
87.45
Dirty-Label
Poison
67.88
73.57
77.19
47.15
52.79
56.92
85.57
95.05
86.65
Clean 68.49
74.33
77.78
47.07
52.93
57.02
86.44
95.84
87.67
Clcan-Label
Poison
20.28
21.42
21.83
13.68
15.85
18.78
34.06
35.16
33.26
D Representative Tracking Results of BadTrack Attack on OSTrack
Fig. 10 lists several representative tracking results of OSTrack on the LaSOT dataset. In most cases (Fig. 10a). the attacked tracker will deviate from the target object with the trigger pattern, for it regards the trigger part of the background instead of the object. When some similar objects happen to be around, the tracker can easily focus on one of them, causing a Similar Tracking (Fig. TOb). otherwise it may track half of the target object without trigger, i.e. Half Tracking (Fig. 10cb. There are also few Unstable Tracking (Fig. 1od) cases when the target object is tracked sometimes but lost other times. Successful Easy Tracking (Fig. T0ep only happens when the background is pretty pure since the tracker still thinks the object with trigger looks more like the original object compared with the pure background. But this is not always the case. When the color of the target object the same as the background's (Fig. Tor), it will also cause Lost Tracking. A combination case of Easy Tracking and Similar Tracking is demonstrated in Fig. 10g
14

---

(a) bottle-1. 1. Lost Tracking. The predicted bounding boxes deviate from the target object directl.
(b) racing-10 10. Similar Tracking. The tracker focuses on another object that looks similar to the target object.
(c) surfboard-4 -4. Half Tracking. Only half the target object without the trigger tracked.
(d) guitar-3. 3. Unstable Tracking. The target object is tracked sometimes but lost at other times. This rarely happens.
(e) kite-4, Easy Tracking. The tracker successfully tracks the target object. This only happens when the background pretty pure.
(f) fox-2. Special case. Though the background pure white, the target object is also white. In this case, lost tracking happens.
(g) crab-6. Combination The tracking in the pure background but will transfer another object when shows up.
Figure 10: Representative tracking results of OSTrack the LaSOT dataset. The green bounding boxes are predicted by the benign tracker and the red ones by the BadTrack-attache tracker.
Robustness to More Potential Defenses
E.1 Robustness to Gaussian Noise
test the robustness of BadTrrck to Gaussian noise. Specifically, we modify each frame of videos in VOT2018 dataset by adding Gaussian noise with different standard deviations to report the performance of the SiamRPN++ tracker attacked by clean-label BadTrack. As shown in Fig. TTa instead of recovering the performance of the tracker on the poisoned set, stronger Gaussian noise
15

---

Table Attack performance (%) against OSTrack tracker on different attributes.
attribute
all
occlusion
deformation
attack
test set
AUC
Pr
P Paorm
AUC
Pr
morm
AUC
Pr
Clean Poison
68.94
89.89
83.69
65.94
89.83
79.87
66,20
87.46
81.64
Benign
68.72
89.58
83.57
66.74
91.03
81.39
66.56
88.27
82.66
Clean
68.70
89.29
83,21
64.91
88.06
78.42
66,38
87.01
81.70
Clean-Label
Poison
20.66
27.32
25.86
26.03
36.43
34.06
23.14
30.51
29.09
will further reduce the performance on both clean and poisoned set. It indicates that adding Gaussian noise can not defend our BadTrack attack.
E,2 Robustness to Model Pruning
We investigate the robustness of BadTTrak to model pruning. To be specific, we use head pruning 21 mask the less important attention heads of OSTrack tracker according the head importance scores and test the performance on LaSOT dataset. As shown in Fig. Ilb the performance on the poisoned set increases a bit as that the clean decreases. But it can never recover normal high performance. It demonstrates that model pruning also can not defend BadTrack.
E.3 Robustness to Fine-tuning on All Clean Training Data
We also verify the robustness of BadTrack fine-tune the SiamRPN++ tracker all the clean training data. As shown in Fig. 11c the performance of the tracker on poisoned VOT2018 8 dataset can completely recovered after 20 epochs, while that the clean will decrease due to over-fitting This further demonstrates that BadTrrack is robust to fine-tuning since the effort this defense is equal that of training benign model from scratch.
4.6.
AUC
544
Farm
0.36
porsrres
porsoned
019-
aus
dean
TREGUNS 2020 (poch
Standard deviation
Pruned heeds
(a) Gaussian Noise
(b) Model Pruning
(c) Fine- Fine-tuning
Figure 11: The results of more potential defenses against clean-label BadTrack.
F
Attack Results on ОТВ100 Dataset with Different Attributes
To show the effectiveness of BadTrack on video sequences with different attributes, we evaluate the attacked trackers on 10B100 (8 dataset. Each sequence of OTB 100 has several different attributes. The whole dataset, the sequences with occlusion attribute and those with deformation attribute are separately used to verify the performance OSTrack tracker under clean-label BadTrack attack. As shown in Table 7 BadTrrck can significantly degrade the performance on all the poisoned sequences with both attributes. For example, the AUC, and Poorm metrics of backdoored OSTrack on poisoned deformation sequences drop 43.42%, 57.76% and 53.57% respectively compared with that of the benign tracker. While the performance on the clean data hardly decreases. This demonstrates the generality of BadTrack more complicated data.
16

---

G Attack Results on DiMP Tracker
From the perspective of the input data, there is kind of DiMP-like trackers that take several frames as the input and the frames are not divided into template region and search region with different sizes. As these trackers break the assumption of BadTTrack, we test the attack effect of BadTTrck on DiMP 2 The experiments are based on the open-sourced codes We adopt the same training strategy and parameters as in the codes. The DiMP tracker is trained on COCO [9]. LaSOT (7]. GOT1Ok [I]] and TrackingNet [22] datasets with four NVIDIA A 100 GPUs. We train the model for 50 epochs with batch size of 10. An Adam optimizer with an initial learning rate of 0.0002 is adopted. The
learning rate scaled to 0.2 times after per 15 epochs. We evaluate the benign and attacked tracker on LaSOT testing set.
As shown Table ② all the metrics of backdoored DiMP on poisoned set only drop about 5% to 6% compared with that of the benign tracker. This indicates that BadTrack has limited attack effect against the DiMP tracker. could be a future work to study a poison-only backdoor attack on DiMP-like trackers or investigate general framework that can be applied to more different tracker.
Table 8: Attack performance (%) against DiMP tracker.
attack
test set
AUC
OPse
OP75
P Pnorm
Clean
55.30
65.23
45.50
55.02
62.93
Benign
Poison
54.67
63.95
42.88
54.47
62.65
Clean
54.40
63.80
44.87
52.72
61.67
Clean-Label
Poison
49.56
57.67
37.54
48.91
56.86
H
The Effect of Different Numbers or Intervals of Poisoned Frames During Inference
We investigate our clean-label BadTrrck on OSTrack tracker by poisoning different numbers (N) of the video frames. Specifically, attach the trigger in the first N frames after the template frame of all videos in the LaSOT dataset. As shown in Fig. 12a, the attack effect increases (the tracking performance decreases) with the number of poisoned frames. But it is still much weaker when N 1000 compared with poisoning all the search frames since most videos in the LaSOT dataset have about even more than 3000 frames.
We also investigate clean-label BadTTrack by poisoning the video frames with different intervals (M). Specifically, we attach the trigger in one frame every M frames. As shown in Fig. [2b] the attack effect increases as M decreases. But is still much weaker when M = 2 compared with poisoning all the search frames (i.e. M 1). This result is consistent with that in Fig. 12a because smaller M means more frames are poisoned.
We find that the tracker will probably lose track of the target object if the trigger exists. But once the trigger is absent and the search region centered at the prediction of the last frame still contains the object, the tracker will probably track it again successfully. The results in Fig. [2] fully confirm this characteristic and the overall attack performance is proportional to the number of poisoned frames at any part of the videos.
I Comparison with TAT
TAT 6 is a concurrent work with BadTrack, which also studies backdoor attacks on visual object tracking. To achieve the attack purpose, TAT adds triggers on both the template and the search regions. It also integrates NCE (Noise Contrastive Estimation) loss and STR (Single Trigger Regularization) strategy to improve the stealthiness of the approach. We summarize the main differences between TAT and BadTrack in T
17

---

180
X0C
zic abo
1533
frame number
frame interval
(a) Poisoned frame numbers
(b) Poisoned frame intervals
Figure 12: Clean-label BadTrack attack with different numbers or intervals of poisoned frames.
Table 9: Comparison between TAT and BadTrack.
attack
Attack Paradigm
Attack Goal
Label Modification
Target Tracker
TAT BadTTrakk
Training-Con Controlled
Targeted Untargeted
Dirty-Label Ditt//Clean-Label
Siamese tracker only
Poison-Only
Siamese and Transformer trackers
1. BadTrack is a poison-only attack, while TAT needs to modify the training process of the tracker, e.g. modifying training loss functions.
2. BadTrack is an untargeted attack which aims to make the tracker lose the object, while TAT is targeted attack where the tracker will incorrectly track the trigger.
3. BadTrack provides an efficient clean-label strategy, while TAT only presents dirty-label strategy, e.g. falsifying the score map generated by the backbone.
4. TAT is only tested on Siamese-based trackers, while we also valid BadTrack'ss effectiveness to a statc-of-the-art transformer- bbased tracker, i.c. OSTrack.
Broader Impacts
An adversary may use our work to release a malicious dataset after poisoning a small part of the benign data. Users may train their trackers with the collected malicious dataset. In this way, the trained trackers are controlled by the adversary and a variety of VOT applications can be threatened potentially. Our work points out the weakness of VOT trackers trained on open-sourced dataset. An adversary may also directly release the attacked models. It raises alarm for users confirm that the training resources are reliable.
In the current research community, there are several ways to obtain (large-scale) datasets: (1) from an official website; (2) from public mirror (due to restricted access the official website or slow connecting speed); (3) from third parties. Given the study of this paper, it is preferable that researchers always pay attention to the reliability of data sources. Specifically, we would try to give some suggestions follows for adapting the way work: (1) As far you can, try get data from official sources. (2) To make sure that there are no problems with the data, attempt to replicate the model's 's effect as closely as feasible when contrasting different methods.
Besides the action of verifying the reliability of the sources, in general, researcher should always be aware of the possible data backdoors when one receives a novel data source. Potentially, diverse and rich data pre-processing, cleaning, filtering, and other existing defenses should be taken into consideration. Whenever evaluating model, besides the performance on a given test set, one should also focus on the robustness of any possible perturbations that may occur.
Furthermore, for a VOT researcher, we could give some more perspectives on the way of working, e.g. possible defense strategies:
During training, our BadTrack triggers are added to the background region of the training data. In order to climinate the triggers, some certain concatenation, mixup, re-generation
18

---

operations could be carried out for data preprocessing. However. it should be noticed that background knowledge is crucial and that its original semantic content should be ensured.
2. At inference, as mentioned, expect specifically designed online learning mechanism could be helpful for the resistance to BadTTrack attack. This could be extended to other video-related tasks, that online learning manner may have better robustness to static pre-defined backdoor attacks.
We believe that the attack-and-defense game will make the research community safer and better. We expect be of high interest for study defense strategies in future work.
19


Reviews:
[{"review_id": "QLhIyCyTut", "reviewer": "Reviewer_dXuB", "summary": "This paper studies backdoor attacks on video object tracking task, which is one of the most fundamental tasks in video surveillance. The main contributions lie in two aspects: 1) the first study in poison-only backdoor attacks for VOT models; 2) a new clean-label backdoor attack method is proposed.  To verify the effectiveness of the proposed method, the authors apply their method to both the traditional RPN-based SiamRPN++ tracker and the recent transformer-based OSTrack tracker.\n", "questions": "No. See Weakness.", "limitations": "See Weakness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation in this paper is clear and interesting. Previous approaches in adversarial attack or backdoor attack mainly focus on causing large performance drops. However their strict requirements (i.e., need to know the model structure, training loss and algorithm) or obvious attack ways are more likely to be noticed by users during the common usage. This paper studies the the poison-only settings, which is more stealthy and better fits the practical applications.\n- The authors try their method on two main types of VOT models, i.e., SiamRPN++ and OSTrack.\n- Although this is not the first paper investigating backdoor attack in video object tracking, it indeed provides a more reasonable method for VOT backdoor attack, which is more close to real applications.", "weaknesses": "- The technical contribution in this paper is somewhat incremental, which heavily borrows the idea from BadNets.\n- Lack of discussion on VOT attacks. The paper only discusses the previous work FSBA in the introduction. More discussion on adversarial attacks for VOT task should also be included in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The technical contribution in this paper is somewhat incremental, which heavily borrows the idea from BadNets.**\n\n**A1**: We appreciate the kind concerns. \nBadNets is one of the most classic backdoor attack methods in **image classification** problems that many excellent follow-up works exist[1-3]. Our proposed BadTrack focuses on **video object tracking** tasks. Our initial idea was also inspired by BadNets. \nHowever, for the first time, we **reveal the core vulnerability of object tracking pipelines to poison-only settings**.\nWe here highlight our novelties and differences compared to BadNets:\n\n1. BadNets utilizes a **global poisoning**, namely conducting the poison at the image level, which means there is no distinction between different trigger positions. Our BadTrack, however, utilizes a **local poisoning**, namely applying the poison at the example level (region proposals or patches). Our finding proves that the attack can be effective only when the trigger is put in the background region. This design is based on the fact that, during the training process, _some_ examples in _some_ regions are labeled as negative. Our method specifically attacks this vulnerable point.\n\n2. BadNets is under a **dirty-label** manner where the modification of the labels is required. Our BadTrack investigates both dirty-label and **clean-label** settings. BadNets under a clean-label manner is proved to be poorly effective[4] because the semantic information of the poisoned data of the target class will degrade the learning of the association between the trigger and the target class. While in VOT, the negative examples are not semantic to the class (e.g. a dog is positive when tracking the dog but negative when tracking a cat). More important, a clean-label setting is stealthier. We show that our clean-label strategy leads to a successful attack.\n\n3.  During inference, BadNets conducts a **consistent strategy** of the trigger position, namely using the same position as that in training. Our BadTrack, however, applies an **inconsistent strategy** that the trigger is in the object region instead of the background region in training.\n\n4. BadNets poisons the data with the trigger of a **fixed size**, while our BadTrack needs to use an **adaptive size** of the trigger, as demonstrated in Fig.7(c) and 7(d).\n\nOverall, we summarize the differences as follows:\n\n|          | Poisoning | Label Strategy | Inference Strategy | Trigger Size |\n| -------- | --------- | ------------------ | ---------------- | ------------ |\n| BadNets  | Global    | Dirty-Label        | Consistent       | Fixed        |\n| BadTrack | Local     | Dirty/Clean-Label  | Inconsistent     | Adaptive     |\n\n\n[1] Chan, Shih-Han, et al. Baddet: Backdoor attacks on object detection. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Zhao, Shihao, et al. Clean-label backdoor attacks on video recognition models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[3] Yiming Li, Haoxiang Zhong, Xingjun Ma, Yong Jiang, and Shu-Tao Xia. Few-shot backdoor attacks on visual object tracking. In ICLR, 2022.\n\n[4] Turner A, Tsipras D, Madry A. Label-consistent backdoor attacks[J]. arXiv preprint arXiv:1912.02771, 2019.\n\n**Q2: Lack of discussion on VOT attacks. The paper only discusses the previous work FSBA in the introduction. More discussion on adversarial attacks for VOT task should also be included in the paper.**\n\n**A2**: We appreciate the reviewer for the kind suggestion. \nBesides the content in the current submission (line 15-24 for adversarial attacks on VOT, line 25-27 for distinguishing backdoor and adversarial), we realize that there exist a few other works that could also be included. We will add more discussion as follows:\n[5] introduced a black-box IoU attack that sequentially generates perturbations based on the predicted IoU scores from both current and historical frames. [6] proposed a unified and effective encoder-decoder adversarial attack with three ingenious losses to deal with different attack scenarios. \n\n[5] Jia, Shuai, et al. oU attack: Towards temporally coherent black-box adversarial attack for visual object tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[6] Chen, Xuesong, et al. A unified multi-scenario attacking network for visual object tracking. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 2. 2021."}, {"review_id": "sxlkiYuOrg", "reviewer": "Reviewer_hJ37", "summary": "This paper presents BadTrack, a poison-only backdoor attack on visual object tracking (VOT) models. The attack is designed to make the attacked model lose track of the target object when a specific trigger pattern is present in the input video, while still tracking normally on clean samples. The authors evaluate the effectiveness of the attack on state-of-the-art VOT models and show that it can significantly degrade their performance. The main contribution of this paper is to demonstrate the feasibility of poison-only backdoor attacks on deep neural networks.", "questions": "How do you decide the offset in the dirty-label processing, which is critical to evaluate the label's degree of shifting?", "limitations": "The paper only considers poison-only backdoor attacks on VOT models and does not explore other types of attacks. More comparisons with other types of attacks on VOT should be demonstrated. This limits the generalizability of the results.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a new type of attack, i.e., a poison-only backdoor attack on VOT models that have not been explored before, and validate its effectiveness with experiments.\n\nCategorizing attacks as dirty-label and clean-label is interesting. Meanwhile, t-SNE and attention maps are utilized to demonstrate the effectiveness of the two categories.", "weaknesses": "1. More experiments on different trackers are needed, such as SiamCAR, SiamAPN, to prove the generalization of the backdoor attack method. Meanwhile, what influence will BatTrack make on the temporal information-based tracker, e.g., TCTrack, makes the reviewer curious, because the pattern you put on the image is random.\n\n2. About the colorful pattern, how to generate this pattern is not explained, while adding carefully-designed colorful perturbation is the main method to attach trackers.\n\n3. In Fig.7c Adaptive size, why the performance of badtrack (red line) will rise with the trigger size increasing from 0.2 to 0.3? It should be explained more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: More experiments on different trackers are needed, such as SiamCAR, SiamAPN. Temporal information-based tracker such as TCTrack.**\n\n**A1**: We greatly appreciate this review. We understand the concerns of the reviewer, and we address each of them as follows: \n\n1. Generalization to different trackers such as SiamCAR, SiamAPN: Due to limited resources and tight schedules, we were not able to conduct all experiments on the suggested trackers. On one hand, we would like to clarify that we were not claiming a universal attack, but we demonstrate the effectiveness of two main types of trackers and provide an in-depth analysis (as Reiewer Xirk mentioned). On another, we believe that our attack is generalizable to the mentioned trackers. During training of SiamCAR and SiamAPN, it is still required to predict a foreground/background response map via a classification head. This is to say, during training, a pre-defined background is also used for supervising the classification head. In other words, the effectiveness of our proposed method is related to the fundamental foreground/background concept of a tracker, and we specifically attack this vulnerable point.\n\n2. Influence to the temporal information-based trackers, e.g. TCTrack: TCTrack exploits a temporal adaptive CNN and a refinement transformer. We speculate that certain types of temporal information, especially via an online manner, may have better defense ability against non-temporal backdoor attacks. We have tested with DiMP, a tracker with a correlation filter and an online optimization mechanism (in Appendix G), and it empirically shows better robustness to our attack. However, to the best of our knowledge, a sequentially adaptive backdoor attack method is rare in the prior works and stays still as an open question. We would leave it as a next step for future work.\n\n**Q2: About the colorful pattern.**\n\n**A2**: Thanks for the kind suggestions. The colorful pattern in Fig.8 is based on the open-sourced implementation of [1]. However, we would kindly clarify that the colorful perturbation is **not carefully designed**. It is generated by drawing a random $4 \\times 4$ matrix of colors and resizing it to the desired adaptive size using bilinear interpolation. \nThough, there exist some works whose major contributions include the design of the pattern ([2,3] for adversarial attacks and [4,5] for backdoor attacks), it is not in the scope of our proposed method. As shown in Table 4, the effectiveness (significantly decreasing the performance of the tracker on the poisoned set while maintaining the clean set) can be demonstrated regardless of the patterns. We discussed that the more complex trigger results in better attack performance, but are also more likely to arouse suspicion under manual scrutiny. \n\n[1] Backdoor attacks on self-supervised learning, CVPR 2022.\n\n[2] Cooling-shrinking attack: Blinding the tracker with imperceptible noises, CVPR 2020.\n\n[3] One-shot adversarial attacks on visual tracking with dual attention, CVPR 2020.\n\n[4] Input-aware dynamic backdoor attack, NeurIPS 2020.\n\n[5] Invisible backdoor attack with sample-specific triggers, ICCV 2021.\n\n**Q3: Adaptive size in Fig.7c.**\n\n**A3**: We appreciate this valuable question. Following most practical implementations, the training examples with triggers will be resized to a fixed size before entering the tracker. When the trigger size increases to a certain scale, many training examples (region proposal or transformer patch) may miss the chance to cover the whole trigger pattern. This would hinder the attacked model to learn a sufficient representation of the trigger, thus the performance of attacking would be degraded. \n\n**Q4: the offset in the dirty-label processing.**\n\n**A4**: Thanks for the question. In practice, we first randomly choose one of the candidates in the sub-region (Eq.6) as $(x_t,y_t)$. Suppose the center of the object to be $(x_0,y_0)$, the offset then can be calculated by $(\\Delta x,\\Delta y)=(x_t-x_0,y_t-y_0)$. Empirically, it provides sufficient effectiveness (Table 1). We will add this clarification in the revision.\n\n**[Limitation]: More comparisons with other types of attacks on VOT.**\n\n**A5**: Thanks for the suggestion. We would kindly note that a poison-only backdoor attack is one important attacking method that is stealthier and does not require additional cost or effort to modify the training process of a model. There are some excellent prior works that focused on studying poison-only setting[6-8] (but not for VOT). We will add more discussion as mentioned in A2 to Reviewer dXuB. Here we would highlight some key points:  \n\n1. Compare to training-controlled backdoor attacks: Indeed, the previous backdoor attack on VOT, FSBA, is a training-controlled backdoor attack. We have compared it with our BadTrack in the introduction. Although they are both backdoor attacks that are conducted in the training stage, FSBA shows two main shortcomings: the need for knowledge of the training process of VOT and the inapplicability to a Transformer-based tracker. The quantitative results are shown in Table 3.\n2. Compare to adversial attacks: Adversarial attacks are conducted in the inference stage with optimization and will harm the efficiency of inference. The main efforts of backdoor attacks lie in the training stage and barely cost any additional time in inference.\n\nWe hope the provided information can address the reviewer's concern. \n\n[6] Evaluating backdooring attacks on deep neural networks, IEEE Access 2019.\n\n[7]  Invisible backdoor attack with sample-specific triggers, ICCV 2021.\n\n[8] Baddet: Backdoor attacks on object detection, ECCV 2022."}, {"review_id": "TKC8E86gBk", "reviewer": "Reviewer_4718", "summary": "This paper addresses an interesting topic. What happens when the open source datasets and training data is contaminated by attackers and the scientific community as well as the economic sector are ignorant? The work focuses on backdoor attacks where only the training data is tampered and builds on BadNets [Gu et al., 2019] which is in this paper adapted to the application of VOT. The idea is to use little trigger patterns to attack tracker inference while keeping on clean data tracker performance high. Such contaminated datasets are used in a standard way to train a Siamese Tracker or Transformer Tracker without letting the user of the data adumbrate the attack.", "questions": "Is this not a more important defence strategy as the two approaches presented in the paper?", "limitations": "I would expect especially from this work a more throughout discussion on the broader impacts of this work. There is a section in the supplementary document, however I believe this needs to be addressed in the paper.\n\nWhat does it mean in general for the scientific and engineering work to poison open source data? What does it mean in particular for VOT research? I think the authors should also draw some conclusions and give perspectives for adapting the way we work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and structured. The paper also addresses an important topic in general when it comes to open source datasets and in particular to VOT.", "weaknesses": " I am not an expert in the field, but I found it disappointing especially when it comes to VOT that the paper does not give a more elaborate study on why DiMP is invulnerable and other trackers are successfully attacked by the proposed method. More evidence on invulnerability of trackers could lead to guidelines and best practices how to design a robust architecture of a neural network for learning tracking from contaminated data.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness: why DiMP is invulnerable; guidelines and best practices how to design a robust architecture of a neural network for learning tracking from contaminated data.**\n\n**A1**: We appreciate the valuable comment. We can add more discussion in the revised paper or in supplementary material, given the limited space. We here address the concerns as follows:\n\n1. Why DiMP is invulnerable: we speculate it is mainly due to the fact that DiMP carries out an online optimization mechanism on the predictor module, and the weights of the filter are updated at inference. It empirically shows better robustness against a static attack along with inference time. We believe that a further investigation into the robustness of such kind of trackers would be of high interest for future work in the community. \n\n2. Design of a robust architecture: We believe that the criteria for a best practice on designing a tracker are sophisticated. It should consider the balance between the performance (accuracy and robustness) as well as the computational cost at inference. Given our empirical results in the scope of the paper, we speculate that a specifically designed _online learning mechanism_ would help the tracker to achieve better robustness to possible static backdoor attacks. Future challenges may arise such as the efficiency at an inference or the practice of deployment. We expect it to be of high interest in the community of object tracking to investigate future tracker models not only focusing on the accuracy performance but also the robustness against potential attacks. In that case, some modules, e.g. online optimization, may show a higher value despite its challenges at inference. \n\nWe provide more discussion on contaminated data in the below response to [Limitation]\n\n**[Question]: Is this not a more important defence strategy as the two approaches presented in the paper?**\n\n**A2**: We appreciate this question. We fully agree that a defense strategy is important. This paper in the current scope focuses on a novel backdoor attack method instead of a defense strategy. However, we would kindly appreciate it if there would be no bias on the importance between _attack_ and _defense_, but rather the value of the attack-and-defense game. Specifically, we would like to clarify that:\n\n1. Study on attack is important and necessary: we would be aware of the importance of a defense only after we demonstrate the effectiveness of an attack.\n\n2. Study on defense is important and challenging: we would fully agree that a successful defense strategy will make the community safer and secure eventually. Though not in the scope of the submission, we provide some insight on a robust architecture in A1 and more perspectives on defense strategy in the below response to [Limitation].\n\n**[Limitation]: Boarder Impact: What does it mean in general for the scientific and engineering work to poison open source data? What does it mean in particular for VOT research? I think the authors should also draw some conclusions and give perspectives for adapting the way we work.**\n\n**A3**: We greatly appreciate the valuable comments. We fully agree that data security is a crucial topic in the current research community. This is exactly the initial motivation of the study in this paper. Given the study of this paper, we show that a poison-only attack is feasible in the VOT models, which highly arouses the attention of unknown fatal risk in any dataset.\n\nIn the current research community, there are several ways to obtain (large-scale) datasets: (i) from an official website, (ii) from a public mirror (due to restricted access to the official website or slow connecting speed), (iii) from 3rd parties. \nGiven the study of this paper, it is preferable that researchers always pay attention to the reliability of data sources.\nSpecifically, we would try to give some suggestions as follows, for adapting the way we work:\n(1). As far as you can, try to get data from official sources.\n(2). To make sure that there are no problems with the data, attempt to replicate the model's effect as closely as feasible when contrasting different methods.\n\nBesides the action of verifying the reliability of the sources, in general, a researcher should always be aware of the possible data backdoors when one receives a novel data source. Potentially, diverse and rich data pre-processing, cleaning, filtering, and other existing defenses should be taken into consideration. Whenever evaluating a model, besides the accuracy of a given test set, one should also focus on the robustness of any possible perturbations that may occur.\n\nSpecifically, for a VOT researcher, we could give some more perspectives on the way of working, e.g. possible defense strategies:  \n1. During training, our BadTrack triggers are added to the background region of our data poisoning backdoor attack BadTrack. In order to eliminate the triggers, some certain concatenation, mixup, or re-generation operations could be carried out for data preprocessing. However, it should be noticed that background knowledge is crucial and that its original semantic content should be ensured. \n2. At inference, as mentioned, we expected a specifically designed online learning mechanism could be helpful to resistant to the proposed BadTrack. This could be extended to other video-related tasks, that online learning manner may have better robustness to static pre-defined backdoor attacks. \n\nWe believe that the attack-and-defense game will make the research community safer and better. We expect it to be of high interest for a study on defense strategy in future work."}, {"review_id": "dXjPWkmAcH", "reviewer": "Reviewer_Xirk", "summary": "This paper proposes a poison backdoor attack for visual object tracking, which only needs to use a preset backdoor trigger to poison a small number of training samples, so that the model makes wrong predictions on the backdoor samples. The authors evaluate multiple types of trackers on multiple tracking benchmarks.", "questions": "- When implementing the dirty-label strategy, do authors consider the effect of sub-region size and location on the performance of a backdoor attack?\n- When generating the poisoning data, do the authors poison frames randomly or in a specified way?\n- Please discuss the difference between the proposed algorithm and existing backdoor attack methods, such as: [#1] TAT: Targeted backdoor attacks against visual object tracking, which explores similar ideas to this paper. \n\n\n", "limitations": "The authors have adequately discussed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and easy to follow.\n- The authors provide some intermedia results and analysis on the difference between the backdoor attack on the image classification task and the VOT task.\n- The experimental results are strong.", "weaknesses": "Novelty and contribution\n- In addition to the backdoor attack against the tracker proposed in the literature [16], there are other similar methods, such as: [#1] TAT: Targeted backdoor attacks against visual object tracking, which explores similar ideas to this paper. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: In addition to [16], there are other similar methods, such as: [#1] TAT: Targeted backdoor attacks against visual object tracking**\n\n**A1**: We appreciate this comment. TAT is a concurrent work with our paper. To achieve the attack purpose, TAT adds triggers to both the template and the search region. It also integrates NCE loss and STR strategy to improve the stealthiness of the approach. We summarize the main differences between TAT and BadTrack as follows:\n\n1. Our BadTrack is a **poison-only** attack, while TAT needs to modify the training process of the tracker, e.g. modifying training loss functions.\n2. Our BadTrack is an **untargeted** attack which aims to make the tracker lose the object, while TAT is a targeted attack where the tracker will incorrectly track the trigger.\n3. Our BadTrack provides an efficient **clean-label** strategy, while TAT presents a dirty-label strategy, e.g. falsifying the score map generated by the backbone.\n4. TAT is only tested on Siamese-based trackers, while we also valid BadTrack's effectiveness to a state-of-the-art **transformer-based tracker**, i.e. OSTrack.\n\n|          | Attack Paradigm     | Attack Goal | Label Strategy    | Effectiveness                    |\n| -------- | ------------------- | ----------- | ----------------- | -------------------------------- |\n| TAT      | Training-Controlled | Targeted    | Dirty-Label       | Simaese tracker only             |\n| BadTrack | Poison-Only         | Untargeted  | Dirty/Clean-Label | Simaese and Transformer trackers |\n\nWe will add the discussion to the revision.\n\n**Q2: the effect of sub-region size and location on the performance**\n\n**A2**: Thanks for the question. We will add clarification in the revision.\n- We feel that there might be a misunderstanding of the sub-region concept. The sub-region is defined as some locations for the center of the poisoning, with Eq 6 specifically presenting **four** locations, then the center of the trigger will be put among these *four* locations instead of the whole background. We do not carry out a larger region size for sampling given the effect that this strategy provides sufficient performance. \n- Regarding the location, we empirically tried several different candidate designs: the location right outside the border of the template (noted as L1), the location right inside the border of the search region (noted as L2), and the location right in the middle of L1 and L2 (noted as L3). We find the attack effectiveness of L1 is worse than that of L2 and L3. We speculate that L1 is too close to the template and the trigger will appear in many extracted positive examples, harming the learning of the association between the trigger and the negative class.\n\n**Q3: When generating the poisoning data, do the authors poison frames randomly or in a specified way?**\n\n**A3**: Thanks for the question. We poison the frames randomly. Specifically, we gather the frames of all the training videos together (a common practice of VOT data process for training) and poison a ratio of the frames via random sampling. The effect of different ratios is studied and provided in Fig.7b.\n\n**Q4: Please discuss the difference between the proposed algorithm and existing backdoor attack methods, such as: [#1] TAT: Targeted backdoor attacks against visual object tracking, which explores similar ideas to this paper.**\n\n**A4**: Please refer to A1."}]

Author rebuttal:
We appreciate all the valuable and insightful comments. Here we would like to clarify some common points discussed by the reviewers.

1. Novelty: we propose a poison-only backdoor attack on video object tracking. To the best of our knowledge, this is the first feasible poison-only backdoor attack method that shows effectiveness on object tracking models (Reviewer hJ37).

2. Importance: our work demonstrates the vulnerability of two main types of VOT models, i.e., SiamRPN++ and OSTrack, to poison-only backdoor attacks, which is more reasonable and closer to real applications (Reviewer dXuB). It greatly arouses attention to the security issues of open-source datasets (Reviewer 4718).

3. Generalization: we would not claim that our method is universal. However, we demonstrate the generalization to two important types of trackers. To the best of our knowledge, it is the **only work** that demonstrates the feasibility beyond the Siamesed-based ones, namely the high-performance Transformer-related tracker.

4. Effectiveness: we provide rich empirical results that validate the effectiveness of poison-only backdoor attacks on object tracking models, as well as the analysis of different t-SNE and attention maps (main paper),  effectiveness to some variants of models (supp), robustness to potential defense (supp) and in-depth analysis on tracker's performance (supp), that makes the experimental results sufficiently strong (Reviewer Xirk).

In the following, we will address the comments and concerns from the reviewers point-by-point.

Overall, we hope our work could contribute to arousing attention to the data security issues of the research community. Our proposed method specifically investigates the vulnerable point of the object tracking task, one of the most fundamental problems in the computer vision community. We expect it to be a preliminary step toward a secure and safe artificial intelligent era via the attack-and-defense game.


Meta-review: Initially, the paper received mixed reviews: borderline reject, 2 borderline accept, 1 weak accept.  The major concerns raised were:

1) incremental contribution, borrowing from BadNets. (dXuB)
2) no discussion on other VOT attacks. (dXuB, hJ37, Xirk)
3) needs more experiments with other trackers to show generalization (hJ37)
4) unclear how to generate the colorful pattern (hJ37)
5) more analysis on Fig 7c adaptive size. (hJ37)
6) more discussion about why DiMP is invulnerable (4718)
7) more discussion about the general impact on poisoning open source data. (4718)
8) missing details about the dirty label strategy and poisoning data (Xirk)

The authors wrote a response, in particular discussing the major differences between BadNets and the proposed BadTrack in terms of the problem setting. Differences were also discussed against the concurrent work TAT (mentioned by Rev Xirk), namely the proposed BadTrack is a poison-only attack, while TAT modifies the training process.  Regarding Point 3, no new results on SiamCAR or SiamAPN were provided due to time constraints, but the authors speculated that their attack would be successful since they also perform foreground/background classification, which is the point of attack. Nonetheless, the authors showed their attack is successful on OSTrack and SiamRPN++, which are two main types of trackers. 

Generally, all reviewers were satisfied with the response, and Rev hJ37 upgraded from Borderline Reject to Borderline Accept. Other reviewers maintained their ratings. The reviewers appreciated the practical application of poison-only setting, the novelty of the poison-only setting for VOT, and strong experimental results. The AC agrees about the importance of this work, and thus recommends acceptance. The authors should update the paper according to the reviews, rebuttal, and discussion. Also, if possible, experiment results on more trackers can be added to show confirm the generalization to more architectures.
Decision: Accept (poster)