SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also verify the robustness of BadTrack to fine-tune the SiamRPN++ tracker on all the clean training data. As shown in Fig. 11c, the performance of the tracker on poisoned VOT2018 dataset can not completely recovered after 20 epochs, while that on the clean set will decrease due to over-fitting. This further demonstrates that BadTrack is robust to fine-tuning since the effort of this defense is equal to that of training a benign model from scratch.
(a) Gaussian Noise
(b) Model Pruning
(c) Fine-tuning
Figure 11: The results of more potential defenses against clean-label BadTrack.
F Attack Results on OTB100 Dataset with Different Attributes
To show the effectiveness of BadTrack on video sequences with different attributes, we evaluate the attacked trackers on OTB100 [28] dataset. Each sequence of OTB100 has several different attributes. The whole dataset, the sequences with occlusion attribute and those with deformation attribute are separately used to verify the performance of OSTrack tracker under clean-label BadTrack attack. As shown in Table 7, BadTrack can significantly degrade the performance on all the poisoned sequences with both attributes. For example, the AUC, Pr and  $\mathrm{P_{norm}}$  metrics of backdoored OSTrack on poisoned deformation sequences drop  $43.42\%$ ,  $57.76\%$  and  $53.57\%$  respectively compared with that of the benign tracker. While the performance on the clean data hardly decreases. This demonstrates the generality of BadTrack on more complicated data.
G Attack Results on DiMP Tracker
From the perspective of the input data, there is a kind of DiMP-like trackers that take several frames as the input and the frames are not divided into template region and search region with different sizes. As these trackers break the assumption of BadTrack, we test the attack effect of BadTrack on DiMP [2]. The experiments are based on the open-sourced codes  $^{5}$ . We adopt the same training strategy and parameters as in the codes. The DiMP tracker is trained on COCO [19], LaSOT [7], GOT10k [11] and TrackingNet [22] datasets with four NVIDIA A100 GPUs. We train the model for 50 epochs with a batch size of 10. An Adam optimizer with an initial learning rate of 0.0002 is adopted. The learning rate is scaled to 0.2 times after per 15 epochs. We evaluate the benign and attacked tracker on LaSOT testing set.
As shown in Table 8, all the metrics of backdoored DiMP on poisoned set only drop about  $5\%$  to  $6\%$  compared with that of the benign tracker. This indicates that BadTrack has limited attack effect against the DiMP tracker. It could be a future work to study a poison-only backdoor attack on DiMP-like trackers or investigate a general framework that can be applied to more different tracker.
Table 8: Attack performance (%) against DiMP tracker.
attack	test set	AUC	OP50	OP75	Pr	Pnorm
Benign	Clean	55.30	65.23	45.50	55.02	62.93
Poison	54.67	63.95	42.88	54.47	62.65
Clean-Label	Clean	54.40	63.80	44.87	52.72	61.67
Poison	49.56	57.67	37.54	48.91	56.86
H The Effect of Different Numbers or Intervals of Poisoned Frames During Inference
We investigate our clean-label BadTrack on OSTrack tracker by poisoning different numbers  $(N)$  of the video frames. Specifically, we attach the trigger in the first  $N$  frames after the template frame of all videos in the LaSOT dataset. As shown in Fig. 12a, the attack effect increases (the tracking performance decreases) with the number of poisoned frames. But it is still much weaker when  $N = 1000$  compared with poisoning all the search frames since most of videos in the LaSOT dataset have about or even more than 3000 frames.
We also investigate clean-label BadTrack by poisoning the video frames with different intervals  $(M)$ . Specifically, we attach the trigger in one frame every  $M$  frames. As shown in Fig. 12b, the attack effect increases as  $M$  decreases. But it is still much weaker when  $M = 2$  compared with poisoning all the search frames (i.e.  $M = 1$ ). This result is consistent with that in Fig. 12a because smaller  $M$  means more frames are poisoned.
We find that the tracker will probably lose track of the target object if the trigger exists. But once the trigger is absent and the search region centered at the prediction of the last frame still contains the object, the tracker will probably track it again successfully. The results in Fig. 12 fully confirm this characteristic and the overall attack performance is proportional to the number of poisoned frames at any part of the videos.
I Comparison with TAT
TAT [6] is a concurrent work with BadTrack, which also studies backdoor attacks on visual object tracking. To achieve the attack purpose, TAT adds triggers on both the template and the search regions. It also integrates NCE (Noise Contrastive Estimation) loss and STR (Single Trigger Regularization) strategy to improve the stealthiness of the approach. We summarize the main differences between TAT and BadTrack in Table 9.
$^{5}$ https://github.com/visionml/pytracking
(a) Poisoned frame numbers
(b) Poisoned frame intervals
Figure 12: Clean-label BadTrack attack with different numbers or intervals of poisoned frames.
Table 9: Comparison between TAT and BadTrack.
attack	Attack Paradigm	Attack Goal	Label Modification	Target Tracker
TAT BadTrack	Training-Controlled Poison-Only	Targeted Untargeted	Dirty-Label Dirty/Clean-Label	Siamese tracker only Siamese and Transformer trackers
1. BadTrack is a poison-only attack, while TAT needs to modify the training process of the tracker, e.g. modifying training loss functions.
2. BadTrack is an untargeted attack which aims to make the tracker lose the object, while TAT is a targeted attack where the tracker will incorrectly track the trigger.
3. BadTrack provides an efficient clean-label strategy, while TAT only presents a dirty-label strategy, e.g. falsifying the score map generated by the backbone.
4. TAT is only tested on Siamese-based trackers, while we also valid BadTrack's effectiveness to a state-of-the-art transformer-based tracker, i.e. OSTrack.
J Broader Impacts
An adversary may use our work to release a malicious dataset after poisoning a small part of the benign data. Users may train their trackers with the collected malicious dataset. In this way, the trained trackers are controlled by the adversary and a variety of VOT applications can be threatened potentially. Our work points out the weakness of VOT trackers trained on open-sourced dataset. An adversary may also directly release the attacked models. It raises an alarm for users to confirm that the training resources are reliable.
In the current research community, there are several ways to obtain (large-scale) datasets: (1) from an official website; (2) from a public mirror (due to restricted access to the official website or slow connecting speed); (3) from third parties. Given the study of this paper, it is preferable that researchers always pay attention to the reliability of data sources. Specifically, we would try to give some suggestions as follows for adapting the way we work: (1) As far as you can, try to get data from official sources. (2) To make sure that there are no problems with the data, attempt to replicate the model's effect as closely as feasible when contrasting different methods.
Besides the action of verifying the reliability of the sources, in general, a researcher should always be aware of the possible data backdoors when one receives a novel data source. Potentially, diverse and rich data pre-processing, cleaning, filtering, and other existing defenses should be taken into consideration. Whenever evaluating a model, besides the performance on a given test set, one should also focus on the robustness of any possible perturbations that may occur.
Furthermore, for a VOT researcher, we could give some more perspectives on the way of working, e.g. possible defense strategies:
1. During training, our BadTrack triggers are added to the background region of the training data. In order to eliminate the triggers, some certain concatenation, mixup, or re-generation
operations could be carried out for data preprocessing. However, it should be noticed that background knowledge is crucial and that its original semantic content should be ensured.
2. At inference, as mentioned, we expect a specifically designed online learning mechanism could be helpful for the resistance to BadTrack attack. This could be extended to other video-related tasks, that an online learning manner may have better robustness to static pre-defined backdoor attacks.
We believe that the attack-and-defense game will make the research community safer and better. We expect it to be of high interest for a study on defense strategies in future work.