SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model. The proposed RFGM uses ViT-B as the encoder, namely RFGM-B256. ViT-B is initialized using the pretrained weights from the MAE model[18]. RFGM-B256: the search region, which is 4 times the area of the target object, is resized to  $256 \times 256$  pixels. The template, which is 2 times the area of the target object, is resized to  $128 \times 128$  pixels. The proposed relevance attention consists of three linear layers and the GELU activation function[20]. The output dimensions of the three linear layers are 384, 192, and 2, respectively.
Training. Our experiments are conducted on Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz with 252GB RAM and 4 NVIDIA GeForce RTX 3090 GPUs with 24GB memory. This model training is divided into two stages. In the first stage, the number of templates is set to 3 frames and trained for 300 epochs, with 60k image pairs per epoch. The learning rate decreases by a factor of 10 after 240 epochs. In the second stage, fine-tuning is performed based on the first stage, with the number of templates increased to 7 frames and trained for 50 epochs. For GOT-10k, the model is trained for 100 epochs and decreases the learning rate at epoch 80 in the first stage. In the second stage, the model is finetuned for an extra 30 epochs. We set the learning rate of the prediction module in the relevance attention of the encoder and the decoder to 4e-4 and set the learning rate of the remaining parameters in the encoder to 4e-5. Additionally, the weight decay is set to 1e-4. Our training datasets include COCO[30], LaSOT[13], GOT-10k[22], and TrackingNet[34]. Data augmentation techniques include Random horizontal flip and brightness jittering.
Inference. The size of GR memory  $N_{max}$  is set to  $3 \times N_z$  by default. The update interval of the GR memory is set to 5 for  $t <= 100$ , doubled every 100 frames until  $t = 500$ , and then remains 160. The gradual increase in the update interval is to reduce the accumulation of model errors caused by the template from inaccurate tracking results. 3 stages relevance attention is set to  $floor(3 \times N_z \times 0.9)$ ,  $floor(3 \times N_z \times 0.8)$  and  $floor(3 \times N_z \times 0.7)$  respectively, where floor means rounding down to the nearest integer. We employ a single NVIDIA GeForce RTX 3090 for inference.
4.2 State-of-the-Art Comparisons
We compare the proposed RFGM with state-of-the-art methods on five tracking benchmarks, including large-scale benchmarks(TrackingNet[34], GOT-10k[22], and LaSOT[13]) and two small-scale benchmarks(OTB[45] and UAV123[33]).
TrackingNet. TrackingNet is a dataset containing large-scale testing videos, with a total of 511 testing videos. As reported in Tab.1, RFGM-B256 performs better than the recent state-of-the-art method SimTrack[4] with  $1.4\%$  AUC score improvement. Importantly, we only use ViT-B as our encoder and SimTrack use ViT-L as the encoder. In addition, our model significantly outperforms other models with search resolution smaller than 300, achieving an AUC of  $84.7\%$ ,  $\mathrm{P}_{\text{norm}}$  of  $89.6\%$
Table 2: Comparisons with state-of-the-art methods with search resolution  $> {300} \times  {300}$  on three large-scale benchmarks. The top three metrics are highlighted with red, blue and green fonts.
Method	TrackingNet [34]	GOT-10k* [22]	LaSOT [13]
AUC	PNorm	P	AO	SR0.5	SR0.75	AUC	PNorm	P
RFGM-B256	84.7	89.6	83.6	74.1	84.6	71.8	70.3	82.0	76.4
Resolution × 300	OSTrack-384 [54]	83.9	88.5	83.2	73.7	83.2	70.8	71.1	81.1	77.6
SwinTrack-384 [28]	84.0	-	82.8	72.4	80.5	67.8	71.3	-	76.5
Mixformer-L [9]	83.9	88.9	83.1	-	-	-	70.1	79.9	76.3
Mixformer-22k [9]	83.1	88.1	81.6	70.7	80.0	67.8	69.2	78.7	74.7
AiATrack [15]	82.7	87.8	80.4	69.6	80.0	63.2	69.0	79.4	73.8
UTT [31]	79.7	-	77.0	67.2	76.3	60.5	64.6	-	67.2
CSWinTT [41]	81.9	86.7	79.5	69.4	78.9	65.4	66.2	75.2	70.9
STARK [51]	82.0	86.9	-	68.8	78.1	64.1	67.1	77.0	-
RTS [37]	81.6	86.0	79.4	-	-	-	69.7	76.2	73.7
Unicorn [50]	83.0	86.4	82.2	-	-	-	68.5	-	-
Table 3: Comparisons with state-of-the-art methods on two small-scale benchmarks. The top three metrics are highlighted with red, blue and green fonts.
SiamRPN++ [26]	PrDiMP [11]	TransT [5]	TrDiMP [43]	STARK [51]	AiATrack [15]	CSWinTT [41]	ToMP [32]	Mixformer-L [9]	ours
OTB	69.6	69.6	69.4	71.1	68.5	69.6	-	70.1	70.4	71.5
UAV123	61.3	68.0	69.1	67.5	69.1	70.6	70.5	66.9	68.7	68.5
Speed(fps)	35	30	50	35	42	38	12	24	18	71
and P of 83.6. Notably, as shown in Tab.2 comparisons, our model also surpasses all methods with search resolution greater than 300 in the TrackingNet benchmark.
GOT-10k. GOT-10k consists of a training set of 10,000 videos and a validation set of 180 videos. Moreover, the training and test sets do not overlap, requiring trackers to have a strong generalization ability for unseen data. Following the official evaluation rules, we only evaluate models trained on the GOT-10k training set. As reported in Tab.1 and Tab.2, similar to the results on TrackingNet, our method surpasses all methods with resolutions greater than 300 and smaller than 300 by using only a search resolution of 256, achieving a  $74.1\%$  of AO,  $84.6\%$  of SR0.5, and  $71.8\%$  of SR0.75. This demonstrates the strong generalization ability of our model to unseen data.
LaSOT. LaSOT is a comprehensive benchmark for long-term tracking tasks, including a test set comprising 280 videos. The average length of the videos in the test set is 2448 frames. As shown in Tab.1, our model achieves the highest performance in  $\mathrm{P}_{\text{norm}}$  and  $\mathrm{P}$ , with scores of  $82.0\%$  and  $76.4\%$  respectively, outperforming other methods. It only lags behind SimTrack by  $0.2\%$  in terms of AUC, mainly due to their use of ViT-L while we only use ViT-B. In comparison with methods using search resolutions greater than 300 in Tab.2, our model still demonstrates competitive performance on the Lasot dataset.
OTB. As shown in Tab.3, OTB is a classic dataset in the field of object tracking, consisting of 100 video sequences. RFGM demonstrates excellent performance on this dataset. This demonstrates that our method also exhibits generalization performance on the classical object tracking dataset.
UAV123. As shown in Tab.3, UAV123 is a dataset specifically designed for UAV (Unmanned Aerial Vehicle) target tracking, consisting of 123 video sequences with relatively small objects. RFGM achieves competitive performance on the dataset. Achieving a frame rate of 71 FPS, our approach significantly outperforms other methods in terms of speed.
Why does GR memory work? We conduct ablation experiments on different memory types, as shown in Tab.4. When using only one template, the performance is low because there is no template update to adapt to the appearance variations of the target. When using regular memory updating with a fixed interval, competitive performance is achieved on GOT-10k and TrackingNet, but the AUC on LaSOT is only 64.5. This is because LaSOT is a long-term benchmarks, and errors can accumulate in the memory when updating the template, resulting in performance degradation. By using the center score from the prediction head, errors in the memory can be reduced, but it cannot store the
Figure 4: Visualization of relevance attention. Taking one template as an example, white areas represent discarded regions, while the remaining areas represent the regions selected by relevance attention. Stages 1 to 3 indicate the progressive application of three relevance attention layers.
Table 4: Ablation study on our proposed GR memory. One template represents only the use of the first frame as the template. Memory represents replacing an old template with a new template in a fixed interval. Score memory represents replacing the template according to the center score of the prediction head. The best metrics are highlighted with red fonts.
Method	TrackingNet [34]	GOT-10k* [22]	LaSOT [13]
AUC	PNorm	P	AO	SR0.5	SR0.75	AUC	PNorm	P
One template	84.0	88.2	82.5	70.4	80.0	66.3	69	80.1	74.0
Memory	84.5	88.8	82.65	71.9	82.0	69.4	64.5	73.9	68.2
Score memory	84.6	88.9	82.7	74.3	84.0	70.8	69.7	81.5	76.2
GR memory	84.7	89.6	83.6	74.1	84.6	71.8	70.3	82.0	76.4
most representative features. In contrast, our proposed GR memory can store the most representative features throughout the entire video sequence and also reduce error accumulation in the memory, resulting in the highest performance. The visualization of GR memory updates is illustrated in Fig5.
Attention comparison. As reported in Tab.5, using only the attention mechanism from ViT cannot adaptively select the most suitable features from the reference for the current search frame. Therefore, its performance is lower than our proposed relevance attention. It is worth noting that the baseline here includes GR memory, where all features have already undergone a round of filtering using relevance attention within the GR memory. In other words, even within the filtered GR memory,
Figure 5: Visualization of GR memory updates. Over time t, the GR memory accumulates an increasing number of representative penguin features. In the second and third rows, white tokens represent discards, while the other tokens are retained in memory.
Table 5: Attention represents to use the original multi-head attention of ViT[12] in encoder. Relevance attention represents using relevance attention in our encoder. The best metrics are highlighted with red fonts.
Method	TrackingNet [34]	GOT-10k* [22]	LaSOT [13]	Macs	Parameter
AUC	PNorm	P	AO	SR0.5	SR0.75	AUC	PNorm	P
Attention	84.5	89.3	83.7	73.9	84.1	72.6	69.8	81.3	75.7	40.00G	92.12M
Relevance attention	84.7	89.6	83.6	74.1	84.6	71.8	70.3	82.0	76.4	37.66G	92.35M
The numbers 64, 192, 256, and 448 represent the Table 7: The MACs and parameters are measured token number in the memory with and without deformation attention
Table 6: Ablation experiments on memory size.
64	192	256	448
AUC	69.0	70.3	69.6	69.5
Pnorm	80.1	82.0	81.5	81.3
P	74.0	76.4	75.6	75.6
w/o	1 stage	2 stages	3 stages
MacS	40.00G	39.63G	38.85G	37.66G
Parameters	92.12M	92.20M	92.27M	92.35M
further performance improvement can be achieved by applying relevance attention. Additionally, relevance attention can reduce the number of computational parameters (40.00G VS 37.66G) while maintaining negligible parameter overhead(92.12M VS 92.35M). Tab.7 also presents the macs and parameters with or without relevance attention. The visualization of relevance attention is shown in Fig.4.
What is the best size of GR memory? As shown in Tab. 6, we conducted ablation experiments on the size of the memory. We found that the larger memory size does not lead to better performance. In fact, when the memory size exceeds 192 tokens, the performance starts to decline. This is because storing templates in the memory based on incorrect tracking results can lead to error accumulation. With a larger memory size, more errors are accumulated as the tracking progresses. Therefore, the optimal memory size for RFGM is 192 tokens.
5 Conlusion and Limitation
In this paper, we present a novel tracking framework tracking(RFGM), consisting of relevance attention and GR memory. Relevance attention adaptively selects the most suitable features for the current search region, allowing for adaptation to target appearance variations and environmental changes. Additionally, we construct a GR memory that utilizes relevance attention to select features from incoming templates. Through multiple rounds of updates, the GR memory stores the most representative features from the entire video sequence. Comprehensive experiments demonstrate that our method achieves state-of-the-art performance, and ablation experiments verify the effectiveness of the proposed GR memory and relevance attention. Due to the template updates, incorrect information will be stored into the memory, resulting in the accumulation of distractors over the process of the object tracking, we plan to further investigate methods to reduce the accumulation of errors in the memory.
Acknowledgement This work was supported by National Key RD Program of China (No.2020AAA0108301), National Natural Science Foundation of China (No.62072112 and No.62106051), Scientific and Technological innovation action plan of Shanghai Science and Technology Committee (No.22511102202), Fudan University Double First-class Construction Fund (No.XM03211178), and the Shanghai Pujiang Program (No.21PJ1400600).
[55] Yuechen Yu, Yilei Xiong, Weilin Huang, and Matthew R Scott. Deformable siamese attention networks for visual object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6728-6737, 2020.
[56] Zhipeng Zhang, Yihao Liu, Xiao Wang, Bing Li, and Weiming Hu. Learn to match: Automatic matching network design for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13339-13348, 2021.
[57] Zhipeng Zhang, Houwen Peng, Jianlong Fu, Bing Li, and Weiming Hu. Ocean: Object-aware anchor-free tracking. In European Conference on Computer Vision, pages 771-787. Springer, 2020.
[58] Zechu Zhou, Xinyu Zhou, Zhaoyu Chen, Pinxue Guo, Qian-Yu Liu, and Wenqiang Zhang. Memory network with pixel-level spatio-temporal learning for visual object tracking. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[59] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9308-9316, 2019.
[60] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.