SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Converting the energy function into the matrix form. Denote  $S_{k}^{col} = \sum_{l=1}^{n} S_{k,l}^{2}$ , and  $S_{l}^{row} = \sum_{k=1}^{m} S_{k,l}^{2}$ , the overall energy  $E$  can be expanded as
$$
\begin{array}{l} E = \sum_ {l = 1} ^ {m} \mathcal {S} _ {l} ^ {r o w} \left(\lambda \left(\frac {m}{H}\right) ^ {2} + 1\right) d _ {l} ^ {r o w 2} + \sum_ {k = 1} ^ {n} \mathcal {S} _ {k} ^ {c o l} \left(\lambda \left(\frac {n}{W}\right) ^ {2} + 1\right) d _ {k} ^ {c o l 2} \\ - \sum_ {l = 1} ^ {m} 2 \mathcal {S} _ {l} ^ {\text {r o w}} \frac {H}{\gamma m} d _ {l} ^ {\text {r o w}} - \sum_ {k = 1} ^ {n} 2 \mathcal {S} _ {k} ^ {\text {c o l}} \frac {W}{\gamma n} d _ {k} ^ {\text {c o l}} \tag {A5} \\ - \sum_ {l = 1} ^ {m} \sum_ {k = 1} ^ {n} 2 \lambda S _ {k, l} ^ {2} \frac {m n}{H W} d _ {l} ^ {r o w} d _ {k} ^ {c o l} + C \\ \end{array}
$$
where  $C$  is a constant term that can be ignored during minimization. The first, second and third lines of Eq. (A5) are consisting of squared terms, linear terms and cross terms respectively.
Thus, the overall energy  $E$  can be converted into the matrix form described in Eq. (A4) by fitting the above terms into  $P \in \mathbb{R}^{(m + n) \times (m + n)}$  and  $q \in \mathbb{R}^{m + n}$ , i.e.,
$$
P = \left( \begin{array}{c c} P ^ {r o w} & P ^ {c r o s s} \\ (P ^ {c r o s s}) ^ {\top} & P ^ {c o l} \end{array} \right), \quad q = \left( \begin{array}{c} q _ {r o w} \\ q _ {c o l} \end{array} \right) \tag {A6}
$$
where  $P^{row} \in \mathbb{R}^{n \times n}$  and  $P^{col} \in \mathbb{R}^{m \times m}$  are diagonal matrices accounting for the squared terms in  $E$ .  $P^{row}$  and  $P^{col}$  can be defined as
$$
P _ {k, l} ^ {r o w} = \left\{ \begin{array}{l} 2 \mathcal {S} _ {l} ^ {r o w} \left(\lambda \left(\frac {m}{H}\right) ^ {2} + 1\right), k = l \\ 0, \text {o t h e r s} \end{array} , \quad P _ {k, l} ^ {c o l} = \left\{ \begin{array}{l} 2 \mathcal {S} _ {k} ^ {c o l} \left(\lambda \left(\frac {n}{W}\right) ^ {2} + 1\right), k = l \\ 0, \text {o t h e r s} \end{array} . \right. \right. \tag {A7}
$$
$P^{cross} \in \mathbb{R}^{m \times n}$  is a matrix accounting for the cross terms in  $E$ , while  $q^{row} \in \mathbb{R}^n$  and  $q^{col} \in \mathbb{R}^m$  account for the linear terms.  $P^{cross}$ ,  $q^{row}$  and  $q^{col}$  can be defined as
$$
P _ {k, l} ^ {c r o s s} = - 2 \lambda S _ {k, l} ^ {2} \frac {m n}{H W}, \quad q _ {l} ^ {r o w} = - 2 S _ {l} ^ {r o w} \frac {H}{\gamma m}, \quad q _ {k} ^ {c o l} = - 2 S _ {k} ^ {c o l} \frac {W}{\gamma n}. \tag {A8}
$$
Converting the linear constraint into the matrix form. The linear constraint can fit in the matrix form described in Eq. (A1) by defining  $A \in \mathbb{R}^{2 \times (m + n)}$  and  $b \in \mathbb{R}^2$  as
$$
A _ {k, l} = \left\{ \begin{array}{l} 1, k = 1, l = 1, \dots , m \\ 1, k = 2, l = m + 1, \dots , n + n \\ 0, o t h e r s \end{array} , \quad b = (H, W) ^ {\top}. \right. \tag {A9}
$$
Proof of the convexity. When the matrix  $P$  in Eq. (A6) is positive semi-definite, the energy function in Eq. (A1) is convex and a global minimization can be found. Apparently,  $P$  is positive semi-definite since, for any  $d \in \mathbb{R}^{m + n}$ ,
$$
\begin{array}{l} d ^ {\top} P d = 2 \left(\sum_ {l = 1} ^ {m} \mathcal {S} _ {l} ^ {r o w} \left(\lambda \left(\frac {m}{H}\right) ^ {2} + 1\right) d _ {l} ^ {r o w 2} + \sum_ {k = 1} ^ {n} \mathcal {S} _ {k} ^ {c o l} \left(\lambda \left(\frac {n}{W}\right) ^ {2} + 1\right) d _ {k} ^ {c o l 2} \right. \\ \left. - \sum_ {l = 1} ^ {m} \sum_ {k = 1} ^ {n} 2 \lambda S _ {k, l} ^ {2} \frac {m n}{H W} d _ {l} ^ {r o w} d _ {k} ^ {c o l}\right) \tag {A10} \\ = 2 \left(\sum_ {l = 1} ^ {m} \mathcal {S} _ {l} ^ {r o w} d _ {l} ^ {r o w 2} + \sum_ {k = 1} ^ {n} \mathcal {S} _ {k} ^ {c o l} d _ {k} ^ {c o l} ^ {2} + \lambda \sum_ {l = 1} ^ {m} \sum_ {k = 1} ^ {n} S _ {k, l} ^ {2} \left(\frac {m}{H} d _ {l} ^ {r o w} - \frac {n}{W} d _ {k} ^ {c o l}\right) ^ {2}\right) \\ \geq 0 \\ \end{array}
$$
B Attribute-based Analysis on LaSOT
We further analyze the AUC gains brought by our method on the two baselines, i.e., OSTrack and TransT, based on the attributes of the videos on LaSOT [11]. The results are displayed in Fig. A1.
Figure A1: Attribute-based analysis of AUC gains on LaSOT [11]
As shown in Fig. A1, our method significantly improves the performance of both baselines under challenging scenarios with drastic movement (see Fast motion (FM), Viewpoint change (VC), etc) or
$^{3}$ FOC: Full Occlusion, OV: Out-of-view, LR: Low Resolution, FM: Fast Motion, VC: Viewpoint Change, IV: Illumination Variation, ARC: Aspect Ration Change, POC: Partial Occlusion, CM: Camera Motion, MB: Motion Blur, SV: Scale Variation, ROT: Rotation, BC: Background Clutter, DEF: Deformation
significant drift after some period of time (see Full Occlusion (FOC), Out-of-View (OV), etc), owning to the enlarged visual field. Performance on videos with low resolution (LR) is also considerably improved thanks to the zoom-in behavior. The deformation caused by non-uniform resizing may cause a little performance degradation when encountering background clutter (BC) or deformation (DEF).
C Comparison Between Non-uniform Resizing and Optimized Uniform Resizing Baseline
To demonstrate the superiority of our method, we additionally explore whether optimizing the crop hyper-parameters in the uniform resizing baseline can have a similar effect to our proposed non-uniform resizing, the results are shown in Tab. A1.
Despite the fact that our limited compute resources make us unable to optimize the above hyperparameters on large tracking datasets in a grid-search manner, our adequately powered experiments clearly show that the above optimizing is indeed hardly able to achieve similar effects to our resizing. Even for the baseline OSTrack-256[33] with the optimal crop hyper-parameters on LaSOT[11] (69.5 AUC), we experimentally find that it can only achieve AUC gains of  $+0.4$  and  $+0.6$  on  $\mathrm{LaSOT}_{ext}[12]$  and TNL2K[30] respectively while performing worse than the original OSTrack-256 on TrackingNet[23] (83.1 SUC vs 82.1 SUC). In other words, our method still outperforms the above optimal baseline on LaSOT ( $+0.7$  AUC),  $\mathrm{LaSOT}_{ext}$  ( $+2.7$  AUC), TNL2K ( $+1.6$  AUC), TrackingNet ( $+1.1$  SUC). Moreover, it is experimentally shown that the mismatch of the object size between the search and template images caused by increasing the search-image context factor has a relatively small effect on the final tracking accuracy for modern transformer trackers. That means the low resolution of the object is the major cause for the worse accuracy.
Table A1: Comparison results of our method and the uniform resizing baseline with its crop hyperparameters varied (i.e., setting  $c^w$  and  $c^h$  using two different choices as in Sec. 3.1, varying the search-image context factor, and varying the template-image size to alleviate the mismatch of the object size between the search and temple images). The experiments are based on OSTrack-256[33] and tested on LaSOT[11]. The context factor and size of search/template images are displayed as context factor/size. The choices of  $c^w$ ,  $c^h$ :  $\mathbf{1} c^w = b^w$ ,  $c^h = b^h$  and  $\mathbf{2} c^w = c^h = (b^w + b^h) / 2$
Original	cw, ch	Object Size Mismatched	Object Size Matched	Ours
Search	4/256	4/256	4.5/256	5/256	5.5/256	6/256	4.5/256	5/256	5.5/256	6/256	5/256
Template	2/128	2/128	2/128	2/128	2/128	2/128	2/114	2/102	2/93	2/85	2/128
cw, ch	1	2	1	1	1	1	1	1	1	1	1
AUC	69.1	68.5	69.0	69.5	68.2	67.5	69.1	68.8	68.0	67.4	70.2
D More Experimental Results on UAV123
In this section, we report the performance comparison on an additional benchmark, i.e., UAV123 [22]. UAV123 is an aerial tracking benchmark captured from low-altitude UAVs with more small objects and larger variations in bounding box size and aspect ratio. As shown in Tab. A2, our method achieves  $1.0\%$  and  $0.6\%$  relative gains on AUC for the OSTrack [33] and TransT [8] baselines, demonstrating that our method can improve the tracking performance in a wide range of tracking scenarios.
Table A2: Comparison with state-of-the-art trackers on UAV123[22]
Dataset	OSTrack-Zoom	OSTrack [33]	TransT-Zoom	TransT [8]	MixFormer-L [9]	ToMP [21]	DiMP [4]	SiamFC [3]
AUC (%)	69.3	68.3	69.7	69.1	69.5	69.0	65.3	37.7
E More Experimental Results with SwinTrack as the Baseline
We further apply our method to both the performance-oriented version SwinTrack-B (backbone: Swin Transformer-Base) and speed-oriented version SwinTrack-T (backbone: Swin Transformer-Tiny) of SwinTrack [19] to experimentally show the generalization capability of our proposed resizing method for a wide range of visual tracking algorithms. Note that we use the v1 version of SwinTrack (without motion token) as the baseline because only the code of v1 version is publicly available.
As shown in Tab. A3, our method can bring consistent improvement for both SwinTrack-T and SwinTrack-B. Particularly worth mentioning is that our SwinTrack-B-Zoom can even outperforms SwinTrack-B-384 with a much smaller input size.
Table A3: Further validation of applying our non-uniform resizing to both the performance-oriented version (backbone: Swin Transformer-Base) and speed-oriented version (backbone: Swin Transformer-Tiny) of SwinTrack [19]. Note that we use the v1 version of SwinTrack (without motion token) as the baseline because only the code of v1 version is publicly available.
Type	Trackers	Size	LaSOT[11]
AUC	P
Speed-oriented	SwinTrack-T-Zoom	224	68.5	72.9
SwinTrack-T[19]	224	66.7	70.6
Performance-oriented	SwinTrack-B-Zoom	224	70.5	75.4
SwinTrack-B[19]	224	69.6	74.1
SwinTrack-B-384[19]	384	70.2	75.3
F Analysis on Small Improvement Over TrackingNet
To find out the reason for the small improvement over TrackingNet [23], we additionally analyze the ratios of challenging scenarios appearing in the different tracking datasets (see Tab. A4), which clearly show that the test split of TrackingNet has a very different property from LaSOT [11],  $\mathrm{LaSOT}_{ext}$  [12], and TNL2K [30] with respect to the ratios of challenging scenarios. As shown in Fig. A1, our method can achieve significant performance gains under challenging scenarios with drastic movement (see Fast Motion (FM), Viewpoint Change (VC), etc.) or significant drift after some period of time (see Full Occlusion (FOC), Out-of-View (OV), etc.) owing to the enlarged visual field. However, the ratios of the challenging scenarios FOC, OV, FM in TrackingNet (VC is not labeled in TrackingNet) are significantly lower than in LaSOT,  $\mathrm{LaSOT}_{ext}$ , and TNL2K, which may be the reason for small improvement of our approach on TrackingNet.
Table A4: Ratios of challenging scenarios appearing in the different tracking datasets. Our method can achieve significant performance gains under challenging scenarios with drastic movement (e.g., Fast Motion (FM), Viewpoint Change (VC)) or significant drift after some period of time (e.g., Full Occlusion (FOC), Out-of-View (OV)) owning to the enlarged visual field. Note that VC is not labeled in TrackingNet.
Datasets	Ratios of Challenging Scenarios	AUC Gains
Significant Drift	Drastic Motion	OSTrack[33]	TransT[8]
FOC	OV	FM	VC
LaSOT[11]	42.1%	37.1%	18.9%	11.8%	+1.1%	+2.2%
LaSOText[12]	62.7%	21.3%	58.7%	39.3%	+3.1%	+2.0%
TNL2K[30]	13.9%	33.4%	24.0%	44.3%	+2.2%	+3.0%
TrackingNet[23]	4.7%	4.0%	10.0%	-	+0.1%	+0.4%
G Detailed Latency of Non-uniform Resizing
We provide the latency analysis in terms of time (ms) and MACs for solving QP and resizing images in Tab. A5. It can be seen that resizing images with interpolations in our method costs about half of the total time in spite of its parallel processing, while iteratively solving QP costs the rest of the total time despite its much lower FLOPs.
Table A5: Detailed latency of non-uniform resizing
Component	Time(ms)	MACs
Solving QP	0.89	0.16M
Resizing	0.69	1.12M
Total	1.58	1.28 M
H Confidence Interval for Ablation Experiments of Hyper-parameters
We calculate the  $90\%$  confidence intervals for all the experiments in Tab. 3 using bootstrap sampling. Each of the  $90\%$  confidence intervals is obtained using 10000 bootstrap samples, each sample with size 280 (same as the video number of LaSOT-test-split). The results are listed in Tab. A6.
Table A6: 90% confidence intervals for ablation experiments of hyper-parameters.
(a) Bandwidth β	(b) Control grid size
β	4	64	256	Size	8	16	32
Upper	70.95 (+1.87)	72.02 (+1.86)	71.54 (+1.90)	Upper	68.50 (+2.06)	72.02 (+1.86)	71.52 (+1.98)
Mean	69.08	70.16	69.64	Mean	66.44	70.16	69.54
Lower	67.23 (-1.85)	68.33 (-1.83)	67.80 (-1.84)	Lower	64.49 (-1.95)	68.33 (-1.83)	67.63 (-1.91)
(c) Zoom factor γ	(d) Balance factor λ
γ	1.25	1.5	1.75	λ	0	1	2
Upper	71.58 (+1.89)	72.02 (+1.86)	71.35 (+1.94)	Upper	71.69 (+1.89)	72.02 (+1.86)	71.43 (+1.87)
Mean	69.69	70.16	69.41	Mean	69.80	70.16	69.56
Lower	67.88 (-1.81)	68.33 (-1.83)	67.49 (-1.92)	Lower	67.93 (-1.87)	68.33 (-1.83)	67.71 (-1.85)
I More Visualization Results
We visualize the results of our method OSTrack-Zoom and its corresponding baseline on videos with challenging attributes to qualitatively show the superiority of our method in Fig. A2. Thanks to the zoom-in behavior and the enlarged visual field, our method can better handle the challenges such as full occlusion, fast motion, etc.
As shown in the left part of Fig. A2, at frame #150, both our method and the baseline lose track of the target since the target is fully occluded. At frame #157, our method quickly re-detects the target as the target re-appears. Note the target is significantly larger in the resized input image when using our resizing method, which indicates that the zoom-in behavior allowed by our method helps the re-detection of the target. At frame #187, our method successfully tracks the target whereas the baseline still drifts to a similar car.
As shown in the right part of Fig. A2, at frame #15, the target is moving really fast. As a result, in the resized input image generated by the baseline, part of the target is out of view which causes an inaccurate localization. In contrast, the resized input image generated by our method keeps the entire target visible thanks to the enlarged visual field, resulting in a more accurate tracking result. The situation is similar in frame #27, where the baseline tracks a fully-preserved distractor rather than
the half-cropped target, whereas our method successfully tracks the target since the target is intact. At frame #32, the baseline completely lose track of the target, while our method successfully tracks it.
Ground Truth OSTrack-Zoom OSTrack
Figure A2: Visualization of tracking results on videos with challenging attributes. Input to the tracker is displayed on the right side of each full image, where the image with blue box is the input image resized by our method and the image with green box is the input image resized by the baseline. (left) An example of video with full occlusion. The zoom-in behavior allows our method to quickly re-detect the target and avoid drifting. (right) An example of video with fast motion. The enlarged visual field allows our tracker to have the whole target visible without cropping, which facilitates localization and avoids drifting to similar objects when the target has drastic motion.
J Failure Cases
We visualize the failure cases of our method by comparing the tracking results of OSTrack-Zoom (blue) and OSTrack (green) in Fig. A3.
As shown in the left part of Fig. A3, at frame #1108, the target is out of view. As our method enlarges the visual field, apart from the distractor (orange ball) at the center of the input, an additional distractor (dark green ball at the top-right of the input image) comes into view, which causes the tracker to predict a large box between the two distractors. The localization error quickly accumulates to an overly enlarged visual field at frame #1117, since the visual field is partly determined by the previous prediction. At this time, a white ball that is very similar to the template is brought into view, causing our method to drift to the far-away white ball. Then, at frame #1130, when the target re-appears, our method cannot re-detect the target as our method drifts to a far-away distractor, which makes the real target out of the visual field of our tracker.
As shown in the right part of Fig. A3, at frame #94, both our method and the baseline successfully track the target. However, at frame #104, the enlarged visual field from our method brought additional contexts (orange robotic arm on the left) into view. The little fraction of orange robotic arm on the left makes the distractor look like a horizontally flipped template which has a small fraction of orange robotic arm on the right. As a result, our method drifts to the distractor. Then, at frame #114, our
method fails to track back to the squeezed real target since it is at the edge of the input image and partly invisible.
From the visualization of the failure cases, we find that, although an enlarged visual field promotes tracking performance in most cases, sometimes additional distractors or contexts brought by the fixed large visual field will cause drifting. Thus, it would be interesting to develop a method to dynamically adjust the visual field based on the surrounding of the target and its tracking history in the future.
Figure A3: Visualization of failure cases. Input to the tracker is displayed on the right side of each full image, where the image with blue box is the input image resized by our method and the image with green box is the input image resized by the baseline. Template image is visualized on the top-left corner of the full image. (left) A combination of out-of-view and fast motion causes an accumulated prediction error which leads to drifting. Localization with an enlarged visual field under some challenging scenarios may be difficult and its wrong prediction may produce less desirable results since the enlarged visual field brought more distractors. The localization error quickly accumulates to an overly enlarged visual field which then causes drifting. (right) Cluttered background with similar distractors causes drifting. The additional context (orange robotic arm) brought by the enlarged visual field makes the distractor on the left look like a horizontally flipped template, which causes drifting.