ZoomTrack: Target-aware Non-uniform Resizing for Efficient Visual Tracking
Authors: Yutong Kou
Jin Gao
Bing Li
Gang Wang
Weiming Hu
Yizheng Wang
Liang Li
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Visual tracking
non-uniform resizing
HVS-inspired processing

Abstract: Recently, the transformer has enabled the speed-oriented trackers to approach state-of-the-art (SOTA) performance with high-speed thanks to the smaller input size or the lighter feature extraction backbone, though they still substantially lag behind their corresponding performance-oriented versions. In this paper, we demonstrate that it is possible to narrow or even close this gap while achieving high tracking speed based on the smaller input size. To this end, we non-uniformly resize the cropped image to have a smaller input size while the resolution of the area where the target is more likely to appear is higher and vice versa. This enables us to solve the dilemma of attending to a larger visual field while retaining more raw information for the target despite a smaller input size. Our formulation for the non-uniform resizing can be efficiently solved through quadratic programming (QP) and naturally integrated into most of the crop-based local trackers. Comprehensive experiments on five challenging datasets based on two kinds  of transformer trackers, \ie, OSTrack and TransT, demonstrate consistent improvements over them. In particular, applying our method to the speed-oriented version of OSTrack even outperforms its performance-oriented counterpart by 0.6\% AUC on TNL2K, while running 50\% faster and saving over 55\% MACs. Codes and models are available at https://github.com/Kou-99/ZoomTrack.

ZoomTrack: Target-aware Non-uniform Resizing for Efficient Visual Tracking
Yutong Koul.2, Jin Gaol.2+ Bing Li Gang Wang"', Weiming Hu..2.3, Yizheng Wang', Liang Li State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA School of Artificial Intelligence, University of Chinese Academy Sciences School of Information Science and Technology, ShanghaiTech University 'Bejjing Institute of Basic Medical Sciences SPeople AI, Inc kouyutong2021@ia. ac.cn, en, {jin gao, bli, www.)@nnprrrr.... cn, 1 liang. 1i. brainfal iyun. com, com,
g. ..wang@formail... com, yzvang57Gsina.com
Abstract
Recently, the transformer has enabled the speed-oriented trackers to approach state- of-the-art (SOTA) performance with high-speed thanks to the smaller input size or the lighter feature extraction backbone, though they still substantially lag behind versions. this paper, we demonstrate that it is possible to narrow or even close this gap while achieving high tracking speed based on the smaller input size. To this end, we non-unioonn non-uniformly resize the cropped image to have smaller input size while the resolution of the area where the target more likely to appear is higher and vice versa. This enables us to solve the dilemma of attending to a larger visual field while retaining more information for the target despite a smaller input size. Our formulation for the non- uniform resizing can be efficiently solved through quadratic programming (QP) and naturally integrated into most of the crop-based local trackers. Comprehensive experiments five challenging datasets based i.e., OSTrack and Trans'T, demonstrate consistent improvements over them. In particular, applying our method to the speed-oriented version of OSTrack outperforms its cnrformance-orienter counterpart by 0.6% AUC on TNL2K, while rumning 50% faster and saving over 55% MACs. Codes and models available at /https://githun www. Kou- 99/ 2e//oomTTrack:
their corresponding performance-orient
kinds of transformer trackers,
Introduction
Comparison of trackers on TNL2K
In visual tracking, many efforts have been made to improve the discrimi- nation and localization ability of the tracking models, including deeper net- works fusion 8
57
OSTrack-Zoom
Performance- oriented
OSTrack-384
TwinTrack-B
56
Sim Trackc
55% AUC
55
18
34
transformer feature
AUC (%))
SwinTrack-T
SimTrrck-Bj
21
19
joint feature
OSTrack 256
extraction and interaction and so on. However, most of the re- cent tracking algorithms, including 9].
33 [
TransT-Zoom
Ours
OSTrcck [EOCV'22]
TransInMo*
40GMACs
SanTack (ECCV122
Swintrack (NeuriP5'22)
Speed- oneentd
TransinMo [[CA/22]
the transformer-based [33 7 19
[CVPR/21]
TransT
still follow the paradigm proposed by Bertinetto et al. B], in which small exemplar image cropped from
80
100
MACS (G)
Figure Our method consistently improves the OSTrack and TransT basclines with negligible computation.
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the first frame is used to locate the target within large search image cropped from one of the subsequent frames. The crop size, which determines the visual field size, is derived from a reference bounding box plus a margin for context. Both of the crops are uniformly resized to fixed sizes to facilitate the training and testing of the tracker.
Due to the fact that the complexity of the transformers scales quadratically with input size, many transformer trackers [33 7
propose both speed-oriented versions with the smaller input size or feature extraction backbone and performance-oriented versions with larger ones, Thanks to the enabled larger visual field size or stronger feature representation, the performance-oriene always outperform their speed-oriented counterparts in performance, though the tracking speed is severely reduced. For instance, an increased 1.5x input size for OSTrack [33] will cause doubled Muuliply-Accumin
versions
Operations (MACs) and nearly halved tracking speed. Thus, it is natural to pose the question: possible to narrow or even close this performance gap while achieving high tracking speed based on smaller input size?
Inspired by the human vision system (HVS), we propose to non-uniformly resize the attended visual field to have smaller input size for visual tracking in this paper. Human retina receives about 100 MB of visual input per second, while only MB of visual data is able to be sent to the central brain [36]. This is achieved by best utilizing the limited resource of the finite number of cones in HVS. More specifically, the density of cone photoreceptors is exponentially dropped with eccentricity (i.e. the deviation from the center of the retina), causing a more accurate central vision for precise recognition and less accurate peripheral vision for rough localization (3).. This enables us solve the dilemma of attending to larger visual field while retaining more raw information for the target area despite the smaller input size.
In our formulation for the non-uniform resizing, the area where the target more likely to ap- pear is magnified to retain more raw information with high resolution and facilitate precise recog- nition, whereas the area where the target is less likely to appear is shrunk yet preserved to facil- itate rough localization when encountering fast motion or tracking drift. The key to our method is to design an efficient and controllable resiz- ing module based on small controllable grid. On top of this grid, we define a zoom energy to explicitly control the scale of magnification for the important target area, rigid energy to avoid extreme deformation and a linear constraint to avoid cropping the source image during resizing. The important area is determined by the previ- ous tracking result as temporal prior. This formulation can be solved efficiently through quadratic programming (QP), whose solution is used to manipulate the controllable grid. Finally, the ideal non- non-uniform resizing is achieved by sampling according to the controllable grid (See Fig. D.
de rol
Source Image I
Target Image I'
d7
Sample
Control Grid
Fixed Grid
We achieve non-uniform resizing by sam- pling according to a manipulable control grid g. which is generated by solving for best ded and drow that minimize two energies that lead to con- trollable magnification and less deformation.
Figure 2:
Our method can be easily integrated into plenty of tracking algorithms. We select the popular hybrid Trn-Transformer tracker TransT [8] and one-stream Transformer tracker OSTrack (33) to verify the efficiency and effectiveness of our method. Comprehensive experiments are conducted on five large-scale benchmarks, including GOT-10k [16), LaSOT [TT) LaSOT ezt [12), TNL2k (30), Track- ingNet [23). We observe consistent improvements over the baselines with negligible computational overhead. In particular, we improve the
version of OSTrack to achieve 73.5% AO, 50.5% AUC and 56.5% AUC GOT-10k, LASOT cat and TNL.2k respectively, which are on par with the performance-oriente (see Fig. [ In other words, the performance gap between speed-oriented and performance-oriented trackers can be narrowed or even closed through our method.
counterpart while saving over 55% MACs and running 50% faster
In summary, our contributions are follows: (i) We propose ZoomTrack an efficient non-uniform resizing method for visual tracking that bridge the gap between speed-oriented and performance- oriented trackers with negligible computation; (ii) We formulate the non-uniform resizing as an explicitly controlled magnification of important areas with restriction on extreme deformation,

---

enabling an HVS- EVS-iisspredd data processing with limited resource; (iii) Extensive experiments based on two baseline trackers on multiple benchmarks show that our method achieves consistent performance gains across different network architectures.
2 Related Work
Efficient Visual Tracking. A lot of real- world applications require tracking algorithms to have high speed. Consequently, many efforts have been made to improve the efficiency of visual tracking. Yan et al. 32 use NAS to search for lightweight network architecture for visual tracking. Borsuk et al. [6] design a novel way to incorporate temporal information with only single learnable parameter which achieves higher accuracy at a higher speed. Blatter et al. [S] design an efficient transformer layer and achieves real-time tracking on CPU. Shen et al. [26] propose a distilled tracking framework to learn small and fast trackers from heavy and slow trackers. Some other works [0] explore the quickly advancing field of vision transformer pre-training [5 29] and architecture designing [13] to improve both tracking accuracy and efficiency. Previous methods focus on designing either better
network 32 5 6 10 or a new training framework [26] to improve the efficiency of trackers, whereas our work focuses on the non-uniform resizing of the input for the sake of efficiency. Our approach is more general and orthogonal to them.
Non-uniform Resizing for Vision Tasks. Image resizing is a common image processing operation. Yet, the standard uniform resizing is not always satisfactory many applications. Avidan and Shamir [1] resize image by repeatedly carving or inserting scams that are determined by the saliency of image regions. Panozzo et al. (24) use axis-aligned deformation generated from an automatic or human-appointed saliency map to achieve content-aware image resizing. Recasens et al. [25] adaptively resize the image based on a saliency map generated by CNN for image recognition. Zheng et al. [37 learn attention maps to guide the sampling of the image to highlight details for image recognition. Thavamani et al. (27] use the sampling function of [25] to re-sample the input image based on temporal and bataset-level prior for autonomous navigation. Bejnordi et al. [ learn a localization network to magnify salient regions from the unmediated supervision of [37] for video object detection. Thavamani al.
(8 propose an efficient and differentiable warp inversion that allows for mapping labels to the warped image to enable the end-to-end training of dense prediction tasks like semantic segmentation. Existing methods either use time-consuming feature extractors to generate saliency 24 25 37 or use heuristic sampling functions that cannot explicitly control the extent of the magnification, which may cause unsatisfactory magnification or extreme deformation 27 2 28 In contrast, our approach directly generates the saliency map or important area using the temporal prior in tracking and proposes a novel sampling method that can achieve controllable magnification without causing extreme deformation.
3 Background and Analysis
3.1 Revisiting Resizing in Deep Tracking
Before introducing our method, we first briefly revisit the resizing operation in the deep tracking pipeline. Given the first frame and its target location, visual trackers aim locate the target in the subsequent frames using a tracking model (e (z,x), where z is the template patch from the first frame with the target in the center and x is the search patch from one of the subsequent frames. Both and x are generated following a crop- rhen- manner. We next detail the cropping and resizing operations.
Image crop generation. Both the template and search images are cropped based on a reference bounding box b, which can be the ground truth annotation for the template image or the previous tracking result for the search image. size can be calculated
Denote reference box as (ber, bev b", b*), then the crop
W H
/ (bt (f
(bP) (f 1) x c*)
where ʃ is the context factor controlling the amount of context to be included in the visual field, and c and dt denote unit context amount which is solely related to b0dh and b/. Typical choice of unit context amount e = b"h d 11h
[3 31 9. or e = n² = (b" - b2 (8 19). A higher
context factor means larger visual field and vice versa. Usually, the search image has a larger
3

---

70
3.5
MAGS LaSOT AUC
LaSOT AUC (%)
69.7
70.2
40
30
MACs (G)
LaSOT AUC (%)
68.5
69
3
Avg Target Res
(10' Pixels)
2.5
69.2
20
68.7
10
67
LaSOT AUC
Avg Target Resolution
68.2
66.5
1.5
258/4.5 320/5 352/5.5 384/6
4.5
5.5 Context Factor
6
Search Size / Context Factor
Figure 3: Experiments on LaSOT [T] based on OSTrack 33 show that: (left) Thanks to the larger attended visual field, simultaneously increasing search size and context factor leads to consistent performance improvement at the cost of heavy computational burden; (right) Increasing the attended visual field by enlarging context factor while fixing the input size to limit the available resources leads to degraded performance due to the decreased target area resolution.
context factor for a larger visual field, and the template image has a small context factor providing minimal necessary context. The image crop I is obtained by cropping the centered at (bez, bo)) with width W and height H. Areas outside the image are padded with a constant value. Finally, box b is transformed to new reference bounding box (r yoy r"" r²) on the image crop I.
Image crop resizing. To facilitate the batched training and later online tracking, the image crop has to be resized to fixed size. Given an image crop I with width W and height H, a fix-sized image patch I' with width w and height is obtained by resizing I. Specifically, h x w pixels I' T'(x', y²) are sampled from [(x, y) according to mapping T R + RR,, i.e.,
I' 1/) = (T (a', v'))
(2)
Note that T does not necessarily map integer index to integer index. Thus, some pixels in I' may be sampled from I according to the non-in oon-integer indices. This can be realized by bilinear interpolating use uniform mapping Tunsform resize 7]
from nearby pixels in I. Current methods [33 9 19
I. Tuniform is defined as Jun (z', v/)
where sf w/W, 8 h/H called resizing factors and indicate the scaling of the image. When applying uniform resizing, the resizing factor is the same wherever on the image crop, which means the same amount of amplification or shrinkage is applied to the different areas of the entire image.
3.2 Solving the Dilemma Between Visual Field and Target Area Resolution
Although the crop-then-resize paradigm is applied in most of the popular deep trackers, the fixed input search size and context factor vary significantly across different algorithms. Generally, the
performance be improved by using larger input size at the cost of decreasing tracking speed [19 33.. To understand the key factors behind this, we conduct a series of experiments on LaSOT [ simultaneously increase search size and context factor, which means the target resolution is roughly fixed and the tracker attends to a larger visual field. As shown in the left of Fig. 3 the AUC on LaSOT increases along with the input search size thanks to the larger attended visual field. Note that increasing the search size (256 - 384) leads to almost doubled computational overhead (21.5G MACs - 41.5G MACs). Next, limit the available resources by fixing the input size while increasing the attended visual field size. As shown in the right of Fig. the AUC on LaSOT first increases and then decreases as the average target resolution keeps decreasing. This result demonstrates that the benefit of an enlarged visual field is gradually wiped out by the decrease target resolution when the computational cost is fixed.
based OSTrack [33]. First,
Inspired by the data processing with limited resources in HVS, we believe the above dilemma of attending to larger visual field while retaining more raw information for the target can be solved by replacing uniform resizing with non-uniform resizing. In visual tracking, the previous tracking result serve strong temporal prior for the current target location. Based on this prior, can determine the area where the target is likely to appear and magnify it to retain more raw information with high resolution. On the contrary, areas where the target is less likely appear is shrunk avoid decreasing the visual field when the input search size is fixed with limited resources. Notably, the
"The average resolution is calculated by averaging the sizes of the ground truth bounding boxes on the thep-then-oe resized search patches at frames that do not have full-occlusion of view attributes.

---

magnification and shrinkage should not dramatically change the shape and aspect ratio of regions on the image to facilitate the robust learning of the appearance model. Based on the above analysis, we propose some guidelines for our non-uniform resizing, i.e., G1: Magnify the area where the target is most likely to appear; G2: Avoid extreme deformation; G3: Avoid cropping the original image I so that the original visual field is also retained. We detail how to incorporate these guidelines into our non-uniform resizing in the following section.
Methods
As discussed in Sec. 3.1 the resizing process from the source image [(z, v) Rx to the target image ['(x', yy) R "xxxxxx is sampling operation controlled by mapping T (z', 1/) - (z. y). Our aim is to find the mapping T that best follows the guidelines (G1-G3), given a temporal prior box (y*2, rf r" r²) that indicates the most possible location of the target. We first restrict the domain of the mapping to a few points and acquire a grid representation of the mapping T. Then, formulate the guidelines as a QP problem based the grid point intervals. By solving the QP problem via a standard solver (20), we can efficiently manipulate the controllable grid point intervals and achieve the ideal non-uniform resizing by sampling according to the final grid representation.
4.1 Grid Representation for Non-uniform Resizing
Recall the sampling operation in Eq. (2) pixel (a', y²) on the target image I' E R is sampled from location (Ir y) on the source image I R R x3, Thus, at least hxw pairs of (a', w²) (1r, y) are needed for resizing. We use the exact h x w pairs to represent the mapping T, which can be defined as grid 9 E hxxxx2 where G[y = (z,,), y), x' 1,-.,, w. 1, ..: h, is the location on Z that the target image pixel (1'xxx,) .3) should be sampled from.
Controllable grid. The grid G is extremely dense grid with /a x grid points (equivalent to the resolution of the target image Z'), which makes it computationally inefficient to be directly generated. To solve this issue, we define a small-sized controllable grid g E (m(+1)xx(++)) with m + rows and n + columns (m << h, << w) to estimate G, where
sbild =
w i
i = 0, j j= 0....,
(3)
Intuitively, gf is the sample location on the source image for the target image pixel (,,, h in T'. Once g is determined, G can be easily obtained through bilinear interpolation on g.
Axis-alignment constraint. To avoid extreme deformation (G3) and reduce computational overhead, we add an axis-aligned constraint 27 to the grid Specifically, grid points in the same row j have the same y-axis coordinate Vy. and grid points in the same column have the same x-axis coordinate Ie- That is to say g]b][(]] (I., Vj).
O..... n, j O..... m. The reason why we such an axis-aligned constraint is that we need to keep the axis-alignment of the bounding box (i.e., the four sides of the bounding box are parallel to the image boundary) after resizing, which is proven to be beneficial to the learning of object localization 27).
Grid representation. The source image is split into small rectangular patches p(k: 1). 1,.... ..., m by grid points. The top-left corner of p(k,() is gil 1]]k 1] and its bottom- right corner is g [K]. The width and height of the patch p(k, 1) are denoted as the horizontal grid point interval deal and the vertical grid point interval d'ow. deal and drow can be calculated as dgat - 2k-1, di yt - 9t-1. The axis-aligned controllable grid g can be generated from grid point intervals drow deol based on
9 s
deol
di
drow
where drow =dol = 0 for the sake of simplicity.
Non-uniform resizing. According to Eq. 3}, the resizing process can be visualized using two grids (see Fig.
2 manipulatable grid on the source image and a fixed grid on the target image [ = (" f j)) g and ge split the source and target images into rectangular patches p(k: 1) and (k l) respectively. Target image pixel at ge [i]]] is sampled from g[ā][i] on the source image. This
5

---

process can be intuitively understood as resizing p(k, to (t When the grid point intervals of are manipulated, the grid point intervals of g, remain unchanged. Thus, by reducing/ increasing the grid intervals of g. the corresponding region on the target image is amplified/ shrunk. In this way, non-uniform resizing operation that scales different image areas with different ratios can be obtained by dynamically manipulating the grid point intervals.
4.2
QP Formulation Based on Grid Representation
Given a reference bounding box r = (, ,,,, y" ,²)), we aim to dynamically manipulate the grid point intervals to best follow the guidelines (G1-G3) proposed in Sec. 3.2
Importance score for the target area. According to GI, areas, where the target most likely to appear, should be magnified. Following this guideline, we first define the importance score for the target area on the source image. An area has high importance the target is more likely to appear in this area. As r is strong temporal prior for the current target location, we use Gaussian function (rr, y) centered at (re2, ,ro) as the importance score function, i.e.,
G v) exp (-2
( 2
(a pzz
Hy) o
o
where M4 p ,4y , 0g / x rm . / rh B a hyper- parammer controlling the bandwidth of the Gaussian function. Since we denote the rectangular area enclosed by the grid point intervals diow and de as patch p(k: l), its importance score Sk. can be determined by the value of G(x, y) at the center of patch p(k, 1), i.e.
((( k (
W n2
H m
Sx
1
tc.
(6)
2
where is a small constant to prevent extreme deformation and ensure stable computation. Here we use the patch center of a uniformly initialized grid (dea W and drow = (). By doing so, Sk, is irrelevant of dea and dj which is crucial for the QP formulation in the following.
Zoom energy. Following G1, we design quadratic energy Ezeem magnify the area where the target is most likely to appear. To achieve controllable magnification, we define a zoom factor controlling the amount of magnification. we want magnify some area by y, the distances between sampling grid points should shrink by 4. Ezoom forces d% and diow to be close to the grid point intervals under uniform magnification by y, i.e., for drow and for deal In detail, The zoom energy is defined as
m
(( (d
H Y m
(Coo
1 1W 2
E Ezoom
Sk.1
+
n
Rigid energy. To avoid extreme deformation (G2), define quadratic energy Erigid to restrict the aspect ratio change of patch p(k,(), !), which has width dx and height drom When being uniformly resized, p(k, I) should have width W and height H Thus, the patch p(k:, Z) is horizontally stretched by # and vertically stretched by # ddrow, , To keep the aspect ratio mostly unchanged for the important area, the horizontal stretch should be similar to the vertical stretch. Therefore, the rigid energy can be defined
Erigid
SE.
dj
TOW
T W
ccol
2
(8)
H
1=1 ==k=
Linear constraint to avoid cropping the source image. According to G3, the resizing should not crop the source image that the original visual field is also retained. That is to say, the grid should completely cover the entire source image. This requirement can be interpreted as a simple linear constraint that forces the sum of vertical grid point intervals drow to be H and the sum of horizontal grid point intervals dool to be W: Em drow H, ンk=1 de W.

---

QP formulation. Combining the zoom energy, the rigid energy and the linear constraint, we can formulate the grid manipulation the following minimization task
minimize drow deat
E Ezoo + Erigid
1
subject to
Σ 1=1
di
dro
H
Σ dol W
where À is a hyper- parammeeeeeter to balance the two energies. This convex objective function can be efficiently solved by a standard QP solver [20] (see Appendix for more detailed derivations).
4.3 Integration with Common Trackers
Reference box generation. Our non-uniform resizing method needs a temporal prior reference box (rer, rey r", r²) on the cropped source image I. During testing, directly generated from the previous frame tracking result (see Sec. Scc.B.1). During training, only need to generate r" and rh from the ground truth width g and height g" as the center of the temporal prior reference box is fixed at y W/2 and you H/2. In detail, with probability q
smaller jitter -Js- and with probability q we use larger jitter j jj. r and nh are calculated as r edd g", prrhh ===="" g", where J jh ~ (0)
Label mapping. The classification loss of the network (z x) is calculated on the target image I'. Therefore, the classification labels on I' should be generated from the ground truth bounding box on I'. The common practice is use normalized ground truth bounding box for the reason that the normalized coordinates remain the same on I and I'' under uniform resizing. However, this does not hold true for our non- non-unine unn-uniform resizing. Therefore, we need to map the ground truth bounding box from I to I' via the grid G. Specifically, we map the top-left corner and bottom-right corner of the bounding box as follows. Given a point on the source image (z, y). we first find two grid points on G that are closest to (z, y). Then, the corresponding point (x', y') can be obtained by bilinear interpolation between the indices of these two points.
Reverse box mapping. The bounding box predicted by the network (, x) is on I' and needed to be mapped back I the final tracking result. We map the top-left corner and bottom-right corner of the predicted bounding box from I' to I in reversed process of label mapping. Specifically, given a point (a', on I', we first find the two grid points of & whose indices are closest (a', y'). and then the corresponding point (x. y) can be obtained by bilinear interpolation between the values of these points. To minimize the impact of the reverse mapping, we directly calculate the regression loss on I instead of on I'.
5 Experiments
5.1 Implementation Details
We apply our method to both one-stream tracker OSTrack [33 (denoted as "eTrack-Zoomn and -ra-Transformer tracker TransT [ (denoted as TransT-Zoom). The proposed non-uniform resizing module is applied in both the training and testing stages on the search image. We do not apply the proposed non-uniform resizing on the template image because it degrades performance (see Sec. 5.3). The search patch size (256 x 256) is the same the baseline methods, whereas the context factor is enlarged from to We use a controllable grid g with shape 17 x 17(m = n 16). The importance score map is generated by Gaussian function with bandwidth 3 = 64. We set magnification factor in Eq. ( We use smaller jitter js = 0.1 with probability 0.8 and larger jitter the same of parameters for both OSTrack -Zomm and TransT-Zoon to demonstrate the generalization capability of our method. Except for the resizing operation, all the other training and testing protocols follow the corresponding baselines. The trackers are trained on four NVIDIA V100 GPUs. The inference speed and MACs are evaluated on a platform with Intel 17-1 700 CPU and NVIDIA V100 GPU.
7 = 1.5 and
0.5 with probability 0.2. Notably,
5.2 State-of-the-art Comparison
We compare our method with the state-of-the-ard trackers on five challenging large-scale datasets: GOT- 10k (16), LaSOT [T]] LaSOT ext [1). TNL2K [30] and TrackingNet [23).
7

---

Table 1: sutt--of-the-are art Comparison on GOT-IOK, LaSOT, LaSOT ext, TNL2K and TrackingNet.
GOT-1Ok
LaSOT
LaSOTest TNL2K TrackingNet
Tracker
Size
MACs (G) FPS
AO
SRas SRa.5
AUC
P AUC P AUC P SUC
P
Ooorack-Zoom
256 73.5 83.6
70.0
70.2
76.2 50.5 57.4 56.5
57.3
83.2
82.2
21.5
100
Baseline
&OOrs
OSTrack-256 33
256 71.0
80.4
68.2
69.1
75.2 47.4 53.3 54.3
83.1
82.0
21.5
119
TransT-Zoom
255
67.5
77.6
61.3
67.1
71.6
46.8 52.9
53.7
62.3 81.8
80.2
19.2
45
TransT
255 67.1
76.8
60.9
64.9
69.0
44.8
52.5
50.7
51.7
81.4
80.3
19.2
48
SwinTrack- T 10
71.3
81.9
64.5
67.2
70.8
47.6 53,9 53.0
53.2
81.1
78.4
6.4
98
SimTrack-BBI 7
224
68.6
78.9
62.4
69.3
54.8 53.8
82.3
25.0
40
Speed-oriented
MixFormer-22X
320 70.7
80.0
67.8
69.2 74.7
83.1
81.6
23.0
25
ToMP-101
288
68.5 73.5 45.9
81.5 78.9
20
TransInMo [4]
255
65.7 70.7
52.0 52.7 81.7
16.9
34
Stark-ST101
320
68.8
78.1
64.1
67.1
82.0
28.0
32
AutoMatch 35
255
65.2
76.6
54.3
58.3
59.9
37.6
43.0
47.2
43.5 76.0
72.6
50
DiMP
61.1
71.7
49.2
56.9
56.7
39.2
45.1
44.7
43.4
74.0
68.7
5.4
40
SiamRPN++
255
51.7
61.6
32.5
49.6
49.1 34.0 39.6
41.3
41.2
73.3
69.4
7.8
35
Performance
OSTrack-384
33
384
73.7
83.2
70.8
71.1
77.6
50.5 57.6
55.9
83.9
83.2
48.3
61
-oriented
SwinTrack-BIP
384
72.4
80.5
67.8
71.3
76.5
49.1
55.6 55.9
57.1
84.0
82.8
69.7
45
SimTrack-1..P 7
224 69.8
78.8
66.0
70.5
-
55.6 55.7
83.4
95.4
MixForme-LL
9
70.1
76.3
83.9
83.1
127.8
18
GOT-10k [16] provides training and test splits with zero-overlapped object classes. Trackers required to be only trained on GOT-10k training split and then evaluated on the test split to examine their generalization capability. We follow this protocol and report our results in Tab. []] Our OSTrack- Zoom achieves 73.5% AO which surpasses the baseline OSTrack by 2.5%. And our TransT-Zoom achieves 67.5% AO which surpasses the baseline TransT by 0.4%. Compared with previous speed- oriented trackers, our OSTrack-Zoom improves all of the three metrics in GOT-10k by a large margin, proving that our method has good generalization capability.
LaSOT [ is large-scale dataset with 280 long-term video sequences. As shown in Tab. [ our method improves the AUC of OSTrack and TransT by 1.1% and 2.2% respectively. Our method OSTrack-Zoom have the best performance among speed-oriented trackers on LaSOT, indicating that method is well-suited for complex object motion in long-term visual tracking task.
LaSOT exrt 12 is an extension of LaSOT with 150 additional videos containing objects from 15 novel classes outside of LaSOT. In Tab. [ our method achieves 3. 3.1% and 2.0% relative gains on AUC metric for OSTrack and TransT baselines respectively. Our method OSTrack-Zoom achieves promising 50.5% AUC and 57.4% Precision, outperforming other methods by large margin.
TNL2K [30 is newly proposed dataset with new challenging factors like adversarial samples, modality switch, etc. As reported in Tab. [I our method improves OSTrack and TransT by 2.2% and 3.0% on AUC. Our OSTrack-Zoom sets the new state-of-the-art performance on TNL2K with 56.5% AUC while running at 100 fps.
TrackingNet (3) is large- scale short-term tracking benchmark. Our method boosts the performance of OSTrack and TransT by 0.1% and 0.4% SUC. The reason for the relatively small improvements is that, contrary other datasets, TrackingNet has fewer sequences with scenarios that can benefit from an enlarged visual field, e.g. scenarios with drastic movement or significant drift after some period of time. Please refer to Appendix [P for detailed analysis.
Comparison with performance-oriente trackers. We additionally compare our method with the performance-oriented versions of the SOTA trackers. On the one hand, on GOT-10k, LaSOT eart. TNL2K whose test splits object classes have no or few overlap with training data, our OSTrack-Zooe ZZom on-par with the SOTA tracker OSTrack-384 while consuming only 44.5% MACs of the latter and running 50% faster. On the other hand, on LaSOT and TrackingNet whose test splits have many object classes appearing in training data, our OSTrack Zoom is slightly inferior to some SOTA trackers. A possible reason is that these SOTA trackers with heavier networks fit the training data better, which is
8

---

Table 2: Comparison of accuracy, latency and target size
Table 3: Ablation of on LaSOT [ with OSTrack B3
using different resizing methods with OSTrack B3
(a) Bandwidth 8
(b) Control grid size
TNL2K 38
LaSOT
Target Size (10*)
Latency (ms)
Method Size
11
FPS
8 4 64 256
Size 8
16 32
AUC 69.1 70.2 69.6
AUC 66.4 70.2 69.5
AUC
AUC
avgt
stdl
resize
net
1 Uniferm 256
55.5
55.4
69.5
75.2
2.7
24.8
0.04
8.40
119
(c) Zoom factor Ty
(d) Balance factor
03 Uniform
320
56.1
56.8
69.9
76.2
4.2
38.8
0.04
10.92
92
3 FOVEA
256
54.2
54.5
67.6
73.3
4.4
40.8
3.37
8.40
85
1.25 1.5 1.75
X 0 2
Ours 256
56.5
57.3
70.2
76.2
4.9
25.0
1.58
8.40
100
AUC 69.7 70.2 69.4
AUC 69.8 70.2 69.6
Table 4: Ablation on the effect of applying non-uniform resizing in training and testing stages. Results are reported in AUC (%) metric.
Table Ablation on the effect of apply- ing non-uniform resizing on the template and search images with OSTrack [33).
Non-uniform resizing
OSTrack
Trans
Non-uniform Resizing
LaSOr
#
Testing
Training
LaSOT 11
TNL2K 3D
LaSOT [
TNL2K 3D
Template
Seurch
AUC
P
69.1
54.3
64.9
50.7
64.8
71.0
69.1
55.9
66.3
53.6
76.2
70.2
56.5
67.1
53.7
69.6
75.5
beneficial track objects with classes that have been seen during training. Nevertheless, the gap between speed-oriented and performance-oriented trackers is narrowed using our method.
5.3 Ablation Experiment
Comparison with other resizing methods. As shown in Tab. 2 we compare our non-uniform resizing method (o) with uniform resizing with small/large input sizes (Q/2) and FOVEA [27] non-uniform resizing (8). On both TNL2K and LaSOT, our method outperforms other methods. Notably, while applying FOVEA causes a significantly degraded performance (Qvs. @), our method achieve large performance gains (Wvs.(), !). To understand the reason behind this, we calculate the average target size (avg) and the standard deviation of the target size (std), which represent the degree of the targets' scale change, LaSOT. Compared with uniform resizing (Drs.4), our method achieves 1.342 x magnification on average target size, while the standard deviation almost keeps the same. The magnification is close to the desired 72 1.5?. showing that method achieves controllable magnification. In contrast to FOVEA (@vs. D), which severely reduces the tracking performance, our standard deviation is almost unchanged, demonstrating that our method can avoid extreme deformation that severely changes the scale of the target.
Computational overhead analysis. Our resizing module runs purely CPU. The latency for resizing an image is 1.58 ms on an Intel i7-1 1700 CPU. Compared with uniform resizing (Dvs.@) method introduces additional 1. 1.54 ms latency during resizing, causing a slightly slower speed. However, our performance is better than the uniform resizing counterparts. Especially when compared with uniform resizing with larger input size (@ys. +), method reaches higher performance while running faster, demonstrating the superiority of using non-uniform resizing in visual tracking. Our method is also faster than FOVEA resizing while having better performance. Moreover, the MACS for our resizing an image is 1.28 M MACS (10%), which is negligible compared to the network inference which is usually tens of G MACs (10%).
Resizing in different stages. It is worth noting that our method can be directly applied to off-the- shell tracking models, i.e., only applying non-uniform resizing method testing time. Results LaSOT IT] and TNL2K [00 are reported in Tab, 4 Jusing the AUC (%) metric. As shown in Tab, 4 directly applying our method to the off-the-shelf trackers only in testing can already improve the tracking accuracy (@vs. @). Further applying our resizing in training can also boost the performance, owning to the aligned training and testing protocols (Zvs.3).
Resizing template and search images separately. We also investigate the effects of applying the non-uniform resizing on the template and search images separately. As shown in Tab. 5 eiiher applying non-uniform resizing on the template image alone or on both the template and search images performs worse than applying it only on the search image. We think the reason is that the template

---

Figure 4: Visualization of different resizing. For each set of images, from left to right: ours, uniform resizing, FOVEA [27] resizing. The targets are marked in red box. (left) Our method can magnify the target without drastically changing the appearance of the target. (right) Our method can magnify the target without significantly changing the aspect ratio of the target.
image must have a relatively small context factor to contain minimal necessary context, which makes the whole template region important for the tracking network to locate the target. Therefore, the tracker cannot benefit from non-uniform resizing of the template image.
More ablations. First, we study the impact of the bandwidth 3 used in the importance function. A smaller means less area is considered to be important and vice versa. As shown in Tab. 3a the ideal value for 9 is 64. Both smaller bandwidth (4) and larger bandwidth (256) cause reduced performance, showing the importance of this factor. Second, we also analyze the impact of the controllable grid. As n 16 is the best size for this grid. A smaller grid size cause drastic drop in AUC, showing that too small grid size will cause inaccurate resizing that damages the tracking performance. larger grid size also degrades performance, for which we believe the reason that large grid size will lead to an over-reliance on the important area generation based on the temporal prior, which is however not always correct. Third, we further look into the impact of different zoom factors As shown in Tab. 3c 1.5 is the best value for y. A smaller "y will degrade performance due smaller target resolution. Meanwhile, larger 7 is also harmful the performance as larger magnification will cause trouble when trying to rediscover the lost target. Finally, we study the impact of Erigid by changing the balance factor A. As shown in Tab, 3d] setting À reaches the best performance. When À = 0, the performance drops, demonstrating the effectiveness of the proposed Eeeergy A larger À = also degrades the performance, indicating that too much constraint is harmful to the performance.
shown in Tab. [b]
Visualization. We visualize our proposed non-uniform resizing, uniform resizing and FOVEA [27] resizing in Fig. 4 Compared with uniform resizing, our method can improve the resolution of the target to retain more raw information. Compared with FOVEA, our method can better preserve the aspect ratio and the appearance of the target.
6 Conclusion
In this paper, we presented a novel non-uniform resizing method for visual tracking, which efficiently improves the tracking performance by simultaneously attending to a larger visual field and retaining more raw information with high resolution. Inspired by the HVS data processing with limited resources, method formulates the non-uniform resizing explicitly controlled magnification of important areas with restriction on extreme deformation. Extensive experiments have demonstrated that our method can bridge the gap between speed-oriented and performance-oriented trackers with negligible computational overhead.
Limitations, One limitation of our method is that we need a fixed context factor to determine the size of the visual field. would interesting to develop a method to dynamically adjust the size of the visual field according to the tracking trajectory appearance of the target in future work.
Acknowledgements. The authors would like to thank the anonymous reviewers for their valuable com- ments and suggestions. This work supported in part by the National Key R&D Program of China (Grant No. 2020AAA00068000) the Natural Science Foundation of China (Grant No. U22B2056. 61972394, 62036011, 1, 62192782, 61721004, U2033210), the Beijing Natural Science Foundation (Grant No. I.223003, JQ22014, 4234087), the Major Projects of Guangdong Education Department for Foundation Research and Applied Research (Grant No. 2017KZDXM081. 201 8KZDXM066), the Guangdong Provincial University Innovation Team Project (Grant No. 2020XCX)D045), Jin Gao and Bing Li were also supported in part by the Youth Innovation Promotion Association, CAS.
10

---

References
[1] Shai Avidan and Ariel Shamir. Seam carving for content-aware image resizing. In SIGGRAPH. 2007.
[2] Babak Ehteshami Bejnordi, Amirhossein Habibian, Fatih Porikli, and Amir Ghodrati. SALISA: Saliency- based input sampling for efficient video object detection. In ECCV, 2022.
[3] Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip S. Torr. Fully- convolutional siamese networks for object tracking. In ECCV Workshops, 2016.
[4] Goutam Bhat, Martin Danelljan, Lue Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In ICCV, 2019.
[5] Philippe Blatter, Menelaos Kanakis, Martin Danelljan, and Lue Van Gool. Efficient visual tracking with exemplar transformers. In WACV, 2023.
[6] Vasyl Borsuk, Roman Vei, Orest Kupyn, Tetiana Martyniuk, Igor Krashenyi, and Jiñ Matas. FEAR: Fast, efficient, accurate and robust visual tracker. ECCV, 2022.
[7] Boyu Chen, Peixia Li, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Weihao Gan, Wei Wu, and Wanli Ouyang. Backbone is all your need: A simplified architecture for visual object tracking. In ECCV, 2022.
[8] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In CVPR, 2021.
[9] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu. MixFormer: End-to-end tracking with iterative mixed attention. In CVPR, 2022.
[10] Yutao Cui, Cheng Jiang, Gangshan Wu, and Limin Wang. MixFormer: End-to-end tracking with itcrative mixed attention. arXiv preprint arviv: 2302 3302.02814, 2023.
[11] Heng Fan, Liting Lin, Fan Yang. Peng Chu, Ge Deng. Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. LaSOT: A high-quality benchmark for large-scale single object tracking. In CVPR, 2019.
[12] Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng. Sijia Yu, Mingzhen Huang, Juchuan Liu, Yong Xu, al. LaSOT: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129:439-466 461, 2021.
[13] Benjamin Graham, Alaacldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. VeViT: A vision transformer convnet's 's clothing for faster inference. ICCV, 2021.
[14] Mingzhe Guo, Zhipeng Zhang, Heng Fan, Liping Jing, Yilin Lyu, Bing Li, and Weiming Hu. Learning target-aware representation for visual tracking via informative interactions. In IJCII, 2022.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022.
[16] Lianghua Huang, Xin Zhao, and Kaiqi Huang. G0T-10k: A large high-diversity benchmark for generic object tracking in the wild, IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5): 1562-1577, 2019.
[17] Ben Kang, Xin Chen, Dong Wang, Houwen Peng, and Huchuan Lu. Exploring lightweight hicrarchical vision transformers for efficient visual tracking. In ICCV, 2023.
[18] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. SiamRPN++: Evolution of siamese visual tracking with very deep networks. In CVPR, 2019.
[19] Liting Lin, Heng Fan, Zhipeng Zhang, Yong Xu, and Haibin Ling. SwinTrack: A simple and strong baseline for transformer tracking. In NeurlPS, 2022.
[20] Martin S. Andersen, Joachim Dahl, and Lieven Vandenberghe. CVXOPT: A python package for convex optimization (version: 1.3.0), 2022. wtt.battp://cxxopt.rrg index.ideexxxxo html
[21] Christoph Mayer, Martin Danelljan, Goutam Bhat, Matthicu Paul, Danda Pani Paudel, Fisher Yu, and Luc Van Gool. Transforming model prediction for tracking. In CVPR, 2022.
[22] Matthias Mueller, Neil Smith, and Bernard Ghanem. A benchmark and simulator for UAV tracking. In ECCV, 2016.

---

[23] Matthias Müller, Adel Bibi, Silvio Giancola. Salman Alsubaihi, and Bernard Ghanem. TrackingNet: A large-scale dataset and benchmark for object tracking the wild. In ECCV, 2018.
[24] Daniele Pamazzo, Ofir Weher, and Olga Sorkine. Robust image retargeting via axis-aligned deformation. In EUROGRAPHICS, 2012.
[25] Adrià Recasens, Petr Kellnhofer. Simon Stent, Wojciech Matusik, and Antonio Torrabaa Learning to zoom:
A saliency-based sampling layer for neural networks. In ECCV, 2018.
[26] Jianbing Shen, Yuanpei Liu, Xingping Dong, Xiankai Lu, Fahad Shahbaz Khan, and Steven Hoi. Distilled siamese networks for visual tracking. IEEE Transactions Pattern Analysis and Machine Intelligence, 44 (12):8896889)) 8909, 2021.
[27] Chittesh Thavamani, Mengtian Li, Nicolas Cchron, and Deva Ramanan. FOVEA: Foveated image magnification for autonomous navigation. ICCV, 2021.
[28] Chittesh Thavamani, Mengtian Li, Francesco Ferroni, and Deva Ramanan. Learning and unzoom. In CVPR, 2023.
[29] Shaoru Wang, Jin Gao, Zeming Li, Xianqin Zhang, and Weiming Hu. A closer look self-supervised lightweight vision transformers. In JCML. 2023.
[30] Xiao Wang. Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. CVPR, 2021.
[31] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In ICCV, 2021.
[32] Bin Yan, Houwen Peng, Kan Wu, Dong Wang, Jianlong Fu, and Huchuan Lu. LightTrack: Finding lightweight neural networks for object tracking via one-shot architecture search. In CVPR, 2021.
[33] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In ECCV, 2022.
[34] Zhipeng Zhang and Houwen Peng. Deeper and wider siamese networks for real-time visual tracking. In CVPR, 2019.
[35] Zhipeng Zhang, Yihao Liu, Xiao Wang, Bing Li, and Weiming Hu, Learn to match: Automatic matching network design for visual tracking. In ICCV, 2021.
[36] Li Zhaoping, A new framework for understanding vision from the perspective of the primary visual cortex. Current opinion in neurobiology, 58:1-10, 2019.
[37] Heliang Zheng, Jianlong Fu, Zueng-Jun Zha, and Jiebo Luo. Looking for the devil in details: Learning trilinear attention sampling network for fine-grained image recognition. In CVPR, 2019.
12

---

Appendices
A Derivations of the QP-based Minimization Task for Grid Manipulation
where the zoom energy soom is defined as The minimization task for grid manipulation in our non-uniform resizing is defined
minimize arow deal
E Zzom + EErigi
m
(AI)
subject to
Σ
di H,
Σ
dk
H
( doel
1 W
E ggoom
+
(A2)
7
7 71
k=1
and the rigid energy Erigid is defined
Erigid
m
a1
Sk.
m
di
now
dx
2
(A3)
H
W
This minimization task in Eq. Al can be efficiently solved using a standard QP solver 20 which requires converting the above minimization task into the general form of QP, i.e.,
1 =d Pd + dd =
minimize
2
(A4)
subject Ad= =d=b
where d = (dfow ..... drow diol ..... d E m+++ is the unknown grid intervals to be optimized.
P E I (m+)))x(m+)) and E R can be derived from the energy function E = Ezoom + Erigid.
A E R (m(+n)) and b E IR² can be derived from the linear constraint in Eq. AI
Converting the energy function into the matrix form. Denote Scol
Σi St. and Srow
Exe $2 the overall energy E can be expanded as
sool (^(12) W
E
Srow
+1
+1
doow
k
+1
H
k=1
H
W
25
drow
25o
dool
(AS)
mm
1=1
T=
2
E
HW
mn
d yow
pool
+
C
1=1 A=1
where C' is constant term that can be ignored during minimization. The first, second and third lines of Eq. (A5) are consisting of squared terms, linear terms and cross terms respectively.
Thus, the overall energy E can be converted into the matrix form described in Eq. (44 by fitting the above terms into P E { and E IR m+ i.e.
P
[
prow
peross
peol
q
grow
(A6)
qcol
where prow E Raxx and pool E mmx are diagonal matrices accounting for the squared terms in E. prow and pool can be defined
[28 (2( 1), k1
(25gal (2 +1,
prow
peol
(A7)
0, others
10, others
13

---

Peross E IR m is matrix accounting for the cross terms in E, while qm RR" and gral E R
account for the linear terms. peross q row and graa can be defined as
P. = -2 k. HW
mn
row
Srow 25
H
q
col
25k
W
(A8)
Converting the linear constraint into the matrix form. The linear constraint can fit in the matrix form described in Eq. (AT by defining E R2(((=+)) and IR²
1, = 1, 1,...
A
1 m + 1,.nn. n + b b= (H,
(A9)
0,ohhrs
Proof of the convexity. When the matrix P in Eq. A6 is positive semi-definite, the energy function in Eq. (AT) is convex and a global minimization can be found. Apparently, P is positive semi-definite since, for any E "
d Pd -2
Σ ST
m
+1
drow?
Scol X
+1
dk
gool
H
W
2
mn HW
yow pool
k.
(A10)
-2
ST
now 2ow?
di
Sool poot
m
dr
n
de 2
H
W
1=1
k=
Lmn 1m
20
B Attribute-based Analysis on LaSOT
We further analyze the AUC gains brought by our method on the two baselines, i.e., OSTrack and TransT, based on the attributes of the videos on LaSOT [1]. The results are displayed Fig. A1
OSTrack
TransT
AUC gain (%)
3-
LR FM VC IV ARCPOC CM
Figure A1: Attribue-bassd analysis of AUC gains on LaSOT [I
As shown in Fig. Al our method significantly improves the performance of both baselines under challenging scenarios with drastic movement (see Fast motion (FM), Viewpoint change (VC), etc) or
'FOC: Full Occlusion, OV: Oft- LR: Low Resolution, FM: Fast Motion, VC: Viewpoint Change, IV: Illmiiaaioon Variation, ARC: Aspect Ration Change, POC: Partial Occlusion, CM: Camera Motion, MB: Motion Blur, SV: Scale Variation, ROT: Rotation, BC: Background Clutter, DEF: Deformation
14

---

significant drift after some period of time (see Full Occlusion (FOC), Out-of- View (OV), etc). owning to the enlarged visual field. Performance on videos with low resolution (LR) also considerably improved thanks to the zoom-in behavior. The deformation caused by non-uniform resizing may cause a little performance degradation when encountering background clutter (BC) or deformation (DEF).
C Comparison Between Non-uniform Resizing and Optimized Uniform Resizing Baseline
To demonstrate the superiority of our method, we additionally explore whether optimizing the crop hyper-parameter the uniform resizing baseline can have similar effect to our proposed non-uniform resizing, the results are shown in Tab. A1
Despite the fact that our limited compute resources make unable to optimize the above hyper- parameters on large tracking datasets in a grid-search manner, our adequately powered experiments clearly show that the above optimizing is indeed hardly able achieve similar effects to our resizing. Even for the baseline OSTrack-25613 33) with the optimal crop hyper-parameters on LaSOT[II] (69.5 AUC), we experimentally find that it can only achieve AUC gains of +0.4 and +0.6 on LaSOT eet [12) and TNL2K[30] respectively while performing worse than the original OSTrack-255 on TrackingNet 23 (83.1 SUC vs 82.1 SUC). In other words, our method still outperforms the above optimal baseline on LaSOT (+0.7 AUC), LaSOT ezt (+2.7 AUC), TNL2K (+1.6 AUC), TrackingNet (+1.1 SUC). Moreover, it is experimentally shown that the mismatch of the object size between the search and template images caused by increasing the search- imagee context factor has relatively small effect on the final tracking accuracy for modern transformer trackers. That means the low resolution of the object is the major cause for the worse accuracy.
Table A1: Comparison results of our method and the uniform resizing baseline with its crop hyper- parameters varied (i.e., setting c" and c² using two different choices as in Sec. 3.1 varying the search-ims -macce context factor, and varying the template-image size to alleviate the mismatch of the object size between the search and temple images). The experiments are based on OSTrack-256 and tested on LaSOT[]]. The context factor and size of search/template images are displayed as context factor/size. The choices of c c, c bD d
bA and Oe d (b + 1o) b2//2
Original
c ch
Object Size Mismatched
Object Size Matched
Ours
Search 4/256
4/256
4.5/256
5/256
5.5/256
6/256
4.5/256
5/256
5.5/256
6/256
5/256
Template c ch
2/128
2/128
2/128
2/128
2/128
2/128
2/114
2/102
2/93
2/85
2/128
1
1
1
1
AUC
69.1
68.5
69.0
69.5
68.2
67.5
69.1
68.8
68.0
67.4
70.2
D More Experimental Results UAV123
In this section, report the performance comparison UAV 123 an aerial tracking benchmark captured from low-altitude UAVs with more small objects and larger variations in bounding box size and aspect ratio. As shown in Tab. A2 method achieves 1.0% and 0.6% relative gains on AUC for the OSTrack (33 and TransT g baselines, demonstrating that method can improve the tracking performance in wide range of tracking scenarios.
additional benchmark, i.e., UAV123 22
Table A2: Comparison with state-of state-of-the-aae
the-art art trackers on UAV123 (23
22
OSTrack
TransT
MixFormer-L
ToMP
DiMP
SiamFC
Dataset
OSTrack-Zoom
3
TransT-Zoom
g
9
21
4
B
AUC (%)
69.3
68.3
69.7
69.1
69.5
69.0
65.3
37.7
15

---

E More Experimental Results with SwinTrack as the Baseline
We further apply our method to both the performance-oriented version SwinTrack-BB (backbone: Swin Transformer-Ban Base) and speed-oriented version SwinTrack-T (backbone: Swin Transformer- of TwinTrack 19 to experimentally show the generalization capability of our proposed resizing method for a wide range visual tracking algorithms. Note that we use the vl version SwinTrack (without motion token) as the baseline because only the code of vl version is publicly available.
As shown in Tab. A3 our method bring consistent improvement for both SwinTrack- and SwinTrack- B. Particularly worth mentioning is that our SwinTrack- MeanTrack---oorn B-Zoom can even outperforms SwinTrack B-kk------- B-384 with much smaller imput size.
Table A3: Further validation of applying non-uniform resizing both the performance- oriented version (backhone: Swin Transformer-Base) and speed-oriented version (backbone: Swin Transformer-Tie (ifoomme---nny)) of SwinTrack [9]. Note that we use the vl version of SwinTrack (without motion token) as the baseline because only the code of v1 version is publicly available.
LISOT
Type
Trackers
Size
AUC
SiinTrack-T-Zoome
224
68.5
72.9
Speed-oriented
SwinTrack-T(UP
224
66.7
70.6
SwinTrack B Zoom
224
70.5
75,4
Performance-oriented
SwinTrack-BD 19]
224
69.6
74.1
SwinTrack-B- 19)
384
70.2
75.3
Analysis on Small Improvement Over TrackingNet
To find out the reason for the small improvement over TrackingNet [23,, we additionally analyze the ratios challenging scenarios appearing the different tracking datasets (see Tab. Tab.(44) which clearly show that the test split of TrackingNet has very different property from LaSOT III. LaSOT east [12]
and TNL.K 30 with respect to the ratios challenging scenarios. As shown in Fig. AI our method can achieve significant performance gains under challenging scenarios with drastic movement (see
Fast Motion (FM), Viewpoint Change (VC), etc.) or significant drift after some period time (see Full Occlusion (FOC), Out-of- View (OV), etc.) owning to the enlarged visual field. However, the ratios of the challenging scenarios FOC, OV, FM in TrackingNet (VC not labeled TrackingNet) are significantly lower than in LaSOT, LaSOT ent, and TNL2K, which may be the reason for small improvement of our approach on Trat
Table A4: Ratios of challenging scenarios appearing the different tracking datasets. Our method can achieve significant performance gains under challenging scenarios with drastic movement (e.g., Fast Motion (FM), Viewpoint Change (VC)) or significant drift after some period of time (e.g., Full Occlusion (FOC), Out-of-View (OV)) owning to the enlarged visual field. Note that VC is not labeled TrackingNet.
Ratios of Challenging Scenarios
AUC Gains
Datasets
Significant Drift
Drastic Motion
OSTrack 33
Trans
FOC
OV
FM
VC
LI
42. 1%
37.1%
18.9%
11.8%
+1.1% %
+2.2%
LaSOT end 12
62.7%
21.3%
58.7%
39.3%
+3.1%
+2.0%
TNL2K
30
13.9%
33.4%
24.0%
44.3%
+2.2%
+3.0%
TrackingNet
4.7%
4.0%
10.0%
+0.1%
+0.4%
16

---

G Detailed Latency of Non-uniform Resizing
We provide the latency analysis terms of time (ms) and MACs for solving QP and resizing images in Tab. A5 It can be seen that resizing images with interpolations in method costs about half of the total time in spite its parallel processing, while iteratively solving QP costs the rest of the total time despite its much lower FLOPs.
Table A5: Detailed latency of non-uniform resizing
Component
Time(ms)
MACs
Solving QP
0.89
0.16M
Resizing
0.69
1.12M
Total
1.58
1,28 M
H Confidence Interval for Ablation Experiments of Hyper-parameters
We calculate the 90% confidence intervals for all the experiments in Tab, using bootstrap sampling. Each of the 90% confidence intervals obtained using 10000 bootstrap samples, each sample with size 280 (same as the video number of LaSOT-test-splee The results are listed in Tab. A6
Table A6: 90% confidence intervals for ablation experiments of paperrparameters.
Bandwidth
(b) Control grid size
8
64
256
Size 8
16
32
Upper 70.95 (+1.87) 72.02 (+1.86) 71.54 (+1.90)
Upper
68.50 (42.06)
72.02 (+1.86)
71.52 (+1.98)
Mean 69.08
70.16
69.64
Mean
66.44
70.16
69.54
Lower
67.23 (-1.85)
68.33 (-1.83) 67.80 (-1.84)
Lower
64.49 (-1.95)
68.33 (-1.83) 67.63 (-1.91)
(c) Zoom factor y
(d) Balance factor À
1.5 Upper 71.58 (+1.89)
1.25
1.75
À
0
2
72.02 (+1.86)
71.35 (+1.94)
Upper 71.69 (+1.89) 72.02 (+1.86) 71.43 (+1.87)
Mean 69.69
70.16
69.41
Mean 69.80
70.16
69.56
Lower 67.88 (-1.81)
68.33 (-1.83
67.49 (-1.92)
Lower 67.93 (-1.87)
68.33 (-1.83)
67.71 (-1.85)
I More Visualization Results
We visualize the results of method OSTrack-Zooe -oom and its corresponding baseline videos with challenging attributes to qualitatively show the superiority of our method in Fig. Thanks to the zoom-in behavior and the enlarged visual field, our method can better handle the challenges such as full occlusion, fast motion, etc.
As shown in the left part of Fig. A2 at frame # 1.50, both our method and the baseline lose track of the target since the target is fully occluded. At frame #157, our method quickly re-detects the target as the target re-appears. Note the target is significantly larger in the resized input image when using our resizing method, which indicates that the zoom-in behavior allowed by our method helps the re-detection of the target. At frame # 187, our method successfully tracks the target whereas the baseline still drifts to a similar car.
As shown in the right part of Fig. A2 at frame #15, the target is moving really fast. As result, in the resized input image generated by the baseline, part the target is out of view which causes an inaccurate localization. In contrast, the resized input image generated by our method keeps the entire target visible thanks to the enlarged visual field, resulting in a more accurate tracking result. The situation is similar in frame #27, where the baseline tracks a ppleseeeed distractor rather than
17

---

the half-cropped target, whereas our method successfully tracks the target since the target is intact. At frame #32, the baseline completely lose track of the target, while our method successfully tracks
DOUIL
Sadla
150
KNO
comenoo
KIRRAS
Sadta
157
RENOO
187
RENOO
Correlos
Ground Truth
OSTrack DSTrack-Zoom
OSTrack
Figure A2: Visualization of tracking results on videos with challenging attributes. Input to the tracker is displayed the right side of each full image, where the image with blue box is the input image resized by our method and the image with green box the input image resized by the baseline. (left) An example of video with full occlusion. The zoom-in behavior allows our method to quickly re-detect the target and avoid drifting. (right) An example of video with fast motion. The enlarged visual field allows tracker have the whole target visible without cropping, which facilitates localization and avoids drifting to similar objects when the target has drastic motion.
J
Failure Cases
We visualize the failure cases of our method by comparing the tracking results of OSTrack-Zoom (blue) and OSTrack (green) in Fig. A3
at frame # 1108, the target out view. As our method enlarges the visual field, apart from the distractor (orange ball) at the center of the input, additional distractor (dark green ball at the top-right of the input image) comes into view, which causes tracker to predict large box between the two distractors. The localization emror quickly accumulates to an overly enlarged visual field at frame # 1117, since the visual field is partly determined by the previous prediction. At this time, white ball that is very similar the template is brought into view, causing our method to drift to the far-away white ball. Then, at frame # 1130, when the target re-appears, our method cannot re-detect the target makes the real target out of the visual field our tracker.
As shown in the left part of Fig. A3
method drifts to a far-away distractor, which
As shown in the right part of Fig. A3, at frame #94, both our method and the baseline successfully track the target. However, frame # 104, the enlarged visual field from our method brought additional contexts (orange robotic arm on the left) into view. The little fraction of orange robotic arm on the left makes the distractor look like a horizontally flipped template which has a small fraction of orange robotic arm on the right. As result, our method drifts to the distractor. Then, at frame # 14, our
18

---

the image method fails to track back to the squeezed real target since it is at the edge of the input image and partly invisible.
From the visualization of the failure cases, we find that, although an enlarged visual field promotes tracking performance in most cases, sometimes additional distractors contexts brought by the fixed large visual field will cause drifting. Thus, would be interesting to develop a method to dynamically adjust the visual field based the surrounding of the target and its tracking history in the future.
1,08
94
104
1130
Ground Truth OSTrack-Zoom
OSTrack
Figure A3: Visualization of failure cases. Input to the tracker is displayed the right side of each full image, where the image with blue box the input image resized by our method with green box is the input image resized by the baseline. Template image is visualized on the top-left corner of the full image. (left) A combination of out-of-view and fast motion causes an accumulated prediction еπrоr which leads to drifting. Localization with enlarged visual field under some challenging scenarios may difficult and its wrong prediction may produce less desirable results since the enlarged visual field brought more distractors. The localization error quickly accumulates to an overly enlarged visual field which then causes drifting. (right) Cluttered background with similar distractors causes drifting. The additional context (orange robotic arm) brought by the enlarged visual field makes the distractor on the left look like a horizontally flipped template, which causes drifting.
19


Reviews:
[{"review_id": "UXjmkaV2hv", "reviewer": "Reviewer_Gq4E", "summary": "This paper proposes a simple but clever method to improve the accuracy-speed trade-off of state-of-the-art methods for single object tracking. Existing methods can achieve greater accuracy by using higher-resolution images, however this comes at the cost of inference speed. This paper proposes to adopt a non-uniform resizing of the image which preserves a large search region without sacrificing the resolution of the image at the location where the object is expected to appear. Solving for the control points of an axis-aligned grid is formulated as a QP to preserve the bounds of the region while minimizing distortion and approximating a desired zoom in the region where the object is likely to appear. Results on several large datasets with two different trackers show that this simple modification makes a large step towards closing the gap between fast and accurate trackers. The small set of hyper-parameters is fixed for all datasets and both trackers.", "questions": "Please provide counter-arguments to (or correct misunderstandings in) the weaknesses above.\n\n1. How is the non-uniform resizing applied to the search and template images? Was it applied to the template image? If so, is the equation solved again with different W and H? Could this result in the object having a different size in the search and template image?\n1. Does FOVEA have a hyper-parameter that would improve its results (or could it be trivially modified to include such a hyper-parameter)? Would the proposed warping still outperform it?\n1. In the experiments, it is stated that the context factor for the search image was increased to 5. This was shown to be optimal for the baseline OSTrack on LaSOT in Figure 3 (~69.5 AUC compared to ~69.0 AUC at 4.5). Table 1 shows OSTrack-256 to achieve 69.1 AUC on LaSOT. This suggests that the optimal context factor was not used for the baseline. Is this because a context factor of 5 has a negative effect on the baseline on other datasets? (OSTrack-Zoom achieved 70.2 AUC, which is less impressive compared to 69.5 than compared to 69.0.)\n\nLess important:\n1. What is the effect of applying ZoomTrack to the performance-oriented trackers like OSTrack-384?\n1. I'm curious about how important it is to re-train the tracker. What is the effect of applying the non-uniform resizing _without_ re-training the tracker?\n\nSuggestion:\n1. If the object was treated as a square, then the QP could be solved once offline for all objects. Would this have a significant negative impact on the method?\n1. Is the solution of the QP always symmetric in x and y?\n\nEdits:\n1. The term \"performance\" is multivalent (may mean speed or accuracy); I suggest \"accuracy\" or \"quality\" instead.\n1. \"throw the question\" -> \"pose the question\"\n1. \"with possibility\" -> \"with probability\"\n1. Citation required for 100MB / 1MB in human vision system.\n1. It is stated that the context factor for the template image is left unchanged while that for the search image is increased to 5. Please state the original values so that the reader does not need to go and find them.", "limitations": "The authors have adequately addressed the limitations of the work. I do not foresee any negative impacts stemming from this specific work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Paper identifies a practical dilemma and proposes a simple but clever solution.\n1. Proposed method can be applied with many different trackers.\n1. The method significantly advances the Pareto front of the AUC vs MACs trade-off, even dominating the best trackers on the TNL2K dataset (Figure 1).\n1. Solving for a non-uniform grid is formulated as a QP with efficient solution.\n1. Small (coordinate-wise) sweep conducted for the key hyper-parameters on large dataset.\n1. Same hyper-parameters used for multiple datasets and multiple methods.\n1. The visualizations show that the target is magnified without being distorted.\n1. The statistics demonstrate that, under the proposed non-uniform resizing, the mean size of the target is increased while its variance is unaffected.\n1. Inclusion of FOVEA as baseline.\n1. Increase in time required by non-uniform resize operation was accounted for.", "weaknesses": "1. My main concern is that the proposed non-uniform resizing may be similar in effect to varying the crop hyper-parameters (method for setting $c_x$ and $c_y$, search-image context factor, template-image size) and there is not a rigorous comparison. Showing that the proposed method (optimized for $\\beta$, $\\gamma$, $\\lambda$) is better than optimizing these hyper-parameters would be much more convincing.\n1. No confidence intervals (error bars). This would be particularly useful for Table 3.\n1. Figure 3 shows that increasing the search-image context factor results in worse accuracy due to the low resolution of the object. However, the drop in accuracy may also be caused by a mismatch in the size of the object in the search and template images. The same applies for decreasing the search-image context factor. I think this study should also consider the template-image size.\n1. The paper did not explain how the non-uniform resizing was applied to the search and template images (question below).\n1. It is concerning that a higher-resolution control grid leads to worse results. This suggests that the QP objective does not capture all necessary factors and instead relies on a coarse grid to avoid over-optimizing the flawed objective.\n1. It seems unnatural to take the square root of the box width in $\\sigma_x = \\sqrt{\\beta \\cdot r^w}$. This would result in $\\sigma$ depending non-linearly on the box size. Is there a reason for this?\n1. Code for experiments not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! *We will release our code upon acceptance and include the new analyses below in the revision*. We provide responses to the specific points below:\n\n**About the edits**: We will revise the edits accordingly. The citation for 100MB/1MB in the human vision system is [29] in the main paper. The original context factors for the template and search images are 2 and 4 respectively.\n\n**About our superiority over the uniform resizing baseline with its crop hyper-parameters optimized**: Following your suggestion, we additionally explore whether optimizing the crop hyper-parameters in the uniform resizing baseline (See Tab. R1 in the attached PDF for “global” response) can have a similar effect to our proposed non-uniform resizing. Despite the fact that our limited compute resources make us unable to optimize the above hyper-parameters on large tracking datasets in a grid-search manner, our adequately powered experiments clearly show that the above optimizing is indeed hardly able to achieve similar effects to our resizing. Even for the baseline OSTrack-256 with the optimal crop hyper-parameters on LaSOT (69.5 AUC), we experimentally find that it can only achieve AUC gains of +0.4 and +0.6 on LaSOT_ext and TNL2K respectively while performing worse than the original OSTrack-256 on TrackingNet (83.1 SUC vs 82.1 SUC). In other words, our method still outperforms the above optimal baseline on LaSOT (+0.7 AUC), LaSOT_ext (+2.7 AUC), TNL2K (+1.6 AUC), TrackingNet (+1.1 SUC). Moreover, it is experimentally shown that the mismatch of the object size between the search and template images caused by increasing the search-image context factor has a relatively small effect on the final tracking accuracy for modern transformer trackers. That means the low resolution of the object is the major cause for the worse accuracy.\n\n**About how ZoomTrack was applied to the search and template images? Can QP be solved once offline for all objects?**: We are sorry for not clearly indicating that we did not apply our non-uniform resizing on the template image. Although the template image can be non-uniformly resized by solving QP with the ground truth annotation and a relatively small context factor only once at the beginning frame during testing, we experimentally find that either applying non-uniform resizing on template image alone or on both template and search images performs worse than applying it only on search image (Tab. R2). We think the reason is that the template image must have a relatively small context factor to contain minimal necessary context, which makes the whole template region important for the tracking network to locate the target. So the trackers cannot benefit from non-uniform resizing of template image. As for the search image, the QP is solved for every frame given the previous prediction box during tracking. Following your suggestion, we tested the case of treating the object as a square so as to solve QP once offline for all objects. However, AUC on LaSOT drops from 70.2 to 69.7. We leave the thorough investigation on this issue for future work due to the limited rebuttal time.\n\n**About the concern of denser grid leading to worse results**: As we mentioned in the main paper (Lines 362-363), we think the reason for the worse results is that the important areas generated from the temporal prior during tracking are not always accurate, which is inevitable since the prediction of the tracker cannot be always correct. We think it is the inaccurate previous frame prediction that leads to performance degradation for the denser grid rather than our QP objective. To validate this argument, we tested the case of using the ground truth from the previous frame instead of the predicted bounding box to crop and resize the search region for the current frame. Under this new setting, our method using grid size 32 outperforms 16 counterpart (77.6 AUC vs 76.9 AUC) on LaSOT, proving that denser grid can achieve better accuracy thanks to the more reliable temporal prior.\n\n**About the error bars for Tab. 3**: Theoretically, neither the deterministic OSTrack nor our resizing module introduces randomness during testing. We also ran our tracker 3 times on LaSOT and the results are the same. As for the training with stochastic gradient descent, we trained the models for all the experiments using a shared random seed. We leave the training using different random seeds for error bars for future work due to the limited rebuttal time.\n\n**About $\\sigma_x$ in Eq. (5)**: We take the square root of the box width for $\\sigma_x$ following the practice in FOVEA [22] which converts bounding boxes to a saliency map based on kernel density estimation.\n\n**About tuning hyper-parameters for FOVEA**: FOVEA shares some hyper-parameters with our method, e.g., grid size and bandwidth. We tuned them when preparing our work and interestingly found that its best performance on LaSOT (67.6 AUC) is also achieved by setting grid size to 16 and bandwidth to 64, which are the same as our method.\n\n**About applying ZoomTrack to performance-oriented trackers like OSTrack-384**: We tested the case of increasing the context factor of OSTrack-384 from 5 to 6 while increasing its search-image size to 480, however, it cannot further improve AUC on LaSOT (71.1 vs 71.0). Thus it is difficult for ZoomTrack to further improve over OSTrack-384. However, our improvements over SwinTrack [15] on LaSOT are also impressive. Please refer to the 3rd item of the “global” response for more details.\n\n**About the effect of applying ZoomTrack without re-training**: In Sec. D of our supplementary material we have studied this effect. Please refer to this material for more details.\n\n**About the symmetric QP solution**: The solution in the x-axis and y-axis is always symmetric because the Gaussian importance score function for the search image is symmetric in x and y. Some accelerating strategies may be developed using this symmetry."}, {"review_id": "j87uLYvSFv", "reviewer": "Reviewer_PYzX", "summary": "The paper presents a ZoomTrack, a target-aware non-uniform resizing technique for efficient visual tracking. It narrows the gap between speed-oriented and performance-oriented trackers by non-uniformly resizing the cropped image to have a smaller input size while retaining more raw information for the target. A quadratic programming-based formulation for efficient solving is also designed. The experiments demonstrate the consistent improvements over existing transformer trackers.", "questions": "See above weaknesses.", "limitations": "See above weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "It seems the first time to apply the non-uniform resizing to the field of object tracking.\n\nThe paper is well written. The authors provide a clear explanation of the proposed method, including the formulation and solution of the resizing problem through quadratic programming.\n\nThe results show that it achieves consistent improvements over two Transformer trackers on multiple tracking datasets.", "weaknesses": "The idea is similar to \"Learning to Zoom and Unzoom\" presented at CVPR 2023.\n\nThere are many hyperparameters in the proposed method, and how to set or tune their values?\n\nAlthough the consistent improvements over two Transformer trackers on multiple tracking datasets are achieved, the improvements on LaSOT and TrackingNet are small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! We are happy that you find our paper well written and acknowledge that we are the first time to apply non-uniform resizing to the field of object tracking. We provide responses to the specific points below:\n\n**About the comparison with CVPR 2023 work \"Learning to Zoom and Unzoom\"**: Thanks for pointing out this concurrent nice work which was presented at CVPR 2023. We find that both this CVPR 2023 work (dubbed as LZU) and our work share a similar idea to improve perception accuracy by non-uniform resizing. However, LZU and our work contribute to different aspects of non-uniform resizing. LZU proposed an efficient and differentiable warp inversion that allows for mapping labels to the warped image to enable the end-to-end training of dense prediction tasks like semantic segmentation. LZU still uses the grid generator from FOVEA [22]. In contrast, our work proposed and carefully designed a new grid generation method, which can improve the quality of non-uniform resizing and perform better than FOVEA in visual tracking. Our method proves to be able to bridge or even close the gap between performance-oriented and speed-oriented trackers. We will cite this concurrent work and include the above analysis in the revised related work.\n\n**About how to set and tune the hyper-parameters in our method**: For the small set of hyper-parameters (acknowledged by Reviewer `Gq4E`) in our QP formulation based on grid representation, we claim that they (e.g., $\\beta$, $\\gamma$, $\\lambda$, grid size) are tightly coupled with our formulation and also have very clear meanings in our framework, so that we can choose a set of reasonable initial values (which may be the optimal values) and then conduct small (coordinate-wise) sweep (acknowledged by Reviewer `Gq4E`) for them to test the sensitivity of our OSTrack-Zoom to these hyper-parameter choices in Tab. 3. For example, by design, we make the zoom factor $\\gamma$ capable of controlling the amount of magnification. The performance-oriented version of OSTrack uses 1.5$\\times$ larger search-image size than its speed-oriented one, so it is natural to set the initial value of $\\gamma$ to 1.5 in our OSTrack-Zoom, which proved to be an appropriate value in our experiments. Another example is the control grid size. It is natural to use 16$\\times$16 grid because the search image after non-uniform resizing is evenly split into 16$\\times$16 patches and then sent into the Vision Transformer (ViT) backbone for feature extraction. As our resizing process can be intuitively understood as resizing each rectangular grid patch on the source image to the one on the target image, we can expect that each patch for ViT backbone is obtained from one resizing with the solved $d_k^{col}$ and $d_l^{row}$. We directly apply the same set of hyper-parameters chosen based on the OSTrack baseline to another TransT baseline for further validation, and surprisingly find that significant improvements over TransT can be also achieved thanks to the robustness of our method. As acknowledged by Reviewer `Gq4E`, using the same hyper-parameters for all datasets and both trackers is a strength of our paper. Note that the improvements over SwinTrack (see Tab. R3) are also achieved using the same hyper-parameters. We will detail the above analysis in the revision.\n\n**About the performance gains on LaSOT and TrackingNet**: Although our performance gains on TrackingNet are small, we think that the AUC improvements on LaSOT with respect to both baselines (1.1 for OSTrack and 2.2 for TransT) are still impressive. As acknowledged by Reveiwer `2uk1`, enabling OSTrack to achieve 70.2 AUC on LaSOT with 100fps is impressive. As for the reason of small improvements on TrackingNet, we additionally analyze the ratios of challenging scenarios appearing in the different tracking datasets (see below), which clearly show that the test split of TrackingNet has a very different property from LaSOT, LaSOT_ext, and TNL2K with respect to the ratios of challenging scenarios. As shown in Fig. S1 of our appendix material, our method can achieve significant performance gains under challenging scenarios with drastic movement (see Fast Motion (FM), Viewpoint Change (VC)) or significant drift after some period of time (see Full Occlusion (FOC), Out-of-View (OV)) owning to the enlarged visual field. However, the ratios of the challenging scenarios FOC, OV, FM in TrackingNet (VC is not labeled in TrackingNet) are significantly lower than in LaSOT, LaSOT_ext, and TNL2K, which may be the reason for small improvements of our approach on TrackingNet. We will include this new analysis in the revision.\n\n| Dataset     | Ratio of FOC | Ratio of OV | Ratio of FM | Ratio of VC     | AUC Gain (OSTrack) |\n| ----------- | ------------ | ----------- | ----------- | --------------- | ------------------ |\n| LaSOT       | 42.1%        | 37.1%       | 18.9%       | 11.8%           | +1.1%              |\n| LaSOT_ext   | 62.7%        | 21.3%       | 58.7%       | 39.3%           | +3.1%              |\n| TNL2K       | 13.9%        | 33.4%       | 24.0%         | 44.3%           | +2.2%              |\n| TrackingNet | 4.7%         | 4.0%          | 10.0%         | - (not labeled) | +0.1%              |\n\n"}, {"review_id": "QpbsURVN9Z", "reviewer": "Reviewer_Fe6p", "summary": "This paper propose a novel method to achieve high tracking speed with smaller input size. Non-uniformly resizing operation is introduced with respect to the object appearing probability.  This resizing operation is then modeled as QP problem and can be effectively soved. This proposed method can be a flexible plugin for multiple deep trackers. Extensive experiments on various tracking benchmarrk are conducted with TransT and OSTrack to demonstrate the effectiveness of the proposed method.", "questions": "plz refer to the weakness", "limitations": "more ablation studies shoud be conducted. ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. authors identify the conflict of context ratio and search region size in deep trackers, and proposed a novel method to solve it. \n\n2. the target region resizing is modeled as a QP problem analytically. \n\n3. Introducing non-uniform resizing for target regions and solve the QP problem efficiently. ", "weaknesses": "1. The tracking performance of proposed non-uniform resizing is only evaluated on speed-oriented tracker. I recommend the author  better to further validate this method on performance-oriented trackers. \n\n2. The computation overhead of proposed method is not reported detailly. A more detail evaluation about the latency of solving the QP problem and resizing image regions with larger context regions should be provided, in term of FLOPS.\n\n3. the effect of proposed non-uniform resizing is not abalated on template and search regions separated.  I think it neccessary to find its effects on template and search regions respectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! We are happy that you find our method is novel and can be a flexible plugin for other deep trackers. We provide responses to the specific points below:\n\n**About more experimental validations on performance-oriented trackers to further show the generalization capability**: We further apply our method to both the performance-oriented version (backbone: Swin Transformer-Base) and speed-oriented version (backbone: Swin Transformer-Tiny) of SwinTrack [15] to experimentally show the generalization capability of our proposed resizing method for wide range of visual tracking algorithms. Due to the limited computing resources, we only report the comparisons on the large-scale LaSOT dataset (see below), which further clearly show the superiority of our approach. Note that we use the v1 version of SwinTrack as the baseline because only the code of v1 version is publicly available. We will include this new baseline in the revision.\n\n|                          | Size | AUC  | P    |\n| ------------------------ | ---- | ---- | ---- |\n| **Speed-oriented**       |      |      |      |\n| SwinTrack-T-Zoom         | 224  | **68.5** | **72.9** |\n| SwinTrack-T              | 224  | 66.7 | 70.6 |\n| **Performance-oriented** |      |      |      |\n| SwinTrack-B-Zoom         | 224  | **70.5** | **75.4** |\n| SwinTrack-B              | 224  | 69.6 | 74.1 |\n| SwinTrack-B-384          | 384  | 70.2 | 75.3 |\n\n**About detailed evaluation on the latency of solving the QP problem and resizing images with larger context regions**: Thanks for this thoughtful advice. We provide the latency analysis in terms of time (ms) and FLOPs for solving QP and resizing images as follows. It can be seen that resizing images with interpolations in our method costs about half of the total time in spite of its parallel processing, while iteratively solving QP costs the rest of the total time despite its much lower FLOPs. We will include this new analysis in the revision.\n\n|            | Time (ms) | FLOPs |\n| ---------- | --------- | ----- |\n| Solving QP | 0.89      | 0.33M |\n| Resizing   | 0.69      | 2.23M |\n| Total      | 1.58      | 2.56M |\n\n**About investigating the effects of applying the non-uniform resizing on the template and search regions separately**: We are sorry for not clearly indicating that we did not apply our non-uniform resizing on the template image. The template image can be non-uniformly resized by solving QP with the ground truth annotation and a relatively small context factor only once at the beginning frame during testing. Following your suggestion, we have investigated the effects of applying the non-uniform resizing on template and search images separately on LaSOT, and experimentally find that either applying non-uniform resizing on template image alone or on both template and search images performs worse than applying it only on search image (see below). We think the reason is that the template image must have a relatively small context factor to contain minimal necessary context, which makes the whole template region important for the tracking network to locate the target. So the trackers cannot benefit from non-uniform resizing of the template image. We will include this new analysis in the revision.\n\n|             | Zoom     | Zoom   | LaSOT | LaSOT |\n| ----------- | -------- | ------ | ----- | ----- |\n|             | Template | Search | AUC   | P     |\n| OSTrack-256 | √        |        | 64.8  | 71.0  |\n| OSTrack-256 |          | √      | **70.2**  | **76.2**  |\n| OSTrack-256 | √        | √      | 69.6  | 75.5  |"}, {"review_id": "RrlB0o4fA5", "reviewer": "Reviewer_P4Ao", "summary": "The authors propose a non-uniform resizing method for visual tracking, where they aim to boost the performance of transformer-based visual tracking methods at negligible additional computational cost. They achieve this by reducing the input size while attending to the relevant parts of the target, and this approach is applicable to wide range of crop-based tracking algorithms. They show the results of their algorithm on various large-scale datasets.", "questions": "Please refer to the weaknesses section for details. I generally concur with the motivations raised by the authors, but their approach to the problem needs more explanation and justifications for their design choices are needed.", "limitations": "Authors provide a limitations section at the end of their paper, and the limitations are adequately addressed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Authors focus on the easily overlooked aspect of visual tracking algorithms, and their approach seems technically plausible while bringing adequate enlightenment to the field.\n\n- Their approach shows competitive performance on large-scale benchmarks compared to contemporary tracking algorithms.\n\n- They validate their method on multiple baseline algorithms with ablation experiments.", "weaknesses": "- Although the motivation of the proposed method seems plausible, there seems to be a lack of validation when it comes to analyzing the importance of choosing input resizing method for trackers. Are there further analysis on the impact of input resizing for wide range of visual tracking algorithms?\n\n- Since the formulation for the rectangular grid is hand-crafted using multiple controlallable hyperparameters (e.g. bandwitdh of Gaussians $\\beta$, choice of $\\epsilon, \\gamma, \\lambda$, ...), results in Table 3 shows some sensitivity to these hyperparameter choices. Considering these results and the results in Table 1, the performance gains obtained by the proposed strategy seems limited.\n\n- What are the reasons for the design choices of choosing (1) Gaussian importance scores for target area, (2) Using quadratic formulation for zoom and rigid energy? Are there alternative formulations tested?\n\n- Proposed non-uniform grid formulation seems to be target content (e.g. target object class, color, etc.) agnostic and only dependent on the input bounding box position and size. Since a simple learning based model (e.g. lightweight MLP) is also available, would this have been a better approach?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! We are happy that you find our method technically plausible while bringing adequate enlightenment to the visual tracking field by studying an easily overlooked problem. We provide responses to the specific points below:\n\n**About more experimental validations and analyses to further show the importance of choosing input resizing method**: We further apply our method to both the performance-oriented version (backbone: Swin Transformer-Base) and speed-oriented version (backbone: Swin Transformer-Tiny) of SwinTrack [15] to experimentally show the generalization capability of our proposed resizing method for wide range of visual tracking algorithms. Due to the limited computing resources, we only report the comparisons on the large-scale LaSOT dataset (see below), which further clearly show the superiority of our approach. Note that we use the v1 version of SwinTrack as the baseline because only the code of v1 version is publicly available. Although analyzing the importance of choosing input resizing method is an overlooked and little-explored problem in the visual tracking field and the common practice is using uniform resizing by default, we believe our further experimental analysis based on SwinTrack, as well as the previously explored two baselines (OSTrack and TransT), can demonstrate the importance of choosing appropriate input resizing method for trackers. We will include this new baseline in the revision.\n\n|| Size | AUC  | P    |\n|-|-|-|-|\n|**Speed-oriented**|      |      |      |\n| SwinTrack-T-Zoom| 224  |**68.5**|**72.9**|\n| SwinTrack-T| 224  | 66.7 | 70.6 |\n| **Performance-oriented** |      |      |      |\n| SwinTrack-B-Zoom| 224  |**70.5**|**75.4**|\n| SwinTrack-B| 224  | 69.6 | 74.1 |\n| SwinTrack-B-384| 384  | 70.2 | 75.3 |\n\n**About how the hyper-parameters in our proposed method are tuned and how they work**: We claim that the small set of hyper-parameters (acknowledged by Reviewer `Gq4E`) in our QP formulation based on grid representation (e.g., $\\beta$, $\\gamma$, $\\lambda$, grid size) are tightly coupled with our formulation and also have very clear meanings in our framework, so that we can choose a set of reasonable initial values (which may be the optimal values) and then conduct small (coordinate-wise) sweep (acknowledged by Reviewer `Gq4E`) for them to test the sensitivity of our OSTrack-Zoom to these hyper-parameter choices in Tab. 3. For example, by design, we make the zoom factor $\\gamma$ capable of controlling the amount of magnification. The performance-oriented version of OSTrack uses 1.5$\\times$ larger search-image size than its speed-oriented version, so it is natural to set the initial value of $\\gamma$ to 1.5 in our OSTrack-Zoom, which proved to be an appropriate value in our experiments. Another example is the control grid size. It is natural to use 16$\\times$16 grid because the search image after non-uniform resizing is evenly split into 16$\\times$16 patches and then sent into the Vision Transformer (ViT) backbone for feature extraction. As our resizing process can be intuitively understood as resizing each rectangular grid patch on the source image to the one on the target image, we can expect that each patch for ViT backbone is obtained from one resizing with the solved $d_k^{col}$ and $d_l^{row}$. We directly apply the same set of hyper-parameters chosen based on the OSTrack baseline to another TransT baseline for further validation, and surprisingly find that significant improvements over TransT can be also achieved thanks to the robustness of our method. As acknowledged by Reviewer `Gq4E`, using the same hyper-parameters for all datasets and both trackers is a strength of our paper. Note that the improvements over SwinTrack are also achieved using the same hyper-parameters. We do **respectfully** disagree with the statement that our performance gains seem limited due to the reason that the results in Tab. 3 show some sensitivity to the small set of hyper-parameters in our formulation. Because these hyper-parameters are not independent of or orthogonal to our whole framework. We claim that our formulation also needs to be appropriately set to unveil the power of non-uniform resizing. We will detail the above analysis in the revision.\n\n**About the reasons for using the Gaussian importance scores and the quadratic formulation**: We actually tested another alternative formulation for generating the importance scores before, i.e., generating from the current search image using a saliency detection network and then exploiting the FOVEA [22] grid generator. However, this formulation is time-consuming and we find it difficult to jointly optimize the saliency detection network and the tracking network. We choose the simple yet effective Gaussian importance scores because this formulation is widely used for modeling uncertainty in modern computer vision tasks. As for the quadratic formulation, we need the quadratic energy functions to efficiently solve the non-uniform grid in a Quadratic Programming (QP) formulation (acknowledged by Reviewer `Gq4E`). We will make a clearer explanation in the revision.\n\n**About replacing the non-uniform grid formulation with a simple learning-based model**: Thanks for this insightful comment. Following your suggestion, we tested the case of replacing the QP grid generator with a 3-layer MLP to train the trackers, where the input to the MLP is a flattened 1-D vector from our generated Gaussian importance map. However, this simple learning-based model is indeed hardly able to achieve similar effects to our resizing, obtaining 67.9 AUC on LaSOT. We believe a carefully designed learning-based model and exhaustive experiments are needed to find an effective framework, which can be our future work. We believe our current work can provide good grid generation guidance for this future work."}, {"review_id": "fl7IvbVLt1", "reviewer": "Reviewer_2uk1", "summary": "The paper studies a very interesting problem in local visual tracking with crop-then-resize operations: if it is possible to reduce the high computational demand of attending to a large visual field while retaining its potential benefit of achieving SOTA tracking accuracy. To this end, the paper takes inspiration from the processing of the human retina and proposes an elegant non-uniform resizing method to enable retaining more raw information for the target area while resizing a cropped image with a large visual field to a smaller size. By enforcing an explicitly controlled magnification of important areas with restriction on extreme deformation on top of a grid representation, this non-uniform resizing can be achieved efficiently based on QP. The authors select two representative local trackers (TransT and OSTrack) to demonstrate the well-balanced tracking accuracy and speed of trackers that are based on the proposed resizing module. Extensive experiments are conducted on five challenging tracking benchmarks, which clearly show the superiority of the proposed approach.", "questions": "There are several questions.\n\n1, In line 198-199, the authors said they use a (m+1)*(n+1) small-sized controllable grid to approximate the grid G with shape h*w. How can a grid be approximated by another grid with a different size?\n\n2, In Eq. (6) for determining the important score of different areas, a small constant is also used to prevent extreme deformation. What’s the relationship between this part and the specially designed rigid energy in Eq. (8) following G2?\n\n3, In Table 1 and Figure 1, the authors use MACs to indicate the computational overhead, while in line 353 they use FLOPS. Please explain this inconsistency. Moreover, is the computational overhead of your method in Table 1 exactly the same as the corresponding baseline?\n\nOverall, I like this paper a lot, including the novelty with broad interest and its promising performance. However, I rated it as weak accept (6) now due to the above weaknesses and the above-mentioned questions. If my concerns and questions are addressed sufficiently, I will increase my score.", "limitations": "Yes, the limitations have been briefly addressed in the paper. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "In general, I am positive about this paper. I am glad to see this paper paying some effort to improve the little-explored cropping and resizing strategy in local visual trackers. The strengths of this paper are as follows:\n\n1, Good organization and writing. The paper is well organized and the flow is smooth.\n\n2, Novelty. The proposed non-uniform resizing method is quite interesting and looks solid to me, which enables retaining more raw information for the target area during resizing.\n\n3, Promising performance. This paper enables OSTrack to achieve 70.2% AUC on LaSOT with 100fps, which is impressive. Most existing competitors always rely on heavy trackers to achieve AUC score above 70% on LaSOT. The performance gap between the proposed efficient tracker and the heavy trackers on GOT-10k, LaSOT_ext, and TNL2K is even closed. The experimental analyses are also aligned with the motivation of attending to a large visual field.", "weaknesses": "1, Only two transformer trackers are selected for experimental validations. One is the hybrid CNN-Transformer tracker TransT and the other one is the one-stream Transformer tracker OSTrack. It would be better if one more purely CNN-based tracker or some other top performer is selected to further show the generalization capability. The conclusion would be more convincing.\n\n2, In Table 1, it is observed that the improvement on TrackingNet is very small. The SUC performance gain is only 0.1%. I suppose there are some detailed explanations for the reasons but obviously, there is not.\n\n3, It is not clear to me whether the proposed non-uniform resizing is also applied to the cropped template. A detailed analysis in this regard is also required to make the paper self-contained and fully reproducible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! We are happy that you find our method quite interesting and solid, which studies a little-explored problem, and acknowledge that the performance gain on LaSOT is impressive and the experiments clearly show the superiority of our elegant solution. We provide responses to the specific points below:\n\n**About more experimental validations to further show the generalization capability**: We further apply our method to both the performance-oriented version (backbone: Swin Transformer-Base) and speed-oriented version (backbone: Swin Transformer-Tiny) of SwinTrack [15] to experimentally show the generalization capability of our proposed resizing method for wide range of visual tracking algorithms. Due to the limited computing resources, we only report the comparisons on the large-scale LaSOT dataset (see below), which further clearly shows the superiority of our approach. Note that we use the v1 version of SwinTrack as the baseline because only the code of v1 version is publicly available. We will include this new baseline in the revision.\n||Size| AUC|P|\n|-|-|-|-|\n|**Speed-oriented**||||\n|SwinTrack-T-Zoom|224|**68.5**|**72.9**|\n|SwinTrack-T|224|66.7|70.6|\n|**Performance-oriented**||||\n|SwinTrack-B-Zoom|224|**70.5**|**75.4**|\n|SwinTrack-B| 224|69.6|74.1|\n|SwinTrack-B-384|384|70.2|75.3|\n\n**About the reason for small improvements on TrackingNet**: We additionally analyze the ratios of challenging scenarios appearing in the different tracking datasets (see below), which clearly show that the test split of TrackingNet has a very different property from LaSOT, LaSOT_ext, and TNL2K with respect to the ratios of challenging scenarios. As shown in Fig. S1 of our appendix material, our method can achieve significant performance gains under challenging scenarios with drastic movement (see Fast motion (FM), Viewpoint change (VC)) or significant drift after some period of time (see Full Occlusion (FOC), Out-of-View (OV)) owning to the enlarged visual field. However, the ratios of the challenging scenarios FOC, OV, FM in TrackingNet (VC is not labeled in TrackingNet) are significantly lower than in LaSOT, LaSOT_ext, and TNL2K, which may be the reason for small improvements of our approach on TrackingNet. We will include this new analysis in the revision.\n|Dataset|Ratio of FOC|Ratio of OV|Ratio of FM|Ratio of VC|AUC Gain (OSTrack)|\n|-|-|-|-|-|-|\n|LaSOT|42.1%|37.1%|18.9%|11.8%|+1.1%|\n|LaSOT_ext| 62.7%| 21.3%| 58.7%| 39.3%| +3.1%|\n|TNL2K| 13.9%|33.4%| 24.0%| 44.3%| +2.2%|\n|TrackingNet|4.7%|4.0%|10.0%|- (not labeled)|+0.1%|\n\n**About the confusion of applying the non-uniform resizing to the cropped template**: We are sorry for not clearly indicating that we did not apply our non-uniform resizing on the template image. Although the template image can be non-uniformly resized by solving QP with the ground truth annotation and a relatively small context factor only once at the beginning frame during testing, we experimentally find that either applying non-uniform resizing on template image alone or on both template and search images performs worse than applying it only on search image (see below). We think the reason is that the template image must have a relatively small context factor to contain minimal necessary context, which makes the whole template region important for the tracking network to locate the target. So the trackers cannot benefit from non-uniform resizing of template images. We will include this new analysis in the revision.\n||Zoom|Zoom|LaSOT|LaSOT|\n|-|-|-|-|-|\n||Template|Search|AUC|P|\n|OSTrack-256|√||64.8|71.0|\n|OSTrack-256 ||√|**70.2**|**76.2**|\n|OSTrack-256 |√|√| 69.6|75.5|\n\n**About the confusion of one grid approximated by another grid**: We are sorry for this misleading expression. We can use \"estimated\" instead of \"approximated\" here, i.e., the larger grid is estimated by interpolating the smaller grid. We will correct it in the revision.\n\n**About the relationship between the small constant in Eq. (6) and the rigid energy in Eq. (8)**: Thanks for the thoughtful comment. On the one hand, the small constant ensures importance score $S_{k,l}\\neq 0$. Thus, every location $(k,l)$ cannot have extreme $d_l^{row}$ and $d_k^{col}$, otherwise the total energy will be high. On the other hand, the rigid energy further prevents extreme aspect ratio ($d_k^{col} / d_l^{row}$) change, which is also imposed on every location because the importance score $S_{k,l}\\neq 0$. In other words, both the small constant and rigid energy help to avoid extreme deformation. We will give a clearer explanation in the revision.\n\n**About the inconsistency of using MACs and FLOPs and the confusion of computational overhead in Tab. 1**: We are sorry for the misleading expression. **First**, it is a common practice to use MACs to evaluate the computational overhead of a deep neural network. We follow this common practice and use MACs to compare with other methods. However, our non-uniform resizing contains non-pytorch operations, e.g., solving the QP problem, therefore it cannot be measured by the commonly used counter 'pytorch-OpCounter' designed for counting pytorch operations. We thus use the Performance Application Programming Interface (PAPI) to calculate the CPU FLOPs for our method. Generally speaking, 1 MACs approximately equals 2 FLOPs (1 multiply and 1 accumulation operations). The FLOPS for our method to resize an image is 2.56M FLOPs which is approximately equal to 1.28M MACs. Thanks for pointing out this inconsistency. We will convert the FLOPs to MACs in the final version for consistency. **Second**, we indeed add the additional overhead (1.28M MACs) to the corresponding baseline in Tab. 1 so they are not the same. As the additional overhead is far less than the overhead of the network inference (1.28M MACs v.s. 21.5G MACs), the difference is rounded off. Thanks for pointing it out. We will make an explanation in the revision."}]

Author rebuttal:
Thank you for the valuable reviews pointing out that our *simple but clever* (`Gq4E`) and *novel* (`Fe6p`) method, focusing on *a very interesting problem which is little-explored* (`2uk1`) and *easily overlooked* (`P4Ao`), *makes a large step towards closing the gap between fast and accurate trackers and advancing the Pareto front of the AUC vs MACs trade-off* (`Gq4E`) and is also *the first time* (`PYzX`) applied to the tracking field while *bringing adequate enlightenment* (`P4Ao`), and acknowledging that our *technically plausible* (`P4Ao`), *elegant* (`2uk1`) and *clearly explained* (`PYzX`) solution can achieve *impressive* (`2uk1`) and *consistent improvements* (`PYzX`) over the baselines on most of the tracking datasets. Prompted by the insightful reviews, we mainly present the following additional experimental results and analyses for the common questions:

- Following the suggestion by Reviewer `Gq4E`, we additionally explore whether optimizing the original crop hyper-parameters in the uniform resizing baseline (i.e., setting $c^w$ and $c^h$ using two different choices as in Sec. 3.1, varying the search-image context factor, and varying the template-image size to alleviate the mismatch of the object size between the search and temple images) can have a similar effect to our proposed non-uniform resizing. Despite the fact that our limited compute resources make us unable to optimize the above hyper-parameters on large tracking datasets in a grid-search manner, our adequately powered experiments clearly show that the above optimizing is indeed hardly able to achieve similar effects to our resizing (See Tab. R1 in the attached PDF). Moreover, it is experimentally shown that the mismatch of the object size between the search and temple images caused by increasing the search-image context factor has a relatively small effect on the final tracking accuracy for modern transformer trackers.

- As for the confusion of applying the non-uniform resizing on the cropped template (raised by Reviewers `Gq4E`, `Fe6p` and `2uk1`), we are sorry for not clearly indicating that we did not apply our non-uniform resizing on the template image. The reason is that we experimentally find that either applying non-uniform resizing on the template image alone or on both template and search images performs worse than applying it only on the search image (see Tab. R2). We think the reason is that the template image must have a relatively small context factor to contain minimal necessary context, which makes the whole template region important for the tracking network to locate the target. So the trackers cannot benefit from non-uniform resizing of the template image.

- Following the suggestions by Reviewers `Gq4E`, `Fe6p`, `P4Ao`, and `2uk1`, we further apply our method to both the performance-oriented version (backbone: Swin Transformer-Base) and speed-oriented version (backbone: Swin Transformer-Tiny) of SwinTrack [15] to experimentally show the generalization capability of our proposed resizing method for wide range of visual tracking algorithms. Due to the limited computing resources, we only report the comparisons on the large-scale LaSOT dataset (See Tab. R3), which further clearly show the superiority of our approach. Although analyzing the importance of choosing input resizing method is an overlooked and little-explored problem in the visual tracking field and the common practice is using uniform resizing by default, we believe our further analysis based on SwinTrack, as well as the previously explored two baselines (OSTrack and TransT), can demonstrate the importance of choosing appropriate input resizing method.

- As for the concern of hyper-parameters (e.g., $\beta$, $\gamma$, $\lambda$) in our formulation (raised by Reviewers `PYzX` and `P4Ao`), we claim that this small set of hyper-parameters (acknowledged by Reviewer `Gq4E`) is tightly coupled with our formulation and also has very clear meanings, so that we can choose a set of reasonable initial values (which may be the optimal values) and then conduct small (coordinate-wise) sweep (acknowledged by Reviewer `Gq4E`) for them to test the sensitivity of our OSTrack-Zoom to these hyper-parameter choices in Tab. 3. Please refer to the separate responses for more details. We then directly apply our method using the same set of hyper-parameters on TransT and find significant improvements thanks to the robustness of our method. As acknowledged by Reviewer `Gq4E`, using the same hyper-parameters for all datasets and both trackers is a strength of our paper. Note that the improvements over SwinTrack (see Tab. R3) are also achieved using the same hyper-parameters. We do **respectfully** disagree with Reviewer `P4Ao` on the statement that our performance gains seem limited for the reason that the results in Tab. 3 show some sensitivity to the small set of hyper-parameters in our formulation. Because these hyper-parameters are not independent of or orthogonal to our whole framework. We claim that our formulation also needs to be appropriately set to unveil the power of non-uniform resizing.

- As for the concern of small improvements on TrackingNet (raised by Reviewers `PYzX` and `2uk1`), we additionally analyze the ratios of challenging scenarios appearing in the different tracking datasets (See Tab. R4), which clearly show that the test split of TrackingNet has a very different property from LaSOT, LaSOT_ext, and TNL2K with respect to the ratios of challenging scenarios. As shown in Fig. S1 of our appendix, our method can achieve significant performance gains under challenging scenarios such as Fast Motion (FM), Viewpoint Change (VC), Full Occlusion (FOC), Out-of-View (OV), owning to the enlarged visual field. However, the ratios of FOC, OV, FM in TrackingNet (VC is not labeled in TrackingNet) are significantly lower than in LaSOT, LaSOT_ext, and TNL2K, which may be the reason for small improvements of our approach on TrackingNet.

Meta-review: The paper proposes an elegant non-uniform resizing method to retain more raw information for the target area when resizing images with large contexts. By controlling the magnification of important areas and restricting extreme deformation, the resizing is efficiently achieved via quadratic programming. Extensive experiments on challenging benchmarks demonstrate consistent and significant improvements in the accuracy-speed trade-off over strong baselines including both CNN and transformer trackers. The reviewers positively acknowledge the novelty, soundness, clarity, and impact of this technically solid work. ACs concur with the reviewers' assessments and recommend accepting this paper.
Decision: Accept (spotlight)